FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Donald Sures [mailto:donald_sures AT symantec.com]
> Sent: March 18, 2003 12:18 PM
> 
> http://www.freetds.org/userguide/a2698.htm
> 
> i've read that .. i've read that applying the Character Set = UTF-8 
> parameter would allow me to query UTF-8 data from a MSSQL 
> NVarChar column 
> .. The data is 9 characters of japanese text .. these 
> characters have the 
> first byte as E3 which makes them 3 byte characters .. 
> somewhere in the 
> conversion from UCS2 to UTF-8 i'm losing bytes .. 
> 
> is iconv responsible for converting ucs2 to utf-8? 

Yes.  

Donald, I'm sorry, because I think I misled you.  You probably read,
"FreeTDS will happily convert and convey your data in any single-byte format
that iconv can provide. In practice, this normally means some form of ISO
8859-x or UTF-8."  I wrote that, but it's not true.  UTF-8 isn't a
single-byte character set, as you well know.  

We can ask iconv to convert UCS-2 -> UTF-8, but I doubt we handle the
resulting converted buffer correctly.  Basically, it's untested.  Even if we
don't overrun our own buffers, we probably don't correctly distinguish
between byte- and character-counts when handing the data off to the client
(Perl, in your case).  

The first question is, Does your query work in tsql?  Until it does, we have
basic issues.  

Next, how does DBD::Sybase know it's dealing with UTF-8 data from FreeTDS?
How does Perl know?  We have to be sure FreeTDS tells the truth when asked.


Then we can examine the Perl<->FreeTDS interface.  

If we get that far, that would be progress.  

--jkl

> -  Perl: v5.8.0 built for i686-linux
> - TDS_VERSION_NO "freetds v0.60"
> - Microsoft SQL Server  2000 - 8.00.578 (Intel X86)
> - MSSQL Column Type NVCHAR(200)
> - MSSQL Column Text : 9 characters
> - Perl Length() reports text is 9 characters
> - Perl prints first 3 characters of string
> 
> - For every 3 Japanese characters in the database, 1 Japanese 
> character is 
> displayed in perl
>   Here are the hex codes for each byte of the data string:
>         ?e-E3 83 86
>         - E3 82 B9 
>         - E3 83 88 
>         - E3 83 86 
>         - E3 82 B9 
>         - E3 83 88 
>         - E3 83 86 
>         - E3 82 B9 
>         - E3 83 88


The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.