FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Eddie Gieze
> Sent: Thursday, April 21, 2005 11:11 AM
> 
> I've been using the UTF-8 conversion capabilities of FreeTDS. Most of 
> the characters are converted correctly, however some characters are
not 
> and even produce illegal surrogate pairs.

Wow, it's just not my day today, is it?  

> For example, i have a piece of text with central european characters
in 
> the Latin Extended-A range. This is what i see in the data returned by

> FreeTDS:
> 
> C4 3F 74 C4 81 6A
> 
> It should be:
> 
> C4 93 74 C4 81 6A
> 
> There are two characters in this piece of text which are not ASCII:
> 
> Unicode               UTF-8
> 
> 0101          C481
> 0113          C493
> 
> As you can see the first character is translated correctly but the 
> second character is not. The range of a second byte in a surrogate
pair 
> is 0x80 to 0xBF. C4 3F should never occur in UTF-8 encoding.

Understood.  

> I know that FreeTDS uses iconv for character conversion. Do you think 
> this problem is caused by iconv? 

It seems unlikely.  Does this work correctly on your machine?:

$ perl -e'print "\xC4\x81"' | iconv -t ucs-2le -f utf-8 | hexdump -C
00000000  01 01                                             |..|
00000002
$ perl -e'print "\x13\x1"' | iconv -f ucs-2le -t utf-8 | hexdump -C
00000000  c4 93                                             |..|
00000002

That looks pretty good to me.  If you want to eliminate the iconv
library as a suspect, you could configure FreeTDS --without-libiconv.
The built-in iconv replacement handles UCS-2 <-> UTF-8 conversion.  

I would be interested to see a TDSDUMP log of a simple query of a row
containing U+0113.  Maybe something else -- collation, server charset,
TDS version, gamma rays -- is a factor.  

Does this happen for *every* conversion from U+0113?  Or only sometimes?
libtds processes data from the server by passing it in 128-byte "chunks"
to iconv(3).  Could it be a boundary condition?  

> I'm currently using OS X and this is my version of libiconv:
> 
>   i   libiconv         1.9.1-11     Character set conversion library
...
> I'm using FreeTDS version freetds-0.64.dev.20050414

Well, that's (I take it) GNU iconv and a current overnight snapshot of
FreeTDS.  At least if it is a bug we can fix, you're set to apply the
patch.  

--jkl


-----------------------------------------
The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s) 
named
above. If you are not the intended recipient, or an employee or agent 
responsible
for delivering this message to the intended recipient, any review, 
dissemination,
distribution or duplication of this communication is strictly prohibited. If 
you are
not the intended recipient, please contact the sender immediately by reply 
e-mail
and destroy all copies of the original message. Please note that we do not 
accept
account orders and/or instructions by e-mail, and therefore will not be 
responsible
for carrying out such orders and/or instructions.  If you, as the intended 
recipient
of this message, the purpose of which is to inform and update our clients, 
prospects
and consultants of developments relating to our services and products, would 
not
like to receive further e-mail correspondence from the sender, please "reply" 
to the
sender indicating your wishes.  In the U.S.: 1345 Avenue of the Americas, New 
York,
NY 10105.