FreeTDS Development Group

[Eddie Gieze <egieze AT techno-design.com>]

Dear List,

I've been using the UTF-8 conversion capabilities of FreeTDS. Most of 
the characters are converted correctly, however some characters are not 
and even produce illegal surrogate pairs.

For example, i have a piece of text with central european characters in 
the Latin Extended-A range. This is what i see in the data returned by 
FreeTDS:

C4 3F 74 C4 81 6A

It should be:

C4 93 74 C4 81 6A

There are two characters in this piece of text which are not ASCII:

Unicode         UTF-8

0101            C481
0113            C493

As you can see the first character is translated correctly but the 
second character is not. The range of a second byte in a surrogate pair 
is 0x80 to 0xBF. C4 3F should never occur in UTF-8 encoding.

I know that FreeTDS uses iconv for character conversion. Do you think 
this problem is caused by iconv? I find it hard to believe that iconv 
creates illegal surrogate pairs. I also think it's strange that FreeTDS 
makes errors in the Latin Extended-A range, because i think there are 
alot of users who have their text in this range.

I'm currently using OS X and this is my version of libiconv:

 i   libiconv         1.9.1-11     Character set conversion library
 i   libiconv-bin     1.9.1-11     Executables for libiconv package
 i   libiconv-dev     1.9.1-11     Developer files for libiconv package

I'm using FreeTDS version freetds-0.64.dev.20050414

Maybe i'm doing something wrong? But then why are most of the 
characters converted correctly? Here are some snapshots of my 
freetds.conf and odbc.ini.

freetds.conf:

client charset = UTF-8

odbc.ini:

[EDQL]
Driver           = /usr/local/lib/libtdsodbc.so
Description     = Eddie SQL Server
Trace           = No
Server          = 192.168.0.112
Database        = UTF8_Test
Port            = 1102
TDS_Version     = 8.0

I hope you guys can help me out. I have alot of data available using 
eastern european, cyrillic and greek characters which i'm testing right 
now. I'm using MS SQL Server 2000.

Kind Regards,
Eddie Gieze