Skip to Content.
Sympa Menu

freetds - [freetds] UTF-8 conversion

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Eddie Gieze <egieze AT techno-design.com>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: [freetds] UTF-8 conversion
  • Date: Thu, 21 Apr 2005 17:10:53 +0200

Dear List,

I've been using the UTF-8 conversion capabilities of FreeTDS. Most of the characters are converted correctly, however some characters are not and even produce illegal surrogate pairs.

For example, i have a piece of text with central european characters in the Latin Extended-A range. This is what i see in the data returned by FreeTDS:

C4 3F 74 C4 81 6A

It should be:

C4 93 74 C4 81 6A

There are two characters in this piece of text which are not ASCII:

Unicode UTF-8

0101 C481
0113 C493

As you can see the first character is translated correctly but the second character is not. The range of a second byte in a surrogate pair is 0x80 to 0xBF. C4 3F should never occur in UTF-8 encoding.

I know that FreeTDS uses iconv for character conversion. Do you think this problem is caused by iconv? I find it hard to believe that iconv creates illegal surrogate pairs. I also think it's strange that FreeTDS makes errors in the Latin Extended-A range, because i think there are alot of users who have their text in this range.

I'm currently using OS X and this is my version of libiconv:

i libiconv 1.9.1-11 Character set conversion library
i libiconv-bin 1.9.1-11 Executables for libiconv package
i libiconv-dev 1.9.1-11 Developer files for libiconv package

I'm using FreeTDS version freetds-0.64.dev.20050414

Maybe i'm doing something wrong? But then why are most of the characters converted correctly? Here are some snapshots of my freetds.conf and odbc.ini.

freetds.conf:

client charset = UTF-8

odbc.ini:

[EDQL]
Driver = /usr/local/lib/libtdsodbc.so
Description = Eddie SQL Server
Trace = No
Server = 192.168.0.112
Database = UTF8_Test
Port = 1102
TDS_Version = 8.0

I hope you guys can help me out. I have alot of data available using eastern european, cyrillic and greek characters which i'm testing right now. I'm using MS SQL Server 2000.

Kind Regards,
Eddie Gieze





Archive powered by MHonArc 2.6.24.

Top of Page