Skip to Content.
Sympa Menu

freetds - Re: [freetds] FreeTDS and SQL Server 2008 UTF-16 characters

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Burkhard AT habmalnefrage.de
  • To: freetds AT lists.ibiblio.org
  • Cc: jklowden AT freetds.org
  • Subject: Re: [freetds] FreeTDS and SQL Server 2008 UTF-16 characters
  • Date: Thu, 23 Sep 2010 11:42:02 +0200

Hi James,

Thanks for your response. The log file indicates that the server reports the
column type as UCS-2:


net.c:671:Received packet
0000 81 01 00 00 00 09 00 63-00 00 20 00 09 04 d0 00 |.......c .. .....|
0010 34 0d 00 62 00 75 00 72-00 6b 00 68 00 61 00 72 |4..b.u.r .k.h.a.r|
0020 00 64 00 5f 00 74 00 65-00 73 00 74 00 04 6e 00 |.d._.t.e .s.t..n.|
0030 61 00 6d 00 65 00 d1 10-64 75 6d 6d 79 20 74 65 |a.m.e... dummy te|
0040 78 74 70 74 72 00 00 00-64 75 6d 6d 79 54 53 00 |xtptr... dummyTS.|
0050 08 00 00 00 41 00 00 d8-d0 dd 42 00 ff 11 00 c1 |....A... ..B.....|
0060 00 01 00 00 00 79 00 00-00 00 fe 00 00 e0 00 00 |.....y.. ........|
0070 00 00 00 - |...|

token.c:510:processing result tokens. marker is 81(TDS7_RESULT)
token.c:1520:processing TDS7 result metadata.
mem.c:563:tds_free_all_results()
token.c:1544:set current_results (1 column) to tds->res_info
token.c:1555:setting up column 0
token.c:3294:adjust_character_column_size:
Server charset: UCS-2LE
Server column_size: 2097152
Client charset: UTF-8
Client column_size: 4194304
token.c:1490:tds7_get_data_info:
colname = name (4 bytes)
type = 35 (text)
server's type = 99 (UCS-2 text)
column_varint_size = 4
column_size = 4194304 (2097152 on server)


(and in the TDS documentation I don't see other ways of sending unicode
characters than UCS-2). On the SQL Server type, there is no difference in the
column type, whether a string contains "surrogate characters" (and needs to
be interpreted as UTF-16) or not. This seems OK to me as UCS-2 is a subset of
UTF-16.

Nevertheless, when I make a query in SQL Server Management Studio, it
correctly converts the characters from UTF-16 to UTF-8 when writing the
results to a report file.

In the example above, the value retrieved from the database is a
three-UTF-16-character string: character "A", the character U+101D0 ("
PHAISTOS DISC SIGN PEDESTRIAN" :-), and "B".

I have also tried using the XML data type instead of NVARCHAR, the FreeTDS
log file looks similar.

>From all that I learned about this, it seems it would be safe intepreting
>the UCS-2 characters as UTF-16, for the conversion to UTF-8 - since UTF-16
>is backwards compatible to UCS-2. (see
>http://en.wikipedia.org/wiki/UTF-16/UCS-2 ). Maybe for now as an option to
>the configure script?

I would appreciate your help with such a change... for sure I can help with
testing it.

Best Regards,
Burkhard
--
Neu: GMX De-Mail - Einfach wie E-Mail, sicher wie ein Brief!
Jetzt De-Mail-Adresse reservieren: http://portal.gmx.net/de/go/demail




Archive powered by MHonArc 2.6.24.

Top of Page