freetds AT lists.ibiblio.org
Subject: FreeTDS Development Group
List archive
[freetds] Unicode Characters/Codepoint Above 0x00FF
- From: Christoph Wittmann <cwittmann AT infosearchintl.com>
- To: freetds AT lists.ibiblio.org
- Subject: [freetds] Unicode Characters/Codepoint Above 0x00FF
- Date: Mon, 29 Jun 2009 16:34:21 -0700
I am using a FreeBSD 7.1 client w/ FreeTDS 0.64_3,1 to insert into a SQL Server 2005 db (SP3), but I cannot get unicode characters above code-point 0x00FF to be properly inserted. However, I can read all unicode characters from the server with a select-statement. Here are the important freetds.conf settings --
[global]
tds version = 8.0
client charset = UTF-8
debug flags = 0x4FFF
I have a test query and the corresponding log file. The test query inserts some text with one non-ascii character: code-point 0x0200 (a capital A with " on top of it). The UTF-8 representation is 0xC880, and the UCS2 and UTF-16 representation is 0x0200 (not accounting for endianism). Here's the query that was used --
if ((erc = dbcmd(dbproc, "INSERT INTO uni (data) VALUES('Hello World: \xC8\x80')")) == FAIL) {
printf("\ndbcmd() ERROR\n");
exit(8);
}
All I get is a question mark in the SQL Server 2005 table. The 3rd-to- last hex-dump in the following log shows the insert-packet, which looks fine. iconv appears to properly convert the insert-query to UCS-2 and the hex shows that the non-ascii character os 0002 (little endian machine). Any ideas?
Thanks in advance for any help.
Chris Wittmann
************************************************************************
FREETDS.LOG
************************************************************************
util.c:288:Starting log file for FreeTDS 0.64
on 2009-06-23 14:44:22 with debug flags 0x4fff.
iconv.c:195:names for ISO-8859-1: ISO-8859-1
iconv.c:195:names for UTF-8: UTF-8
iconv.c:195:names for UCS-2LE: UCS-2LE
iconv.c:195:names for UCS-2BE: UCS-2BE
iconv.c:361:iconv to convert client-side data to the "UTF-8" character set
iconv.c:514:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
iconv.c:514:tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
net.c:168:Connecting to 192.168.50.122 port 1433.
write.c:134:tds_put_string converting 20 bytes of "ws01.infosearch.priv"
write.c:162:tds_put_string wrote 40 bytes
write.c:134:tds_put_string converting 2 bytes of "sa"
write.c:162:tds_put_string wrote 4 bytes
write.c:162:tds_put_string wrote 0 bytes
write.c:134:tds_put_string converting 9 bytes of "desktop05"
write.c:162:tds_put_string wrote 18 bytes
write.c:134:tds_put_string converting 10 bytes of "DB-Library"
write.c:162:tds_put_string wrote 20 bytes
write.c:134:tds_put_string converting 10 bytes of "us_english"
write.c:162:tds_put_string wrote 20 bytes
write.c:162:tds_put_string wrote 0 bytes
token.c:310:tds_process_login_tokens()
net.c:446:Received header
0000 04 01 01 7d 00 39 01 00- |...}.9..|
net.c:542:Received packet
0000 e3 1b 00 01 06 6d 00 61-00 73 00 74 00 65 00 72 |.....m.a .s.t.e.r|
0010 00 06 6d 00 61 00 73 00-74 00 65 00 72 00 ab 68 |..m.a.s. t.e.r..h|
0020 00 45 16 00 00 02 00 25-00 43 00 68 00 61 00 6e |.E.....% .C.h.a.n|
0030 00 67 00 65 00 64 00 20-00 64 00 61 00 74 00 61 |.g.e.d. .d.a.t.a|
0040 00 62 00 61 00 73 00 65-00 20 00 63 00 6f 00 6e |.b.a.s.e . .c.o.n|
0050 00 74 00 65 00 78 00 74-00 20 00 74 00 6f 00 20 |.t.e.x.t . .t.o. |
0060 00 27 00 6d 00 61 00 73-00 74 00 65 00 72 00 27 |.'.m.a.s .t.e.r.'|
0070 00 2e 00 09 44 00 45 00-53 00 4b 00 54 00 4f 00 |....D.E. S.K.T.O.|
0080 50 00 30 00 35 00 00 01-00 e3 08 00 07 05 09 04 |P.0.5... ........|
0090 d0 00 34 00 e3 17 00 02-0a 75 00 73 00 5f 00 65 |..4..... .u.s._.e|
00a0 00 6e 00 67 00 6c 00 69-00 73 00 68 00 00 ab 6c |.n.g.l.i .s.h...l|
00b0 00 47 16 00 00 01 00 27-00 43 00 68 00 61 00 6e |.G.....' .C.h.a.n|
00c0 00 67 00 65 00 64 00 20-00 6c 00 61 00 6e 00 67 |.g.e.d. .l.a.n.g|
00d0 00 75 00 61 00 67 00 65-00 20 00 73 00 65 00 74 |.u.a.g.e . .s.e.t|
00e0 00 74 00 69 00 6e 00 67-00 20 00 74 00 6f 00 20 |.t.i.n.g . .t.o. |
00f0 00 75 00 73 00 5f 00 65-00 6e 00 67 00 6c 00 69 |.u.s._.e .n.g.l.i|
0100 00 73 00 68 00 2e 00 09-44 00 45 00 53 00 4b 00 |.s.h.... D.E.S.K.|
0110 54 00 4f 00 50 00 30 00-35 00 00 01 00 ad 36 00 |T.O.P.0. 5.....6.|
0120 01 71 00 00 01 16 4d 00-69 00 63 00 72 00 6f 00 |.q....M. i.c.r.o.|
0130 73 00 6f 00 66 00 74 00-20 00 53 00 51 00 4c 00 |s.o.f.t. .S.Q.L.|
0140 20 00 53 00 65 00 72 00-76 00 65 00 72 00 00 00 | .S.e.r. v.e.r...|
0150 00 00 09 00 05 77 e3 13-00 04 04 34 00 30 00 39 |.....w.. ...4.0.9|
0160 00 36 00 04 34 00 30 00-39 00 36 00 fd 00 00 00 |.6..4.0. 9.6.....|
0170 00 00 00 00 00 - |.....|
token.c:314:looking for login token, got e3(ENVCHANGE)
token.c:105:tds_process_default_tokens() marker is e3(ENVCHANGE)
read.c:179:tds_get_string: reading 12 from wire to give 6 to client.
read.c:179:tds_get_string: reading 12 from wire to give 6 to client.
token.c:314:looking for login token, got ab(INFO)
token.c:105:tds_process_default_tokens() marker is ab(INFO)
token.c:2439:tds_process_msg() reading message from server
read.c:179:tds_get_string: reading 74 from wire to give 37 to client.
read.c:179:tds_get_string: reading 18 from wire to give 9 to client.
token.c:2501:tds_process_msg() calling client msg handler
token.c:2514:tds_process_msg() returning TDS_SUCCEED
token.c:314:looking for login token, got e3(ENVCHANGE)
token.c:105:tds_process_default_tokens() marker is e3(ENVCHANGE)
iconv.c:514:tds_iconv_info_init: converting "UTF-8"->"CP1252"
token.c:314:looking for login token, got e3(ENVCHANGE)
token.c:105:tds_process_default_tokens() marker is e3(ENVCHANGE)
read.c:179:tds_get_string: reading 20 from wire to give 10 to client.
token.c:314:looking for login token, got ab(INFO)
token.c:105:tds_process_default_tokens() marker is ab(INFO)
token.c:2439:tds_process_msg() reading message from server
read.c:179:tds_get_string: reading 78 from wire to give 39 to client.
read.c:179:tds_get_string: reading 18 from wire to give 9 to client.
token.c:2501:tds_process_msg() calling client msg handler
token.c:2514:tds_process_msg() returning TDS_SUCCEED
token.c:314:looking for login token, got ad(LOGINACK)
read.c:179:tds_get_string: reading 44 from wire to give 22 to client.
token.c:314:looking for login token, got e3(ENVCHANGE)
token.c:105:tds_process_default_tokens() marker is e3(ENVCHANGE)
read.c:179:tds_get_string: reading 8 from wire to give 4 to client.
read.c:179:tds_get_string: reading 8 from wire to give 4 to client.
token.c:2327:increasing block size from 4096 to 4096
token.c:314:looking for login token, got fd(DONE)
token.c:105:tds_process_default_tokens() marker is fd(DONE)
token.c:2143:tds_process_end: more_results = 0
was_cancelled = 0
error = 0
done_count_valid = 0
token.c:2159:tds_process_end() state set to TDS_IDLE
util.c:119:Changing query state from IDLE to IDLE
token.c:387:leaving tds_process_login_tokens() returning 1
dblib.c:991:dbuse()
dblib.c:907:dbcmd() bufsz = 0
dblib.c:957:in dbsqlexec()
dblib.c:5867:in dbsqlsend()
util.c:119:Changing query state from IDLE to QUERYING
mem.c:519:tds_free_all_results()
write.c:134:tds_put_string converting 15 bytes of "use development"
write.c:162:tds_put_string wrote 30 bytes
util.c:119:Changing query state from QUERYING to PENDING
net.c:673:Sending packet
0000 01 01 00 26 00 00 01 00-75 00 73 00 65 00 20 00 |...&.... u.s.e. .|
0010 64 00 65 00 76 00 65 00-6c 00 6f 00 70 00 6d 00 |d.e.v.e. l.o.p.m.|
0020 65 00 6e 00 74 00 - |e.n.t.|
dblib.c:3976:in dbsqlok()
net.c:446:Received header
0000 04 01 00 b9 00 39 01 00- |.....9..|
net.c:542:Received packet
0000 e3 25 00 01 0b 64 00 65-00 76 00 65 00 6c 00 6f |.%...d.e .v.e.l.o|
0010 00 70 00 6d 00 65 00 6e-00 74 00 06 6d 00 61 00 |.p.m.e.n .t..m.a.|
0020 73 00 74 00 65 00 72 00-ab 72 00 45 16 00 00 01 |s.t.e.r. .r.E....|
0030 00 2a 00 43 00 68 00 61-00 6e 00 67 00 65 00 64 |.*.C.h.a .n.g.e.d|
0040 00 20 00 64 00 61 00 74-00 61 00 62 00 61 00 73 |. .d.a.t .a.b.a.s|
0050 00 65 00 20 00 63 00 6f-00 6e 00 74 00 65 00 78 |.e. .c.o .n.t.e.x|
0060 00 74 00 20 00 74 00 6f-00 20 00 27 00 64 00 65 |.t. .t.o . .'.d.e|
0070 00 76 00 65 00 6c 00 6f-00 70 00 6d 00 65 00 6e |.v.e.l.o .p.m.e.n|
0080 00 74 00 27 00 2e 00 09-44 00 45 00 53 00 4b 00 |.t.'.... D.E.S.K.|
0090 54 00 4f 00 50 00 30 00-35 00 00 01 00 e3 08 00 |T.O.P.0. 5.......|
00a0 07 05 09 04 d0 00 34 00-fd 00 00 e2 00 00 00 00 |......4. ........|
00b0 00 - |.|
dblib.c:3995:dbsqlok() marker is e3
util.c:119:Changing query state from PENDING to READING
token.c:526:processing result tokens. marker is e3(ENVCHANGE)
token.c:105:tds_process_default_tokens() marker is e3(ENVCHANGE)
read.c:179:tds_get_string: reading 22 from wire to give 11 to client.
read.c:179:tds_get_string: reading 12 from wire to give 6 to client.
token.c:526:processing result tokens. marker is ab(INFO)
token.c:105:tds_process_default_tokens() marker is ab(INFO)
token.c:2439:tds_process_msg() reading message from server
read.c:179:tds_get_string: reading 84 from wire to give 42 to client.
read.c:179:tds_get_string: reading 18 from wire to give 9 to client.
token.c:2501:tds_process_msg() calling client msg handler
token.c:2514:tds_process_msg() returning TDS_SUCCEED
token.c:526:processing result tokens. marker is e3(ENVCHANGE)
token.c:105:tds_process_default_tokens() marker is e3(ENVCHANGE)
token.c:526:processing result tokens. marker is fd(DONE)
token.c:2143:tds_process_end: more_results = 0
was_cancelled = 0
error = 0
done_count_valid = 0
token.c:2159:tds_process_end() state set to TDS_IDLE
util.c:119:Changing query state from READING to IDLE
util.c:119:Changing query state from IDLE to PENDING
dblib.c:4042:dbsqlok() end status was success
dblib.c:1161:dbresults()
token.c:514:tds_process_tokens() state is COMPLETED
dblib.c:907:dbcmd() bufsz = 16
dblib.c:4978:in dbfreebuf()
dblib.c:957:in dbsqlexec()
dblib.c:5867:in dbsqlsend()
util.c:119:Changing query state from IDLE to QUERYING
mem.c:519:tds_free_all_results()
write.c:134:tds_put_string converting 18 bytes of "TRUNCATE TABLE uni"
write.c:162:tds_put_string wrote 36 bytes
util.c:119:Changing query state from QUERYING to PENDING
net.c:673:Sending packet
0000 01 01 00 2c 00 00 01 00-54 00 52 00 55 00 4e 00 |...,.... T.R.U.N.|
0010 43 00 41 00 54 00 45 00-20 00 54 00 41 00 42 00 |C.A.T.E. .T.A.B.|
0020 4c 00 45 00 20 00 75 00-6e 00 69 00 |L.E. .u. n.i.|
dblib.c:3976:in dbsqlok()
net.c:446:Received header
0000 04 01 00 11 00 39 01 00- |.....9..|
net.c:542:Received packet
0000 fd 00 00 ea 00 00 00 00-00 |........ .|
dblib.c:3995:dbsqlok() marker is fd
util.c:119:Changing query state from PENDING to READING
token.c:526:processing result tokens. marker is fd(DONE)
token.c:2143:tds_process_end: more_results = 0
was_cancelled = 0
error = 0
done_count_valid = 0
token.c:2159:tds_process_end() state set to TDS_IDLE
util.c:119:Changing query state from READING to IDLE
util.c:119:Changing query state from IDLE to PENDING
dblib.c:4042:dbsqlok() end status was success
dblib.c:907:dbcmd() bufsz = 19
dblib.c:4978:in dbfreebuf()
dblib.c:957:in dbsqlexec()
dblib.c:5867:in dbsqlsend()
util.c:119:Changing query state from IDLE to QUERYING
mem.c:519:tds_free_all_results()
write.c:134:tds_put_string converting 48 bytes of "INSERT INTO uni (data) VALUES('Hello World: Ȁ')"
write.c:162:tds_put_string wrote 94 bytes
util.c:119:Changing query state from QUERYING to PENDING
net.c:673:Sending packet
0000 01 01 00 66 00 00 01 00-49 00 4e 00 53 00 45 00 |...f.... I.N.S.E.|
0010 52 00 54 00 20 00 49 00-4e 00 54 00 4f 00 20 00 |R.T. .I. N.T.O. .|
0020 75 00 6e 00 69 00 20 00-28 00 64 00 61 00 74 00 |u.n.i. . (.d.a.t.|
0030 61 00 29 00 20 00 56 00-41 00 4c 00 55 00 45 00 |a.). .V. A.L.U.E.|
0040 53 00 28 00 27 00 48 00-65 00 6c 00 6c 00 6f 00 |S.(.'.H. e.l.l.o.|
0050 20 00 57 00 6f 00 72 00-6c 00 64 00 3a 00 20 00 | .W.o.r. l.d.:. .|
0060 00 02 27 00 29 00 - |..'.).|
dblib.c:3976:in dbsqlok()
net.c:446:Received header
0000 04 01 00 11 00 39 01 00- |.....9..|
net.c:542:Received packet
0000 fd 10 00 c3 00 01 00 00-00 |........ .|
dblib.c:3995:dbsqlok() marker is fd
util.c:119:Changing query state from PENDING to READING
token.c:526:processing result tokens. marker is fd(DONE)
token.c:2143:tds_process_end: more_results = 0
was_cancelled = 0
error = 0
done_count_valid = 1
token.c:2159:tds_process_end() state set to TDS_IDLE
util.c:119:Changing query state from READING to IDLE
token.c:2175: rows_affected = 1
util.c:119:Changing query state from IDLE to PENDING
dblib.c:4042:dbsqlok() end status was success
mem.c:519:tds_free_all_results()
util.c:119:Changing query state from IDLE to DEAD
dblib.c:4978:in dbfreebuf()
- [freetds] Unicode Characters/Codepoint Above 0x00FF, Christoph Wittmann, 06/29/2009
Archive powered by MHonArc 2.6.24.