FreeTDS Development Group

[Marc Abramowitz <marca AT surveymonkey.com>]

Some more info…

I realized that CP1252 is a single byte encoding that isn't capable of 
representing Unicode strings and I read
in the FreeTDS manual that it encodes things to UCS-2 to send to the database 
and I
found examples of that in tds/query.c — so I borrowed from some other spots
in tds/query.c and changed the code to this:

diff --git a/src/tds/query.c b/src/tds/query.c
index 168c4d5..95e5a4b 100644
--- a/src/tds/query.c
+++ b/src/tds/query.c
@@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
  /* we need to convert data before */
  /* TODO this can be a waste of memory... */
  converted = 1;
- s = tds_convert_string(tds, curcol->char_conv, s, colsize, &output_size);
+ tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling tds_convert_string on 
\"%s\" (strlen %d); colsize = %d; client_charset = \"%s\"; server_charset = 
\"%s\"\n",
+    s, (int) strlen(s), (int) colsize, 
tds->char_convs[client2ucs2]->client_charset.name, 
tds->char_convs[client2ucs2]->server_charset.name);
+ s = tds_convert_string(tds, tds->char_convs[client2ucs2], s, colsize, 
&output_size);
+ tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of tds_convert_string => 
\"%s\"\n", s);
  colsize = (TDS_INT)output_size;
  if (!s) {
  /* on conversion error put a empty string */

And that seems to help although now I get back a result that is encoded in 
UCS-2
(not sure if that is a FreeTDS bug or a pymssql bug).

Freetds log:

query.c:1437:tds_put_data_info putting param_name.
query.c:1471:tds_put_data_info putting status.
query.c:1572:tds_put_data: colsize = 20
query.c:1629:tds_put_data: calling tds_convert_string on "Здравствуй^A" 
(strlen 21); colsize = 20; client_charset = "UTF-8"; server_charset = 
"UCS-2LE"
query.c:1632:tds_put_data: result of tds_convert_string => 
"^W^D4^D@^D0^D2^DA^DB^D2^DC^D9^Dt"
query.c:1648:tds_put_data: not null param varint_size = 2
util.c:156:Changed query state from QUERYING to PENDING
net.c:741:Sending packet
0000 03 01 00 54 00 00 01 00-14 00 73 00 6f 00 6d 00 |...T.... ..s.o.m.|
0010 65 00 50 00 72 00 6f 00-63 00 57 00 69 00 74 00 |e.P.r.o. c.W.i.t.|
0020 68 00 4f 00 6e 00 65 00-50 00 61 00 72 00 61 00 |h.O.n.e. P.a.r.a.|
0030 6d 00 00 00 00 00 a7 14-00 09 04 d0 00 34 14 00 |m....... .....4..|
0040 17 04 34 04 40 04 30 04-32 04 41 04 42 04 32 04 |..4.@.0. 2.A.B.2.|
0050 43 04 39 04            -                        |C.9.|

Result:

AssertionError: u'\x17\x044\x04@\x040\x042\x04A\x04B\x042\x04C\x049\x04!' != 
u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439!'

I am expecting to get back the Unicode string on the right, but what I get 
back is the UTF-16 string on the left.

From: Marc Abramowitz <marca AT surveymonkey.com<mailto:marca AT surveymonkey.com>>
Date: Wednesday, February 27, 2013 2:58 PM
To: "freetds AT lists.ibiblio.org<mailto:freetds AT lists.ibiblio.org>" 
<freetds AT lists.ibiblio.org<mailto:freetds AT lists.ibiblio.org>>
Subject: Re: Sending Unicode (UTF-8) data to dbrpcparam - SQL Server receives 
blank string

A little more info…

If I add the following:

diff --git a/src/tds/query.c b/src/tds/query.c
index 168c4d5..17f227c 100644
--- a/src/tds/query.c
+++ b/src/tds/query.c
@@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
                /* we need to convert data before */
                /* TODO this can be a waste of memory... */
                converted = 1;
+               tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling 
tds_convert_string on \"%s\"; colsize = %d; client_charset = \"%s\"; 
server_charset = \"%s\"\n",
+                           s, colsize, 
curcol->char_conv->client_charset.name, 
curcol->char_conv->server_charset.name);
                s = tds_convert_string(tds, curcol->char_conv, s, colsize, 
&output_size);
+               tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of 
tds_convert_string => \"%s\"\n", s);
                colsize = (TDS_INT)output_size;
                if (!s) {
                        /* on conversion error put a empty string */

Here's what I get:

query.c:1624:tds_put_data: calling tds_convert_string on "Здравствуй^A"; 
colsize = 20; client_charset = "UTF-8"; server_charset = "CP1252"
util.c:331:tdserror(0x101506eb0, 0x10159fdb0, 2402, 0)
dblib.c:7929:dbperror(0x10159fac0, 2402, 0)
dblib.c:7981:2402: "Error converting characters into server's character set. 
Some character(s) could not be converted"
dblib.c:8002:"Error converting characters into server's character set. Some 
character(s) could not be converted", client returns 2 (INT_CANCEL)
util.c:361:tdserror: client library returned TDS_INT_CANCEL(2)
util.c:384:tdserror: returning TDS_INT_CANCEL(2)
query.c:1627:tds_put_data: result of tds_convert_string => "(null)"

Should I be suspicious of the "^A" (null byte?) tacked on to the end of my 
string? Maybe that 's why the conversion is failing?