Skip to Content.
Sympa Menu

freetds - Re: [freetds] Sending Unicode (UTF-8) data to dbrpcparam - SQL Server receives blank string

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Frediano Ziglio <freddy77 AT gmail.com>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] Sending Unicode (UTF-8) data to dbrpcparam - SQL Server receives blank string
  • Date: Thu, 28 Feb 2013 21:22:53 +0000

Actually git version does what you want, if you look dbrpcparam

1) http://gitorious.org/freetds/freetds/blobs/master/src/dblib/rpc.c#line160
2)
http://gitorious.org/freetds/freetds/blobs/Branch-0_91/src/dblib/rpc.c#line164

#1 is the last git version, #2 is the last 0.91, git have some lines

if (type == SYBVARCHAR && IS_TDS7_PLUS(dbproc->tds_socket->conn))
type = XSYBNVARCHAR;

that turns type to XSYBNVARCHAR instead of SYBVARCHAR. You should get
same result on 0.91 with

if (type == SYBVARCHAR && IS_TDS7_PLUS(dbproc->tds_socket))
type = XSYBNVARCHAR;

Frediano

2013/2/28 Marc Abramowitz <marca AT surveymonkey.com>:
> Some more info…
>
> I realized that CP1252 is a single byte encoding that isn't capable of
> representing Unicode strings and I read
> in the FreeTDS manual that it encodes things to UCS-2 to send to the
> database and I
> found examples of that in tds/query.c — so I borrowed from some other spots
> in tds/query.c and changed the code to this:
>
> diff --git a/src/tds/query.c b/src/tds/query.c
> index 168c4d5..95e5a4b 100644
> --- a/src/tds/query.c
> +++ b/src/tds/query.c
> @@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
> /* we need to convert data before */
> /* TODO this can be a waste of memory... */
> converted = 1;
> - s = tds_convert_string(tds, curcol->char_conv, s, colsize, &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling tds_convert_string on
> \"%s\" (strlen %d); colsize = %d; client_charset = \"%s\"; server_charset =
> \"%s\"\n",
> + s, (int) strlen(s), (int) colsize,
> tds->char_convs[client2ucs2]->client_charset.name,
> tds->char_convs[client2ucs2]->server_charset.name);
> + s = tds_convert_string(tds, tds->char_convs[client2ucs2], s, colsize,
> &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of tds_convert_string =>
> \"%s\"\n", s);
> colsize = (TDS_INT)output_size;
> if (!s) {
> /* on conversion error put a empty string */
>
> And that seems to help although now I get back a result that is encoded in
> UCS-2
> (not sure if that is a FreeTDS bug or a pymssql bug).
>
> Freetds log:
>
> query.c:1437:tds_put_data_info putting param_name.
> query.c:1471:tds_put_data_info putting status.
> query.c:1572:tds_put_data: colsize = 20
> query.c:1629:tds_put_data: calling tds_convert_string on "Здравствуй^A"
> (strlen 21); colsize = 20; client_charset = "UTF-8"; server_charset =
> "UCS-2LE"
> query.c:1632:tds_put_data: result of tds_convert_string =>
> "^W^D4^D@^D0^D2^DA^DB^D2^DC^D9^Dt"
> query.c:1648:tds_put_data: not null param varint_size = 2
> util.c:156:Changed query state from QUERYING to PENDING
> net.c:741:Sending packet
> 0000 03 01 00 54 00 00 01 00-14 00 73 00 6f 00 6d 00 |...T.... ..s.o.m.|
> 0010 65 00 50 00 72 00 6f 00-63 00 57 00 69 00 74 00 |e.P.r.o. c.W.i.t.|
> 0020 68 00 4f 00 6e 00 65 00-50 00 61 00 72 00 61 00 |h.O.n.e. P.a.r.a.|
> 0030 6d 00 00 00 00 00 a7 14-00 09 04 d0 00 34 14 00 |m....... .....4..|
> 0040 17 04 34 04 40 04 30 04-32 04 41 04 42 04 32 04 |..4.@.0. 2.A.B.2.|
> 0050 43 04 39 04 - |C.9.|
>
> Result:
>
> AssertionError: u'\x17\x044\x04@\x040\x042\x04A\x04B\x042\x04C\x049\x04!'
> != u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439!'
>
> I am expecting to get back the Unicode string on the right, but what I get
> back is the UTF-16 string on the left.
>
> From: Marc Abramowitz
> <marca AT surveymonkey.com<mailto:marca AT surveymonkey.com>>
> Date: Wednesday, February 27, 2013 2:58 PM
> To: "freetds AT lists.ibiblio.org<mailto:freetds AT lists.ibiblio.org>"
> <freetds AT lists.ibiblio.org<mailto:freetds AT lists.ibiblio.org>>
> Subject: Re: Sending Unicode (UTF-8) data to dbrpcparam - SQL Server
> receives blank string
>
> A little more info…
>
> If I add the following:
>
> diff --git a/src/tds/query.c b/src/tds/query.c
> index 168c4d5..17f227c 100644
> --- a/src/tds/query.c
> +++ b/src/tds/query.c
> @@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
> /* we need to convert data before */
> /* TODO this can be a waste of memory... */
> converted = 1;
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling
> tds_convert_string on \"%s\"; colsize = %d; client_charset = \"%s\";
> server_charset = \"%s\"\n",
> + s, colsize,
> curcol->char_conv->client_charset.name,
> curcol->char_conv->server_charset.name);
> s = tds_convert_string(tds, curcol->char_conv, s, colsize,
> &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of
> tds_convert_string => \"%s\"\n", s);
> colsize = (TDS_INT)output_size;
> if (!s) {
> /* on conversion error put a empty string */
>
> Here's what I get:
>
> query.c:1624:tds_put_data: calling tds_convert_string on "Здравствуй^A";
> colsize = 20; client_charset = "UTF-8"; server_charset = "CP1252"
> util.c:331:tdserror(0x101506eb0, 0x10159fdb0, 2402, 0)
> dblib.c:7929:dbperror(0x10159fac0, 2402, 0)
> dblib.c:7981:2402: "Error converting characters into server's character
> set. Some character(s) could not be converted"
> dblib.c:8002:"Error converting characters into server's character set. Some
> character(s) could not be converted", client returns 2 (INT_CANCEL)
> util.c:361:tdserror: client library returned TDS_INT_CANCEL(2)
> util.c:384:tdserror: returning TDS_INT_CANCEL(2)
> query.c:1627:tds_put_data: result of tds_convert_string => "(null)"
>
> Should I be suspicious of the "^A" (null byte?) tacked on to the end of my
> string? Maybe that 's why the conversion is failing?
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds




Archive powered by MHonArc 2.6.24.

Top of Page