FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

Actually git version does what you want, if you look dbrpcparam

1) http://gitorious.org/freetds/freetds/blobs/master/src/dblib/rpc.c#line160
2) 
http://gitorious.org/freetds/freetds/blobs/Branch-0_91/src/dblib/rpc.c#line164

#1 is the last git version, #2 is the last 0.91, git have some lines

        if (type == SYBVARCHAR && IS_TDS7_PLUS(dbproc->tds_socket->conn))
                type = XSYBNVARCHAR;

that turns type to XSYBNVARCHAR instead of SYBVARCHAR. You should get
same result on 0.91 with

        if (type == SYBVARCHAR && IS_TDS7_PLUS(dbproc->tds_socket))
                type = XSYBNVARCHAR;

Frediano

2013/2/28 Marc Abramowitz <marca AT surveymonkey.com>:
> Some more info…
>
> I realized that CP1252 is a single byte encoding that isn't capable of 
> representing Unicode strings and I read
> in the FreeTDS manual that it encodes things to UCS-2 to send to the 
> database and I
> found examples of that in tds/query.c — so I borrowed from some other spots
> in tds/query.c and changed the code to this:
>
> diff --git a/src/tds/query.c b/src/tds/query.c
> index 168c4d5..95e5a4b 100644
> --- a/src/tds/query.c
> +++ b/src/tds/query.c
> @@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
>   /* we need to convert data before */
>   /* TODO this can be a waste of memory... */
>   converted = 1;
> - s = tds_convert_string(tds, curcol->char_conv, s, colsize, &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling tds_convert_string on 
> \"%s\" (strlen %d); colsize = %d; client_charset = \"%s\"; server_charset = 
> \"%s\"\n",
> +    s, (int) strlen(s), (int) colsize, 
> tds->char_convs[client2ucs2]->client_charset.name, 
> tds->char_convs[client2ucs2]->server_charset.name);
> + s = tds_convert_string(tds, tds->char_convs[client2ucs2], s, colsize, 
> &output_size);
> + tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of tds_convert_string => 
> \"%s\"\n", s);
>   colsize = (TDS_INT)output_size;
>   if (!s) {
>   /* on conversion error put a empty string */
>
> And that seems to help although now I get back a result that is encoded in 
> UCS-2
> (not sure if that is a FreeTDS bug or a pymssql bug).
>
> Freetds log:
>
> query.c:1437:tds_put_data_info putting param_name.
> query.c:1471:tds_put_data_info putting status.
> query.c:1572:tds_put_data: colsize = 20
> query.c:1629:tds_put_data: calling tds_convert_string on "Здравствуй^A" 
> (strlen 21); colsize = 20; client_charset = "UTF-8"; server_charset = 
> "UCS-2LE"
> query.c:1632:tds_put_data: result of tds_convert_string => 
> "^W^D4^D@^D0^D2^DA^DB^D2^DC^D9^Dt"
> query.c:1648:tds_put_data: not null param varint_size = 2
> util.c:156:Changed query state from QUERYING to PENDING
> net.c:741:Sending packet
> 0000 03 01 00 54 00 00 01 00-14 00 73 00 6f 00 6d 00 |...T.... ..s.o.m.|
> 0010 65 00 50 00 72 00 6f 00-63 00 57 00 69 00 74 00 |e.P.r.o. c.W.i.t.|
> 0020 68 00 4f 00 6e 00 65 00-50 00 61 00 72 00 61 00 |h.O.n.e. P.a.r.a.|
> 0030 6d 00 00 00 00 00 a7 14-00 09 04 d0 00 34 14 00 |m....... .....4..|
> 0040 17 04 34 04 40 04 30 04-32 04 41 04 42 04 32 04 |..4.@.0. 2.A.B.2.|
> 0050 43 04 39 04            -                        |C.9.|
>
> Result:
>
> AssertionError: u'\x17\x044\x04@\x040\x042\x04A\x04B\x042\x04C\x049\x04!' 
> != u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439!'
>
> I am expecting to get back the Unicode string on the right, but what I get 
> back is the UTF-16 string on the left.
>
> From: Marc Abramowitz 
> <marca AT surveymonkey.com<mailto:marca AT surveymonkey.com>>
> Date: Wednesday, February 27, 2013 2:58 PM
> To: "freetds AT lists.ibiblio.org<mailto:freetds AT lists.ibiblio.org>" 
> <freetds AT lists.ibiblio.org<mailto:freetds AT lists.ibiblio.org>>
> Subject: Re: Sending Unicode (UTF-8) data to dbrpcparam - SQL Server 
> receives blank string
>
> A little more info…
>
> If I add the following:
>
> diff --git a/src/tds/query.c b/src/tds/query.c
> index 168c4d5..17f227c 100644
> --- a/src/tds/query.c
> +++ b/src/tds/query.c
> @@ -1621,7 +1621,10 @@ tds_put_data(TDSSOCKET * tds, TDSCOLUMN * curcol)
>                 /* we need to convert data before */
>                 /* TODO this can be a waste of memory... */
>                 converted = 1;
> +               tdsdump_log(TDS_DBG_INFO1, "tds_put_data: calling 
> tds_convert_string on \"%s\"; colsize = %d; client_charset = \"%s\"; 
> server_charset = \"%s\"\n",
> +                           s, colsize, 
> curcol->char_conv->client_charset.name, 
> curcol->char_conv->server_charset.name);
>                 s = tds_convert_string(tds, curcol->char_conv, s, colsize, 
> &output_size);
> +               tdsdump_log(TDS_DBG_INFO1, "tds_put_data: result of 
> tds_convert_string => \"%s\"\n", s);
>                 colsize = (TDS_INT)output_size;
>                 if (!s) {
>                         /* on conversion error put a empty string */
>
> Here's what I get:
>
> query.c:1624:tds_put_data: calling tds_convert_string on "Здравствуй^A"; 
> colsize = 20; client_charset = "UTF-8"; server_charset = "CP1252"
> util.c:331:tdserror(0x101506eb0, 0x10159fdb0, 2402, 0)
> dblib.c:7929:dbperror(0x10159fac0, 2402, 0)
> dblib.c:7981:2402: "Error converting characters into server's character 
> set. Some character(s) could not be converted"
> dblib.c:8002:"Error converting characters into server's character set. Some 
> character(s) could not be converted", client returns 2 (INT_CANCEL)
> util.c:361:tdserror: client library returned TDS_INT_CANCEL(2)
> util.c:384:tdserror: returning TDS_INT_CANCEL(2)
> query.c:1627:tds_put_data: result of tds_convert_string => "(null)"
>
> Should I be suspicious of the "^A" (null byte?) tacked on to the end of my 
> string? Maybe that 's why the conversion is failing?
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds