FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

2009/6/9 Andrew Victor <avictor.za AT gmail.com>:
> hi Frediano,
>
>> I saw patch about CTLib and nvarchar. I forgot to comment and review
>> this patch but looking at unitests results it seems that this patch
>> fix nothing and break too much.
>
> The patch actually allows you to store UTF8-encoded data in an
> NVARCHAR field using CTLib.
>

Hi,
  thanks to be still here, sometimes my comments, I must admit are too
"strong"...
I don't know ctlib that much but looking at documentation (see
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc35396_1500/html/occprc/X33463.htm
and Datatypes link) it seems that client datatype does not support
wide characters, isn't it? So you are correctly trying to bind client
utf-8 (multi-byte) to server wide characters (nvarchar in mssql),
isn't it?

>
>> The patch should try to fix ct_param
>> with wide charset source
>
> No, the issue is at a lower level than ct_param().
> The source data is already 'wide" UTF-8 ("client charset = UTF-8").
> FreeTDS must just remember the data is "wide" so it calls iconv() to
> convert the data to UCS2, and not CP1252 when it submits it to the
> server.
>
>

ok

>
> Here follows the original bug report email with trace I submitted to
> the mailing list ...
>
>
> What is the correct CTLib bind type to use when calling a
> stored-procedure having a NVARCHAR parameter?
>
> I set the CS_DATAFMT.datatype field to CS_UNICHAR_TYPE in the call to
> ct_param().
> The data being bound is a UTF-8 string (and can be converted to UCS-2
> using the command-line iconv utility).
> My freetds.conf contains "client charset = UTF-8"
>
> When calling ct_send(), the following client callback is generated:
>
> [2008/11/19 16:32:59.069] ERROR : Client message:
> [2008/11/19 16:32:59.070] ERROR : Message number: 98, Severity = 9,
> Layer = 0, Origin = 0
> [2008/11/19 16:32:59.070] ERROR : Message: Error converting characters
> into server's character set. Some character(s) could not be converted
>

So... client use (correctly) utf-8 and you want wide-characters on
server. This should be compatible with libTDS !!!

Do you have an example so I can write a small unittests??

I think that
- src/tds/convert.c doesn't require any patch, just use SYBVARCHAR
(client multibye encode)
- tds_set_param_type (data.c) can support SYBNVARCHAR or just pass
XSYBNVARCHAR (TODO: check Sybase behaviour)
- tds_get_cardinal_type (data.c) should use SYBVARCHAR (libTDS doesn't
handle wide characters)
- cs_convert (cs.c) shouldn't need any change, data should already be
in utf8 encoding
- _ct_get_client_type (ct.c) require some more works, it should test
col->on_server.column_type
- paraminfoalloc (ct.c)... require some additional computation...

> The dump file contains:
> 16:32:59.069348 4673 (ct.c:215):setting command state from IDLE to READY
> 16:32:59.069377 4673 (ct.c:3364):ct_param()
> 16:32:59.069391 4673 (ct.c:3365):ct_param() data addr = 0x804d6b7 data
> length = 12
> 16:32:59.069405 4673 (ct.c:4341): _ct_fill_param() status = 256
> 16:32:59.069418 4673 (ct.c:2009):_ct_get_server_type(0)
> 16:32:59.069430 4673 (ct.c:3396): ct_param() added rpc parameter @Id
> 16:32:59.069576 4673 (ct.c:3364):ct_param()
> 16:32:59.069590 4673 (ct.c:3365):ct_param() data addr = 0x891e7d8 data
> length = 4
> 16:32:59.069604 4673 (ct.c:4341): _ct_fill_param() status = 256
> 16:32:59.069618 4673 (ct.c:2009):_ct_get_server_type(25)
> 16:32:59.069631 4673 (ct.c:3396): ct_param() added rpc parameter @eMail
> 16:32:59.069645 4673 (ct.c:844):ct_send() command_type = 149
> 16:32:59.069664 4673 (ct.c:4256):paraminfoalloc: status = 256, maxlen 0
> 16:32:59.069678 4673 (ct.c:4257):paraminfoalloc: name = @Id, varint
> size 2 column_type 47 size 12, 12 column_cur_size 12 column_output = 0
> 16:32:59.069692 4673 (ct.c:4085):paramrowalloc, size = 12, data =
> 0x891eca0, row_size = 0
> 16:32:59.069707 4673 (ct.c:4256):paraminfoalloc: status = 256, maxlen 0
> 16:32:59.069719 4673 (ct.c:4257):paraminfoalloc: name = @eMail, varint
> size 2 column_type 39 size 4, 4 column_cur_size 4 column_output = 0
> 16:32:59.069733 4673 (ct.c:4085):paramrowalloc, size = 4, data =
> 0x89416b8, row_size = 0
> 16:32:59.069749 4673 (mem.c:563):tds_free_all_results()
> 16:32:59.069762 4673 (util.c:162):Changed query state from IDLE to QUERYING
> 16:32:59.069777 4673 (iconv.c:634):      XX - tds_iconv() charset
> to_server: UTF-8 -> UCS-2LE
> 16:32:59.069791 4673 (query.c:1292):tds_put_data_info putting param_name
> 16:32:59.069804 4673 (iconv.c:634):     XX - tds_iconv() charset
> to_server: UTF-8 -> UCS-2LE
> 16:32:59.069818 4673 (query.c:1326):tds_put_data_info putting status
> 16:32:59.069831 4673 (query.c:1420):tds_put_data: colsize = 12
> 16:32:59.069843 4673 (query.c:1449):tds_put_data: not null param 
> varint_size = 2
> 16:32:59.069856 4673 (iconv.c:634):     XX - tds_iconv() charset
> to_server: UTF-8 -> CP1252
> 16:32:59.069870 4673 (query.c:1292):tds_put_data_info putting param_name
> 16:32:59.069883 4673 (iconv.c:634):     XX - tds_iconv() charset
> to_server: UTF-8 -> UCS-2LE
> 16:32:59.069904 4673 (query.c:1326):tds_put_data_info putting status
> 16:32:59.069917 4673 (query.c:1420):tds_put_data: colsize = 4
> 16:32:59.069930 4673 (query.c:1449):tds_put_data: not null param 
> varint_size = 2
> 16:32:59.069942 4673 (iconv.c:634):     XX - tds_iconv() charset
> to_server: UTF-8 -> CP1252
> 16:32:59.069956 4673 (util.c:334):tdserror(0x891e878, 0x891ea00, 2402, 0)
> 16:32:59.070035 4673 (util.c:368):tdserror: client library returned
> TDS_INT_CANCEL(2)
> 16:32:59.070049 4673 (util.c:389):tdserror: returning TDS_INT_CANCEL(2)
> 16:32:59.070063 4673 (util.c:162):Changed query state from QUERYING to 
> PENDING
> 16:32:59.070076 4673 (net.c:849):Sending packet
>
> The first parameter (@Id) is an ASCII string (CS_CHAR_TYPE) and the
> 2nd parameter (@eMail) is a UTF8 (CS_UNICHAR_TYPE) string.
>

So you bind @email with datafmt->datatype == CS_UNICHAR_TYPE, isn't it ??

> The lines of debugging marked with "XX" I added.
> So it looks like the CS_UNICHAR_TYPE parameter is being converted to
> CP1252, which is the server's single-byte charset.
>
> Am I using the wrong types, or is this a bug in FreeTDS?
>

Probably the second :) Well... a bug or a missing feature :)
CS_UNIVARCHAR_TYPE isn't better ??

>
> Regards,
>  Andrew Victor

bye
  freddy77