[freetds] Status of UTF-8 support in 0.83 dev
Sebastien FLAESCH
sf at 4js.com
Thu Aug 14 09:12:34 EDT 2008
It appears that a conversion failure occurs in tds_put_data, near line 1477:
/* we need to convert data before */
/* TODO this can be a waste of memory... */
s = tds_convert_string(tds, curcol->char_conv, s, colsize, &colsize);
if (!s) {
/* FIXME this is a bad place to return error... */
/* TODO on memory failure we should compute converted size and use chunks */
return TDS_FAIL;
}
converted = 1;
Occurs when string contains UTF-8 characters encoded with 3 bytes.
2-byte chars pass without problem...
When I print the string of curcol->column_data I get some strange characters at
the end of the string... Hence UTF-8 characters should be displayed without prob
by using fprintf, as long as LANG=en_US.utf8 ... no?
I will double check the UTF-8 string I put in my buffers, as well as the size
passed to SQLBindParameter, but when I print the string in the ODBC program,
it displays without additional characters at the end...
In my ODBC program:
UTF-8 = [abcéô鴻] ( size= 10 )
In tds.c code:
UTF-8 = [abcéô鴻01000] ( size= 10 )
...
Seb
Sebastien FLAESCH wrote:
> Thank you James for looking at this!
>
> UTF-8 support is critical for one of of customers in Asia...
>
> I have identified a first issue: I must of course use N'xxx' string
> constants when inserting UNICODE/UTF-8 data in the database.
> In previous sample program, the string constants where identified as
> non-Unicode strings in the current database locale. Sorry...
>
> I have added a SELECT to fetch inserted rows, and it seems to be fine
> so far. I have checked the values in the database with SQL Server's
> query tool and the characters are correct.
>
> Still the insert statement using UTF-8 data fails with:
>
> Error -1 at: SQLExecute 1.2 StmtH
> Diagnostic info:
> SQL State: 42000
> SQL code : 8005
> Message : [FreeTDS][SQL Server]The incoming tabular data stream
> (TDS) remote procedure call (RPC) protocol stream is
> incorrect. Parameter 4: The parameter name is invalid.
>
>
> Note that the first insert with latin characters "abcéÓÎ" / "abcÛÊé"
> (2 bytes long) works, it's the second insert using Asian characters
> (3 bytes long)...
>
>
> FYI I have rebuild freetds with iconv 1.12 now:
>
> [root at fox ~]# ldd -r /opt3/dbs/tds/0.83/lib/libtdsodbc.so
> libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x00111000)
> librt.so.1 => /lib/tls/librt.so.1 (0x00a7b000)
> libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00957000)
> libc.so.6 => /lib/tls/libc.so.6 (0x001ed000)
> /lib/ld-linux.so.2 (0x00739000)
>
> Thanks,
> Seb
>
>
> James K. Lowden wrote:
>> Sebastien FLAESCH wrote:
>>> Log shows now:
>>>
>>> log.c:190:Starting log file for FreeTDS 0.83.dev.20080812
>>> on 2008-08-12 18:11:48 with debug flags 0x4fff.
>>> iconv.c:197:names for ISO-8859-1: ISO-8859-1
>>> iconv.c:197:names for UTF-8: UTF-8
>>> iconv.c:197:names for UCS-2LE: UCS-2LE
>>> iconv.c:197:names for UCS-2BE: UCS-2BE
>>> iconv.c:363:iconv to convert client-side data to the "UTF-8" character
>>> set iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
>>> iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
>>
>> Hi Sebastien,
>> I modified the log messages from iconv.c to make them clearer. HTH.
>>> But still, I get:
>>>
>>>
>>> Error -1 at: SQLExecute 1.2 StmtH
>>> Diagnostic info:
>>> SQL State: 42000
>>> SQL code : 8005
>>> Message : [FreeTDS][SQL Server]The incoming tabular data stream
>>> (TDS) remote procedure call (RPC) protocol stream is incorrect.
>>> Parameter 4: The parameter name is invalid.
>>
>> This happens because of these errors in the log:
>>
>>> query.c:1326:tds_put_data_info putting status
>>> query.c:1420:tds_put_data: colsize = 9
>>> query.c:1449:tds_put_data: not null param varint_size = 2
>>> util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
>>> odbc.c:2106:msgno 2402 20003
>>> util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
>>> util.c:389:tdserror: returning TDS_INT_CANCEL(2)
>>> query.c:1326:tds_put_data_info putting status
>>> query.c:1420:tds_put_data: colsize = 9
>>> query.c:1449:tds_put_data: not null param varint_size = 2
>>> util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
>>> odbc.c:2106:msgno 2402 20003
>>> util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
>>> util.c:389:tdserror: returning TDS_INT_CANCEL(2)
>>> util.c:162:Changed query state from QUERYING to PENDING
>>> net.c:768:Sending packet
>>
>> The resulting sp_execute call is incorrectly formed, and the server
>> complains.
>> 2402 is TDSEICONVO and indicates an invalid multibyte input sequence was
>> encountered while sending data to the server.
>> I'm too tired to debug it further. I would guess the parameter data are
>> assumed to be in ISO-8859-1, not in the client's encoding.
>> Regards,
>> --jkl
>> _______________________________________________
>> FreeTDS mailing list
>> FreeTDS at lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds
More information about the FreeTDS
mailing list