Skip to Content.
Sympa Menu

freetds - Re: [freetds] Status of UTF-8 support in 0.83 dev

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Sebastien FLAESCH <sf AT 4js.com>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] Status of UTF-8 support in 0.83 dev
  • Date: Thu, 14 Aug 2008 15:12:34 +0200

It appears that a conversion failure occurs in tds_put_data, near line 1477:

/* we need to convert data before */
/* TODO this can be a waste of memory... */
s = tds_convert_string(tds, curcol->char_conv, s, colsize, &colsize);
if (!s) {
/* FIXME this is a bad place to return error... */
/* TODO on memory failure we should compute converted size and use
chunks */
return TDS_FAIL;
}
converted = 1;

Occurs when string contains UTF-8 characters encoded with 3 bytes.
2-byte chars pass without problem...

When I print the string of curcol->column_data I get some strange characters
at
the end of the string... Hence UTF-8 characters should be displayed without
prob
by using fprintf, as long as LANG=en_US.utf8 ... no?

I will double check the UTF-8 string I put in my buffers, as well as the size
passed to SQLBindParameter, but when I print the string in the ODBC program,
it displays without additional characters at the end...

In my ODBC program:

UTF-8 = [abcéô鴻] ( size= 10 )

In tds.c code:

UTF-8 = [abcéô鴻01000] ( size= 10 )

...

Seb

Sebastien FLAESCH wrote:
Thank you James for looking at this!

UTF-8 support is critical for one of of customers in Asia...

I have identified a first issue: I must of course use N'xxx' string
constants when inserting UNICODE/UTF-8 data in the database.
In previous sample program, the string constants where identified as
non-Unicode strings in the current database locale. Sorry...

I have added a SELECT to fetch inserted rows, and it seems to be fine
so far. I have checked the values in the database with SQL Server's
query tool and the characters are correct.

Still the insert statement using UTF-8 data fails with:

Error -1 at: SQLExecute 1.2 StmtH
Diagnostic info:
SQL State: 42000
SQL code : 8005
Message : [FreeTDS][SQL Server]The incoming tabular data stream
(TDS) remote procedure call (RPC) protocol stream is
incorrect. Parameter 4: The parameter name is invalid.


Note that the first insert with latin characters "abcéÓÎ" / "abcÛÊé"
(2 bytes long) works, it's the second insert using Asian characters
(3 bytes long)...


FYI I have rebuild freetds with iconv 1.12 now:

[root@fox ~]# ldd -r /opt3/dbs/tds/0.83/lib/libtdsodbc.so
libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x00111000)
librt.so.1 => /lib/tls/librt.so.1 (0x00a7b000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00957000)
libc.so.6 => /lib/tls/libc.so.6 (0x001ed000)
/lib/ld-linux.so.2 (0x00739000)

Thanks,
Seb


James K. Lowden wrote:
Sebastien FLAESCH wrote:
Log shows now:

log.c:190:Starting log file for FreeTDS 0.83.dev.20080812
on 2008-08-12 18:11:48 with debug flags 0x4fff.
iconv.c:197:names for ISO-8859-1: ISO-8859-1
iconv.c:197:names for UTF-8: UTF-8
iconv.c:197:names for UCS-2LE: UCS-2LE
iconv.c:197:names for UCS-2BE: UCS-2BE
iconv.c:363:iconv to convert client-side data to the "UTF-8" character
set iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"

Hi Sebastien,
I modified the log messages from iconv.c to make them clearer. HTH.
But still, I get:


Error -1 at: SQLExecute 1.2 StmtH
Diagnostic info:
SQL State: 42000
SQL code : 8005
Message : [FreeTDS][SQL Server]The incoming tabular data stream
(TDS) remote procedure call (RPC) protocol stream is incorrect.
Parameter 4: The parameter name is invalid.

This happens because of these errors in the log:

query.c:1326:tds_put_data_info putting status query.c:1420:tds_put_data: colsize = 9
query.c:1449:tds_put_data: not null param varint_size = 2
util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
odbc.c:2106:msgno 2402 20003
util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
util.c:389:tdserror: returning TDS_INT_CANCEL(2)
query.c:1326:tds_put_data_info putting status query.c:1420:tds_put_data: colsize = 9
query.c:1449:tds_put_data: not null param varint_size = 2
util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
odbc.c:2106:msgno 2402 20003
util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
util.c:389:tdserror: returning TDS_INT_CANCEL(2)
util.c:162:Changed query state from QUERYING to PENDING
net.c:768:Sending packet

The resulting sp_execute call is incorrectly formed, and the server
complains. 2402 is TDSEICONVO and indicates an invalid multibyte input sequence was
encountered while sending data to the server. I'm too tired to debug it further. I would guess the parameter data are
assumed to be in ISO-8859-1, not in the client's encoding. Regards,
--jkl
_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/freetds



------------------------------------------------------------------------

_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/freetds





Archive powered by MHonArc 2.6.24.

Top of Page