[freetds] Status of UTF-8 support in 0.83 dev

ZIGLIO, Frediano, VF-IT Frediano.Ziglio at vodafone.com
Tue Sep 9 04:26:02 EDT 2008

> Frediano,
> Sorry my YES was too fast: I should have checked with a 
> simple ODBC program.
> Even inserting does not fully work for now:
> When binding with a small ColumnSize (like 2), I get invalid 
> data in the
> SQL Server Profiler trace (only pieces of the value).
> Looks like the conversion from UTF-8 to UCS-2 relies on the ColumnSize
> and does not work properly when that size is small...
> (Maybe the conversion buffer used is allocated according to 
> ColumnSize,
> as a number of bytes?)

Bingo, yes, code in src/odbc/sql2tds.c.
Oh... I committed patch I sent you yesterday.

> See attached example: Using ColumnSize=2, it does not insert all data,
> but when using ColumnSize>=6 it works...

I updated utf8 test, see 
currently it doesn't work, it give 

$ ./utf8
CREATE TABLE mytab王鴻 (k int, c NCHAR(10), vc NVARCHAR(10))
INSERT INTO mytab王鴻 VALUES (1,N'aaa',N'aaa')
INSERT INTO mytab王鴻 VALUES (2,N'abcéáô',N'abcéîô')
INSERT INTO mytab王鴻 VALUES (3,N'abc王鴻',N'abc王鴻傑王鴻傑')
insert #1
insert #2
insert #3
insert #1
insert #2
utf8.c:106 SQLExecute
SQL error 42000 -- [FreeTDS][SQL Server]Character set conversion is not available between client character set '%.*s' and server character set '%.*s'

mmm... error is a bit wrong...

> Binding first value (4 bytes) with ColumnSize=4, it works, and second
> value (6 bytes) it works with ColumnSize=6...
> Since ColumnSize defines a number of CHARACTERS, something is 
> wrong here.
> Seb

I got it working replacing (sql2tds.c)

    } else {
        if (is_unicode_type(dest_type))
            curcol->on_server.column_size = curcol->column_size * 2;
            curcol->on_server.column_size = curcol->column_size;


    } else {
        if (is_unicode_type(dest_type)) {
            curcol->on_server.column_size = curcol->column_size * 2;
            curcol->column_size *= 4;
        } else
            curcol->on_server.column_size = curcol->column_size;

It merely allocate correct buffer... I think in this case it would be better to either
- avoid buffer allocation, fill column_data if possible... I think the main problem here is 
- use server encoding for buffer, I think this is related to bcp todo ("Native bcp has no iconv support; character bcp files are assumed be encoded with the client's charset.  More flexibility one both sides would be good.").
I think that column size == count in bytes is required cause:
- its possible to compute characters number from byte count and buffer but it's not easy from characters count and buffer to byte count (it require terminated strings)
- currently on_server.column_size is always a byte count
- memcpy/memmove it's easier if you have byte count
- client libraries provide byte count for strings and buffers

There are some cases where however having different encoding in libTDS buffer is useful
- having different client encoding
- avoiding converting characters twice (for instance if client use multibyte and wide at the same time as odbc does)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3129 bytes
Desc: not available
Url : http://lists.ibiblio.org/pipermail/freetds/attachments/20080909/56e51de7/attachment.bin 

More information about the FreeTDS mailing list