FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

2010/2/5 James K. Lowden <jklowden AT freetds.org>:
> Michael Mata wrote:
>>
>> Since we can't specify client charset in a DSN-less configuration, I
>> reverted to using freetds.conf and I'm still getting str returned
>> instead of unicode.
>
> I think I found a bug, but I don't think it's causing what you're seeing.
> It's only preventing me from helping you.
>
> I created a table as follows:
>
> bsqldb -q -S$S -U$U -P$(password $S $U) <<SQL
> create table test_utf8
>        ( id int IDENTITY
>        , name nchar(10) not NULL
>        , primary key (id)
>        )
> insert test_utf8 (name) values ('hallå')
> insert test_utf8 (name) values ('adjö')
> insert test_utf8 (name) values ('Århus')
> go
> SQL
>
> (I hope those are readable in your mail client.)
>
> When I fetch these rows with bsqlodbc, I provoke an error:
>
> $ printf "select * from test_utf8\ngo\n" | bsqlodbc -S$S -U$U -P$(password
> $S $U)
>        id  name
> ----------  ----------
> "[FreeTDS][SQL Server]Data truncated"
>         1  hall
> "[FreeTDS][SQL Server]Data truncated"
>         2  adj
> "[FreeTDS][SQL Server]Data truncated"
>         3  rhus
>
> Using bsqldb I don't get an error but I also don't get correct output: the
> non-ASCII characters are missing.  I don't know why yet.
>
> $ printf "select * from test_utf8\ngo\n" | TDSDUMP=bsqldb.dump bsqldb
> -S${S}utf8 -U$U -P$(password $S $U) |hexdump -C
>         id  name
> -----------  ----------------------------------------
> 3 rows affected
> 00000000  20 20 20 20 20 20 20 20  20 20 31 20 20 68 61 6c  |          1
> hal|
> 00000010  6c 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |l
>  |
> 00000020  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |
>  |
> 00000030  20 20 20 20 20 0a 20 20  20 20 20 20 20 20 20 20  |     .
>  |
> 00000040  32 20 20 61 64 6a 20 20  20 20 20 20 20 20 20 20  |2  adj
>  |
> 00000050  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |
>  |
> 00000060  20 20 20 20 20 20 20 20  20 20 20 0a 20 20 20 20  |           .
>  |
> 00000070  20 20 20 20 20 20 33 20  20 72 68 75 73 20 20 20  |      3  rhus
>  |
> 00000080  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |
>  |
> *
> 000000a0  20 0a                                             | .|
> 000000a2
>
> $ grep conversions bsqldb.dump
> iconv.c:364:setting up conversions for client charset "UTF-8"
>

Conversions are ok.... what is not ok is nchar or perhaps column len
returned by SQLDescribeCol. Why.....

Using nchar(10) "foo" get stored as "foo       " ("foo" followed by 7
spaces). Using utf-8 encoding a 5 character get stored in 10 nchar but
if a character is not ascii get encoded in more than 10 characters.
SQLDescribeCol return 10 characters and you provided 10 character
buffer which is insufficient to store original data. Now... On
SQLDescribeCol

 ColumnSizePtr

    [Output] Pointer to a buffer in which to return the size (in
characters) of the column on the data source.

so is fine to return 10 or we should return that maximum buffer space
(40 in this case) ?? Or perhaps we should return 0 (undeterminated) ?
....

freddy77