[freetds] Status of UTF-8 support in 0.83 dev
Sebastien FLAESCH
sf at 4js.com
Thu Aug 14 11:05:17 EDT 2008
James K. Lowden wrote:
> Hi Sebastien,
>
>> It appears that a conversion failure occurs in tds_put_data, near line
>> 1477:
>>
>> /* we need to convert data before */
>> /* TODO this can be a waste of memory... */
>> s = tds_convert_string(tds, curcol->char_conv, s, colsize,
>> &colsize); if (!s) {
>> /* FIXME this is a bad place to return error... */
>> /* TODO on memory failure we should compute converted size and
>> use chunks */ return TDS_FAIL;
>> }
>> converted = 1;
>>
>> Occurs when string contains UTF-8 characters encoded with 3 bytes.
>> 2-byte chars pass without problem...
>
> I marked your message for later investigation.
>
> When you say 3-byte characters fail, do you mean that when you prefix the
> string with 'N' in a simple SQL statement, the data are mangled on the
> server? That would imply a logic error using iconv.
>
> If we're talking about unprefixed strings, the problem could be on the
> server (as we both know now).
>
> Or, are you talking about prepared statement parameters?
I talk about parameterized queries, using SQLBindParameter() to pass UTF-8
strings... String literals with/without N prefix work fine for me.
See attachments...
>
> I'm at a disadvantage here because I don't read any ideographic language
> and don't have CJK fonts installed. To work on this problem -- not
> something I'm spending a lot of time on now in the height of summer -- we
> need to put together a unit test using known code points. For example, we
> could use a known Japanese string, perhaps some famous phrase or haiku,
> encoded as UTF-8 in a file. Then the unit test could formulate an SQL
> INSERT statement to write the text to a table, select the row, and write a
> new file. By comparing input to output, even an illiterate like me could
> know whether or not it worked correctly. Then we could repeat the process
> using a parameterized query, finishing up with an output parameter.
>
> If you will work up the unit test, I'll be able to at least help find any
> bugs in the guts of the data transmission logic. (Remember ODBC isn't my
> game, but I do want parameterized queries to work, and I'm the fellow who
> stuffed iconv into the middle of FreeTDS.)
Yes I can help such ODBC test, but will be off next 2 weeks...
>
>> When I print the string of curcol->column_data I get some strange
>> characters at the end of the string... Hence UTF-8 characters should be
>> displayed without prob by using fprintf, as long as LANG=en_US.utf8 ...
>> no?
>
> There are no null-terminated strings internally, and all character data in
> TDS have a length prefix.
Yep that's was my conclusion too.
>
> Regards,
>
> --jkl
> _______________________________________________
> FreeTDS mailing list
> FreeTDS at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds
>
More information about the FreeTDS
mailing list