FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

Hi Sebastien, 

> It appears that a conversion failure occurs in tds_put_data, near line
> 1477:
> 
>     /* we need to convert data before */
>     /* TODO this can be a waste of memory... */
>     s = tds_convert_string(tds, curcol->char_conv, s, colsize,
>     &colsize); if (!s) {
>         /* FIXME this is a bad place to return error... */
>         /* TODO on memory failure we should compute converted size and
>         use chunks */ return TDS_FAIL;
>     }
>     converted = 1;
> 
> Occurs when string contains UTF-8 characters encoded with 3 bytes.
> 2-byte chars pass without problem...

I marked your message for later investigation.  

When you say 3-byte characters fail, do you mean that when you prefix the
string with 'N' in a simple SQL statement, the data are mangled on the
server?  That would imply a logic error using iconv.  

If we're talking about unprefixed strings, the problem could be on the
server (as we both know now).  

Or, are you talking about prepared statement parameters?  

I'm at a disadvantage here because I don't read any ideographic language
and don't have CJK fonts installed.  To work on this problem -- not
something I'm spending a lot of time on now in the height of summer -- we
need to put together a unit test using known code points.  For example, we
could use a known Japanese string, perhaps some famous phrase or haiku,
encoded as UTF-8 in a file.  Then the unit test could formulate an SQL
INSERT statement to write the text to a table, select the row, and write a
new file.  By comparing input to output, even an illiterate like me could
know whether or not it worked correctly. Then we could repeat the process
using a parameterized query, finishing up with an output parameter.  

If you will work up the unit test, I'll be able to at least help find any
bugs in the guts of the data transmission logic.  (Remember ODBC isn't my
game, but I do want parameterized queries to work, and I'm the fellow who
stuffed iconv into the middle of FreeTDS.) 

> When I print the string of curcol->column_data I get some strange
> characters at the end of the string... Hence UTF-8 characters should be
> displayed without prob by using fprintf, as long as LANG=en_US.utf8 ...
> no?

There are no null-terminated strings internally, and all character data in
TDS have a length prefix.  

Regards, 

--jkl