FreeTDS Development Group

[Sebastien FLAESCH <sf AT 4js.com>]

James K. Lowden wrote:
> Hi Sebastien, 
>
> > It appears that a conversion failure occurs in tds_put_data, near line
> > 1477:
> >
> >     /* we need to convert data before */
> >     /* TODO this can be a waste of memory... */
> >     s = tds_convert_string(tds, curcol->char_conv, s, colsize,
> >     &colsize); if (!s) {
> >         /* FIXME this is a bad place to return error... */
> >         /* TODO on memory failure we should compute converted size and
> >         use chunks */ return TDS_FAIL;
> >     }
> >     converted = 1;
> >
> > Occurs when string contains UTF-8 characters encoded with 3 bytes.
> > 2-byte chars pass without problem...
>
> I marked your message for later investigation.  
>
> When you say 3-byte characters fail, do you mean that when you prefix the
> string with 'N' in a simple SQL statement, the data are mangled on the
> server?  That would imply a logic error using iconv.  
>
> If we're talking about unprefixed strings, the problem could be on the
> server (as we both know now).  
>
> Or, are you talking about prepared statement parameters?

I talk about parameterized queries, using SQLBindParameter() to pass UTF-8
strings... String literals with/without N prefix work fine for me.

See attachments...


> I'm at a disadvantage here because I don't read any ideographic language
> and don't have CJK fonts installed.  To work on this problem -- not
> something I'm spending a lot of time on now in the height of summer -- we
> need to put together a unit test using known code points.  For example, we
> could use a known Japanese string, perhaps some famous phrase or haiku,
> encoded as UTF-8 in a file.  Then the unit test could formulate an SQL
> INSERT statement to write the text to a table, select the row, and write a
> new file.  By comparing input to output, even an illiterate like me could
> know whether or not it worked correctly. Then we could repeat the process
> using a parameterized query, finishing up with an output parameter.  
>
> If you will work up the unit test, I'll be able to at least help find any
> bugs in the guts of the data transmission logic.  (Remember ODBC isn't my
> game, but I do want parameterized queries to work, and I'm the fellow who
> stuffed iconv into the middle of FreeTDS.) 

Yes I can help such ODBC test, but will be off next 2 weeks...

> > When I print the string of curcol->column_data I get some strange
> > characters at the end of the string... Hence UTF-8 characters should be
> > displayed without prob by using fprintf, as long as LANG=en_US.utf8 ...
> > no?
>
> There are no null-terminated strings internally, and all character data in
> TDS have a length prefix.  

Yep that's was my conclusion too.

> Regards, 
>
> --jkl
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds