FreeTDS Development Group

[Sebastien FLAESCH <sf AT 4js.com>]

It appears that a conversion failure occurs in tds_put_data, near line 1477:

   /* we need to convert data before */
   /* TODO this can be a waste of memory... */
   s = tds_convert_string(tds, curcol->char_conv, s, colsize, &colsize);
   if (!s) {
       /* FIXME this is a bad place to return error... */
       /* TODO on memory failure we should compute converted size and use 
chunks */
       return TDS_FAIL;
   }
   converted = 1;

Occurs when string contains UTF-8 characters encoded with 3 bytes.
2-byte chars pass without problem...

When I print the string of curcol->column_data I get some strange characters 
at
the end of the string... Hence UTF-8 characters should be displayed without 
prob
by using fprintf, as long as LANG=en_US.utf8 ... no?

I will double check the UTF-8 string I put in my buffers, as well as the size
passed to SQLBindParameter, but when I print the string in the ODBC program,
it displays without additional characters at the end...

In my ODBC program:

   UTF-8 = [abcéô鴻]  ( size= 10 )

In tds.c code:

   UTF-8 = [abcéô鴻01000]  ( size= 10 )

...

Seb

Sebastien FLAESCH wrote:
> Thank you James for looking at this!
>
> UTF-8 support is critical for one of of customers in Asia...
>
> I have identified a first issue: I must of course use N'xxx' string
> constants when inserting UNICODE/UTF-8 data in the database.
> In previous sample program, the string constants where identified as
> non-Unicode strings in the current database locale. Sorry...
>
> I have added a SELECT to fetch inserted rows, and it seems to be fine
> so far. I have checked the values in the database with SQL Server's
> query tool and the characters are correct.
>
> Still the insert statement using UTF-8 data fails with:
>
> Error -1 at: SQLExecute 1.2 StmtH
> Diagnostic info:
>   SQL State: 42000
>   SQL code : 8005
>   Message  : [FreeTDS][SQL Server]The incoming tabular data stream
>              (TDS) remote procedure call (RPC) protocol stream is
>              incorrect. Parameter 4: The parameter name is invalid.
>
>
> Note that the first insert with latin characters "abcéÓÎ" / "abcÛÊé"
> (2 bytes long) works, it's the second insert using Asian characters
> (3 bytes long)...
>
>
> FYI I have rebuild freetds with iconv 1.12 now:
>
> [root@fox ~]# ldd -r /opt3/dbs/tds/0.83/lib/libtdsodbc.so
>         libiconv.so.2 => /usr/local/lib/libiconv.so.2 (0x00111000)
>         librt.so.1 => /lib/tls/librt.so.1 (0x00a7b000)
>         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00957000)
>         libc.so.6 => /lib/tls/libc.so.6 (0x001ed000)
>         /lib/ld-linux.so.2 (0x00739000)
>
> Thanks,
> Seb
>
>
> James K. Lowden wrote:
> > Sebastien FLAESCH wrote:
> > > Log shows now:
> > >
> > > log.c:190:Starting log file for FreeTDS 0.83.dev.20080812
> > >          on 2008-08-12 18:11:48 with debug flags 0x4fff.
> > > iconv.c:197:names for ISO-8859-1: ISO-8859-1
> > > iconv.c:197:names for UTF-8: UTF-8
> > > iconv.c:197:names for UCS-2LE: UCS-2LE
> > > iconv.c:197:names for UCS-2BE: UCS-2BE
> > > iconv.c:363:iconv to convert client-side data to the "UTF-8" character
> > > set iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
> > > iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
> >
> > Hi Sebastien,
> > I modified the log messages from iconv.c to make them clearer.  HTH.
> > > But still, I get:
> > >
> > >
> > > Error -1 at: SQLExecute 1.2 StmtH
> > > Diagnostic info:
> > >    SQL State: 42000
> > >    SQL code : 8005
> > >    Message  : [FreeTDS][SQL Server]The incoming tabular data stream
> > >    (TDS) remote procedure call (RPC) protocol stream is incorrect.
> > >    Parameter 4: The parameter name is invalid.
> >
> > This happens because of these errors in the log:
> >
> > > query.c:1326:tds_put_data_info putting status 
> > > query.c:1420:tds_put_data: colsize = 9
> > > query.c:1449:tds_put_data: not null param varint_size = 2
> > > util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
> > > odbc.c:2106:msgno 2402 20003
> > > util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
> > > util.c:389:tdserror: returning TDS_INT_CANCEL(2)
> > > query.c:1326:tds_put_data_info putting status 
> > > query.c:1420:tds_put_data: colsize = 9
> > > query.c:1449:tds_put_data: not null param varint_size = 2
> > > util.c:334:tdserror(0x98cb050, 0x98cc3d8, 2402, 0)
> > > odbc.c:2106:msgno 2402 20003
> > > util.c:368:tdserror: client library returned TDS_INT_CANCEL(2)
> > > util.c:389:tdserror: returning TDS_INT_CANCEL(2)
> > > util.c:162:Changed query state from QUERYING to PENDING
> > > net.c:768:Sending packet
> >
> > The resulting sp_execute call is incorrectly formed, and the server
> > complains. 
> > 2402 is TDSEICONVO and indicates an invalid multibyte input sequence was
> > encountered while sending data to the server. 
> > I'm too tired to debug it further.  I would guess the parameter data are
> > assumed to be in ISO-8859-1, not in the client's encoding. 
> > Regards,
> > --jkl
> > _______________________________________________
> > FreeTDS mailing list
> > FreeTDS AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/freetds
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds