FreeTDS Development Group

[Sebastien FLAESCH <sf AT 4js.com>]

FYI:

We have tested with this configure option:

--disable-libiconv

and now it works...

Whatever iconv issue it is, I believe that FreeTDS should check iconv status
and stop with a fatal error if iconv is not able to handle charset 
conversions.

Character set conversion is critical, when subject of data loss or corruption
with no special warning for the user.

Seb

On 02/05/2013 03:22 PM, Sebastien FLAESCH wrote:
> It appears that compiling with libiconv does not help.
> Seb
>
>
> On 02/05/2013 12:46 PM, Sebastien FLAESCH wrote:
> > On 02/04/2013 06:57 PM, Frediano Ziglio wrote:
> > > 2013/2/4 Sebastien FLAESCH<sf AT 4js.com>:
> > > > Frediano,
> > > >
> > > > I believe it's related to the client charset conversions...
> > > >
> > > > I did following simple test:
> > > >
> > > > My database collation is CP1252, so to avoid iconv usage, in the ODBC
> > > > data source definition, I have now:
> > > >
> > > > ClientCharset = CP1252
> > > >
> > > > The program uses only pure ASCII chars, so this is ok.
> > > >
> > > > Now there is no more iconv error in the log, and the first call to
> > > > SQLGetData() returns 10000 bytes as expected...
> > > >
> > > > If fact for character data, ODBC returns the char pieces with an
> > > > additional \0 terminator.
> > > >
> > > > So, with a buffer of 4096 bytes:
> > > >
> > > > Call #1 to SQLGetData() returns len=10000 bytes (ok)
> > > > Call #2 to SQLGetData() returns len=5905 bytes
> > > >       there was 5905 bytes left to read, because only 4095 bytes could
> > > >       be written to the buffer, because of additional \0 terminator.
> > > >       10000 - 4095 = 5905 ...
> > > > Call #3 to SQLGetData() returns len=1810 bytes
> > > >        5905 - 4095 = 1810 ...
> > > > Last call: 1810 bytes fit in the 4096 buffer.
> > >
> > > Well, this is expected. But I don't understand what's going wrong with 
> > > Solaris.
> > > Could be possible that Solaris do not convert some characters and so
> > > it convert a byte characters to 0 bytes ??
> >
> > Possible...
> >
> > > Are you sure that there is only ASCII characters (0-127) ??
> >
> > Pure ASCII.
> >
> > > Why should Solaris report some warning converting ASCII ??
> >
> > My C locale is set to POSIX/C, I don't know why it ends as ISO-8859-1
> > for the iconv conversions in FreeTDS. Maybe it's the default.
> >
> > Is there some mapping between the C locale (LANG/LC_ALL) and the iconv
> > locale name (I know these can be different names, how is this mapped?)
> >
> > > A dump would be helpful. Send privately to me if it contains sensitive
> > > information.
> >
> > I send you a dump.
> >
> > > You could try using libiconv.
> >
> > We'll try this too.
> >
> > > > I hope this helps...
> > > >
> > > > Seb
> > >
> > > Frediano
> > >
> > >
> > > > On 02/03/2013 12:12 PM, Frediano Ziglio wrote:
> > > > > Hi,
> > > > >       where is SQL_DATA_AT_EXEC in your code?
> > > > >
> > > > > Which DM are you using? Is possible that for some reason DM call some
> > > > > additional function to do some checks. This could discovered enabling
> > > > > FreeTDS dump.
> > > > >
> > > > > Quire strange it works correctly on Linux but not on Solaris.
> > > > > Does data contains no-ASCII characters?
> > > > > How long are usually these TEXT fields?
> > > > > Which TDS protocol version are you using?
> > > > > There is a specific getdata test for this stuff and should work.
> > > > >
> > > > > Regards,
> > > > >       Frediano
> > > > >
> > > > > 2013/2/2 Sebastien FLAESCH<sf AT 4js.com>:
> > > > > > Hi Frediano,
> > > > > >
> > > > > > FYI, we use a single-byte encoding (LANG=POSIX) ...
> > > > > >
> > > > > > Here is the code we use, nothing particular, this is the same code we
> > > > > > use for different ODBC drivers...
> > > > > >
> > > > > > #define BLOB_BLOCK_SIZE 4096
> > > > > > ....
> > > > > > static SQLRETURN getBlobPieces(SqlStatement * st, SQLUSMALLINT fpos,
> > > > > >                                     SQLSMALLINT ctype, MyLocator * loc,
> > > > > >                                     FILE * out, SQLLEN * plen)
> > > > > > {
> > > > > >          SQLRETURN r = 0;
> > > > > >          char buf[BLOB_BLOCK_SIZE];
> > > > > >          SQLLEN len;
> > > > > >          int i = 0;
> > > > > >          while (1) {
> > > > > >              r = SQLGetData(st->stmtHandle, fpos, ctype,
> > > > > >                             (SQLPOINTER) buf, sizeof(buf),&len);
> > > > > >              if (!SQL_SUCCEEDED(r) || len == 0 || len == SQL_NULL_DATA)
> > > > > >                  break;
> > > > > >              if (len>     (SQLLEN) sizeof(buf) || len == SQL_NO_TOTAL)
> > > > > >                  len = sizeof(buf);
> > > > > >              /* Char data piece get \0, these must be removed! */
> > > > > >              if (ctype == SQL_C_CHAR) {
> > > > > >                  if (buf[len - 1] == '\0') {
> > > > > >                      len--;
> > > > > >                  }
> > > > > >              }
> > > > > >              fwrite(buf, 1, len, out);
> > > > > >          }
> > > > > >          *plen = len;
> > > > > >          return r;
> > > > > > }
> > > > > >
> > > > > > Am I missusing SQLGetData()? As you can see, I modify the len after
> > > > > > Does this affect the next call to SQLGetData()?
> > > > > >
> > > > > > Anyway, the INITIAL len returned by SQLGetData() is for sure:
> > > > > >
> > > > > >         (total-bytes - 1)
> > > > > >
> > > > > > When using UTF-8, I get SQL_NO_TOTAL in the first call, but in ASCII,
> > > > > > I get the (total size - 1)...
> > > > > >
> > > > > > Seb
> > > > > >
> > > > > > On 02/02/2013 08:52 AM, Frediano Ziglio wrote:
> > > > > > > Il giorno 31/gen/2013, alle ore 14:28, Sebastien FLAESCH<sf AT 4js.com>      ha 
> > > > > > > scritto:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Context:
> > > > > > > >
> > > > > > > > $ uname -a
> > > > > > > > SunOS s64x100 5.10 Generic_120012-14 i86pc i386 i86pc
> > > > > > > >
> > > > > > > > FreeTDS versions tested:
> > > > > > > > - freetds-0.92.dev.20120312
> > > > > > > > - freetds-dev.0.92.377
> > > > > > > >
> > > > > > > > When fetching a TEXT from the database (SQL Server 2005), using 
> > > > > > > > SQL_DATA_AT_EXEC,
> > > > > > > > it appears that the SQLGetData() function returns the size of the text data,
> > > > > > > > minus one...
> > > > > > > >
> > > > > > > > I have tested the same FreeTDS version on a Linux 32b, and SQLGetData() 
> > > > > > > > returns
> > > > > > > > the correct length as expected...
> > > > > > >
> > > > > > > Strange! How do you read the length?
> > > > > > >
> > > > > > > > I will continue to investigate, but I wanted to ask if this is a know issue.
> > > > > > > >
> > > > > > > > When connecting to SQL Server 2008 R2, this does not occur, because 
> > > > > > > > SQLGetData()
> > > > > > > > returns SQL_NO_TOTAL and we fetch text pieces in a different way...
> > > > > > >
> > > > > > > This is done to deal with different encodings.
> > > > > > >
> > > > > > > > Thanks
> > > > > > > > Seb
> > > > > > >
> > > > > > > Frediano
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > FreeTDS mailing list
> > > > > > > FreeTDS AT lists.ibiblio.org
> > > > > > > http://lists.ibiblio.org/mailman/listinfo/freetds
> > > > > >
> > > > > > _______________________________________________
> > > > > > FreeTDS mailing list
> > > > > > FreeTDS AT lists.ibiblio.org
> > > > > > http://lists.ibiblio.org/mailman/listinfo/freetds
> > > > > _______________________________________________
> > > > > FreeTDS mailing list
> > > > > FreeTDS AT lists.ibiblio.org
> > > > > http://lists.ibiblio.org/mailman/listinfo/freetds
> > > >
> > > > _______________________________________________
> > > > FreeTDS mailing list
> > > > FreeTDS AT lists.ibiblio.org
> > > > http://lists.ibiblio.org/mailman/listinfo/freetds
> > > _______________________________________________
> > > FreeTDS mailing list
> > > FreeTDS AT lists.ibiblio.org
> > > http://lists.ibiblio.org/mailman/listinfo/freetds
> >
> > _______________________________________________
> > FreeTDS mailing list
> > FreeTDS AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/freetds
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds