FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

2013/3/14 Sebastien FLAESCH <sf AT 4js.com>:
> Hi,
>
> I have already reported an issue regarding SQLGetData()/iconv, see mails 
> with title:
>
> "SQLGetData() returns TEXT length minus one on Solaris 64b with SQL Server 
> 2005"
>
> But now we have a similar problem on AIX 6.1, still with version 0.92.377.
>

Fixed recently on master branch (don't ask me the version). Commits were

http://gitorious.org/freetds/freetds/commit/f3bb4ca07490b3e1d32266223b571ef6da588c9c
and
http://gitorious.org/freetds/freetds/commit/a3b9c0307a62e2a6fb1a16ba5760081820cadbe5

> After inserting an empty TEXT, we fetch it back (see mail history for 
> details),
> and SQLGetData() returns a len of 4095 bytes, when zero is expected.
>
> This occurs if no ClientCharset is defined in the ODBC data source...
>
> When setting:
>
>     ClientCharset = en_US.8859-15
>
> or:
>
>     ClientCharset = en_US.UTF-8
>
> The len returned by SQLGetData() is zero, as expected.
>
> Some news about this?
>

I'll add a test for this case.

> As I suggested, if there is some iconv conversion issues, FreeTDS should 
> stop
> with a fatal error, to avoid any data loss / mis-conversion. It should not
> silently send or return invalid character data.
>
> Seb
>
>

Yes, you are right. Probably the better thinks is to make the
connection fails with a proper error.

Frediano

>
>
> On 02/06/2013 11:21 AM, Sebastien FLAESCH wrote:
>> FYI:
>>
>> We have tested with this configure option:
>>
>> --disable-libiconv
>>
>> and now it works...
>>
>> Whatever iconv issue it is, I believe that FreeTDS should check iconv
>> status
>> and stop with a fatal error if iconv is not able to handle charset
>> conversions.
>>
>> Character set conversion is critical, when subject of data loss or
>> corruption
>> with no special warning for the user.
>>
>> Seb
>>
>> On 02/05/2013 03:22 PM, Sebastien FLAESCH wrote:
>>> It appears that compiling with libiconv does not help.
>>> Seb
>>>
>>>
>>> On 02/05/2013 12:46 PM, Sebastien FLAESCH wrote:
>>>> On 02/04/2013 06:57 PM, Frediano Ziglio wrote:
>>>>> 2013/2/4 Sebastien FLAESCH<sf AT 4js.com>:
>>>>>> Frediano,
>>>>>>
>>>>>> I believe it's related to the client charset conversions...
>>>>>>
>>>>>> I did following simple test:
>>>>>>
>>>>>> My database collation is CP1252, so to avoid iconv usage, in the ODBC
>>>>>> data source definition, I have now:
>>>>>>
>>>>>> ClientCharset = CP1252
>>>>>>
>>>>>> The program uses only pure ASCII chars, so this is ok.
>>>>>>
>>>>>> Now there is no more iconv error in the log, and the first call to
>>>>>> SQLGetData() returns 10000 bytes as expected...
>>>>>>
>>>>>> If fact for character data, ODBC returns the char pieces with an
>>>>>> additional \0 terminator.
>>>>>>
>>>>>> So, with a buffer of 4096 bytes:
>>>>>>
>>>>>> Call #1 to SQLGetData() returns len=10000 bytes (ok)
>>>>>> Call #2 to SQLGetData() returns len=5905 bytes
>>>>>> there was 5905 bytes left to read, because only 4095 bytes could
>>>>>> be written to the buffer, because of additional \0 terminator.
>>>>>> 10000 - 4095 = 5905 ...
>>>>>> Call #3 to SQLGetData() returns len=1810 bytes
>>>>>> 5905 - 4095 = 1810 ...
>>>>>> Last call: 1810 bytes fit in the 4096 buffer.
>>>>>>
>>>>>
>>>>> Well, this is expected. But I don't understand what's going wrong
>>>>> with Solaris.
>>>>> Could be possible that Solaris do not convert some characters and so
>>>>> it convert a byte characters to 0 bytes ??
>>>>
>>>> Possible...
>>>>
>>>>> Are you sure that there is only ASCII characters (0-127) ??
>>>>
>>>> Pure ASCII.
>>>>
>>>>> Why should Solaris report some warning converting ASCII ??
>>>>
>>>> My C locale is set to POSIX/C, I don't know why it ends as ISO-8859-1
>>>> for the iconv conversions in FreeTDS. Maybe it's the default.
>>>>
>>>> Is there some mapping between the C locale (LANG/LC_ALL) and the iconv
>>>> locale name (I know these can be different names, how is this mapped?)
>>>>
>>>>> A dump would be helpful. Send privately to me if it contains sensitive
>>>>> information.
>>>>
>>>> I send you a dump.
>>>>
>>>>> You could try using libiconv.
>>>>
>>>> We'll try this too.
>>>>
>>>>>> I hope this helps...
>>>>>>
>>>>>> Seb
>>>>>>
>>>>>
>>>>> Frediano
>>>>>
>>>>>
>>>>>>
>>>>>> On 02/03/2013 12:12 PM, Frediano Ziglio wrote:
>>>>>>> Hi,
>>>>>>> where is SQL_DATA_AT_EXEC in your code?
>>>>>>>
>>>>>>> Which DM are you using? Is possible that for some reason DM call some
>>>>>>> additional function to do some checks. This could discovered enabling
>>>>>>> FreeTDS dump.
>>>>>>>
>>>>>>> Quire strange it works correctly on Linux but not on Solaris.
>>>>>>> Does data contains no-ASCII characters?
>>>>>>> How long are usually these TEXT fields?
>>>>>>> Which TDS protocol version are you using?
>>>>>>> There is a specific getdata test for this stuff and should work.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Frediano
>>>>>>>
>>>>>>> 2013/2/2 Sebastien FLAESCH<sf AT 4js.com>:
>>>>>>>> Hi Frediano,
>>>>>>>>
>>>>>>>> FYI, we use a single-byte encoding (LANG=POSIX) ...
>>>>>>>>
>>>>>>>> Here is the code we use, nothing particular, this is the same
>>>>>>>> code we
>>>>>>>> use for different ODBC drivers...
>>>>>>>>
>>>>>>>> #define BLOB_BLOCK_SIZE 4096
>>>>>>>> ....
>>>>>>>> static SQLRETURN getBlobPieces(SqlStatement * st, SQLUSMALLINT fpos,
>>>>>>>> SQLSMALLINT ctype, MyLocator * loc,
>>>>>>>> FILE * out, SQLLEN * plen)
>>>>>>>> {
>>>>>>>> SQLRETURN r = 0;
>>>>>>>> char buf[BLOB_BLOCK_SIZE];
>>>>>>>> SQLLEN len;
>>>>>>>> int i = 0;
>>>>>>>> while (1) {
>>>>>>>> r = SQLGetData(st->stmtHandle, fpos, ctype,
>>>>>>>> (SQLPOINTER) buf, sizeof(buf),&len);
>>>>>>>> if (!SQL_SUCCEEDED(r) || len == 0 || len == SQL_NULL_DATA)
>>>>>>>> break;
>>>>>>>> if (len> (SQLLEN) sizeof(buf) || len == SQL_NO_TOTAL)
>>>>>>>> len = sizeof(buf);
>>>>>>>> /* Char data piece get \0, these must be removed! */
>>>>>>>> if (ctype == SQL_C_CHAR) {
>>>>>>>> if (buf[len - 1] == '\0') {
>>>>>>>> len--;
>>>>>>>> }
>>>>>>>> }
>>>>>>>> fwrite(buf, 1, len, out);
>>>>>>>> }
>>>>>>>> *plen = len;
>>>>>>>> return r;
>>>>>>>> }
>>>>>>>>
>>>>>>>> Am I missusing SQLGetData()? As you can see, I modify the len after
>>>>>>>> Does this affect the next call to SQLGetData()?
>>>>>>>>
>>>>>>>> Anyway, the INITIAL len returned by SQLGetData() is for sure:
>>>>>>>>
>>>>>>>> (total-bytes - 1)
>>>>>>>>
>>>>>>>> When using UTF-8, I get SQL_NO_TOTAL in the first call, but in
>>>>>>>> ASCII,
>>>>>>>> I get the (total size - 1)...
>>>>>>>>
>>>>>>>> Seb
>>>>>>>>
>>>>>>>> On 02/02/2013 08:52 AM, Frediano Ziglio wrote:
>>>>>>>>> Il giorno 31/gen/2013, alle ore 14:28, Sebastien
>>>>>>>>> FLAESCH<sf AT 4js.com> ha scritto:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Context:
>>>>>>>>>>
>>>>>>>>>> $ uname -a
>>>>>>>>>> SunOS s64x100 5.10 Generic_120012-14 i86pc i386 i86pc
>>>>>>>>>>
>>>>>>>>>> FreeTDS versions tested:
>>>>>>>>>> - freetds-0.92.dev.20120312
>>>>>>>>>> - freetds-dev.0.92.377
>>>>>>>>>>
>>>>>>>>>> When fetching a TEXT from the database (SQL Server 2005), using
>>>>>>>>>> SQL_DATA_AT_EXEC,
>>>>>>>>>> it appears that the SQLGetData() function returns the size of
>>>>>>>>>> the text data,
>>>>>>>>>> minus one...
>>>>>>>>>>
>>>>>>>>>> I have tested the same FreeTDS version on a Linux 32b, and
>>>>>>>>>> SQLGetData() returns
>>>>>>>>>> the correct length as expected...
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Strange! How do you read the length?
>>>>>>>>>
>>>>>>>>>> I will continue to investigate, but I wanted to ask if this is
>>>>>>>>>> a know issue.
>>>>>>>>>>
>>>>>>>>>> When connecting to SQL Server 2008 R2, this does not occur,
>>>>>>>>>> because SQLGetData()
>>>>>>>>>> returns SQL_NO_TOTAL and we fetch text pieces in a different
>>>>>>>>>> way...
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is done to deal with different encodings.
>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Seb
>>>>>>>>>
>>>>>>>>> Frediano
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> FreeTDS mailing list
>>>>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> FreeTDS mailing list
>>>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>>> _______________________________________________
>>>>>>> FreeTDS mailing list
>>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeTDS mailing list
>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>> _______________________________________________
>>>>> FreeTDS mailing list
>>>>> FreeTDS AT lists.ibiblio.org
>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>
>>>>
>>>> _______________________________________________
>>>> FreeTDS mailing list
>>>> FreeTDS AT lists.ibiblio.org
>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>
>>>
>>> _______________________________________________
>>> FreeTDS mailing list
>>> FreeTDS AT lists.ibiblio.org
>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>
>>
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds