FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

2013/2/6 Sebastien FLAESCH <sf AT 4js.com>:
> FYI:
>
> We have tested with this configure option:
>
> --disable-libiconv
>
> and now it works...
>
> Whatever iconv issue it is, I believe that FreeTDS should check iconv status
> and stop with a fatal error if iconv is not able to handle charset 
> conversions.
>
> Character set conversion is critical, when subject of data loss or 
> corruption
> with no special warning for the user.
>
> Seb
>

Mmm... not that good. It means that Solaris iconv does something that
is not expected by FreeTDS. Probably FreeTDS expect a 1 to 1
conversion while lib iconv provided by Solaris does not respect this.
I should write a test if libiconv linked in is FreeTDS-aware. Probably
using GNU libiconv would fix your issues. However is hard to reproduce
using a libiconv that works as expected.

Frediano

> On 02/05/2013 03:22 PM, Sebastien FLAESCH wrote:
>> It appears that compiling with libiconv does not help.
>> Seb
>>
>>
>> On 02/05/2013 12:46 PM, Sebastien FLAESCH wrote:
>>> On 02/04/2013 06:57 PM, Frediano Ziglio wrote:
>>>> 2013/2/4 Sebastien FLAESCH<sf AT 4js.com>:
>>>>> Frediano,
>>>>>
>>>>> I believe it's related to the client charset conversions...
>>>>>
>>>>> I did following simple test:
>>>>>
>>>>> My database collation is CP1252, so to avoid iconv usage, in the ODBC
>>>>> data source definition, I have now:
>>>>>
>>>>> ClientCharset = CP1252
>>>>>
>>>>> The program uses only pure ASCII chars, so this is ok.
>>>>>
>>>>> Now there is no more iconv error in the log, and the first call to
>>>>> SQLGetData() returns 10000 bytes as expected...
>>>>>
>>>>> If fact for character data, ODBC returns the char pieces with an
>>>>> additional \0 terminator.
>>>>>
>>>>> So, with a buffer of 4096 bytes:
>>>>>
>>>>> Call #1 to SQLGetData() returns len=10000 bytes (ok)
>>>>> Call #2 to SQLGetData() returns len=5905 bytes
>>>>>       there was 5905 bytes left to read, because only 4095 bytes could
>>>>>       be written to the buffer, because of additional \0 terminator.
>>>>>       10000 - 4095 = 5905 ...
>>>>> Call #3 to SQLGetData() returns len=1810 bytes
>>>>>        5905 - 4095 = 1810 ...
>>>>> Last call: 1810 bytes fit in the 4096 buffer.
>>>>>
>>>>
>>>> Well, this is expected. But I don't understand what's going wrong with 
>>>> Solaris.
>>>> Could be possible that Solaris do not convert some characters and so
>>>> it convert a byte characters to 0 bytes ??
>>>
>>> Possible...
>>>
>>>> Are you sure that there is only ASCII characters (0-127) ??
>>>
>>> Pure ASCII.
>>>
>>>> Why should Solaris report some warning converting ASCII ??
>>>
>>> My C locale is set to POSIX/C, I don't know why it ends as ISO-8859-1
>>> for the iconv conversions in FreeTDS. Maybe it's the default.
>>>
>>> Is there some mapping between the C locale (LANG/LC_ALL) and the iconv
>>> locale name (I know these can be different names, how is this mapped?)
>>>
>>>> A dump would be helpful. Send privately to me if it contains sensitive
>>>> information.
>>>
>>> I send you a dump.
>>>
>>>> You could try using libiconv.
>>>
>>> We'll try this too.
>>>
>>>>> I hope this helps...
>>>>>
>>>>> Seb
>>>>>
>>>>
>>>> Frediano
>>>>
>>>>
>>>>>
>>>>> On 02/03/2013 12:12 PM, Frediano Ziglio wrote:
>>>>>> Hi,
>>>>>>       where is SQL_DATA_AT_EXEC in your code?
>>>>>>
>>>>>> Which DM are you using? Is possible that for some reason DM call some
>>>>>> additional function to do some checks. This could discovered enabling
>>>>>> FreeTDS dump.
>>>>>>
>>>>>> Quire strange it works correctly on Linux but not on Solaris.
>>>>>> Does data contains no-ASCII characters?
>>>>>> How long are usually these TEXT fields?
>>>>>> Which TDS protocol version are you using?
>>>>>> There is a specific getdata test for this stuff and should work.
>>>>>>
>>>>>> Regards,
>>>>>>       Frediano
>>>>>>
>>>>>> 2013/2/2 Sebastien FLAESCH<sf AT 4js.com>:
>>>>>>> Hi Frediano,
>>>>>>>
>>>>>>> FYI, we use a single-byte encoding (LANG=POSIX) ...
>>>>>>>
>>>>>>> Here is the code we use, nothing particular, this is the same code we
>>>>>>> use for different ODBC drivers...
>>>>>>>
>>>>>>> #define BLOB_BLOCK_SIZE 4096
>>>>>>> ....
>>>>>>> static SQLRETURN getBlobPieces(SqlStatement * st, SQLUSMALLINT fpos,
>>>>>>>                                     SQLSMALLINT ctype, MyLocator * 
>>>>>>> loc,
>>>>>>>                                     FILE * out, SQLLEN * plen)
>>>>>>> {
>>>>>>>          SQLRETURN r = 0;
>>>>>>>          char buf[BLOB_BLOCK_SIZE];
>>>>>>>          SQLLEN len;
>>>>>>>          int i = 0;
>>>>>>>          while (1) {
>>>>>>>              r = SQLGetData(st->stmtHandle, fpos, ctype,
>>>>>>>                             (SQLPOINTER) buf, sizeof(buf),&len);
>>>>>>>              if (!SQL_SUCCEEDED(r) || len == 0 || len == 
>>>>>>> SQL_NULL_DATA)
>>>>>>>                  break;
>>>>>>>              if (len>     (SQLLEN) sizeof(buf) || len == SQL_NO_TOTAL)
>>>>>>>                  len = sizeof(buf);
>>>>>>>              /* Char data piece get \0, these must be removed! */
>>>>>>>              if (ctype == SQL_C_CHAR) {
>>>>>>>                  if (buf[len - 1] == '\0') {
>>>>>>>                      len--;
>>>>>>>                  }
>>>>>>>              }
>>>>>>>              fwrite(buf, 1, len, out);
>>>>>>>          }
>>>>>>>          *plen = len;
>>>>>>>          return r;
>>>>>>> }
>>>>>>>
>>>>>>> Am I missusing SQLGetData()? As you can see, I modify the len after
>>>>>>> Does this affect the next call to SQLGetData()?
>>>>>>>
>>>>>>> Anyway, the INITIAL len returned by SQLGetData() is for sure:
>>>>>>>
>>>>>>>         (total-bytes - 1)
>>>>>>>
>>>>>>> When using UTF-8, I get SQL_NO_TOTAL in the first call, but in ASCII,
>>>>>>> I get the (total size - 1)...
>>>>>>>
>>>>>>> Seb
>>>>>>>
>>>>>>> On 02/02/2013 08:52 AM, Frediano Ziglio wrote:
>>>>>>>> Il giorno 31/gen/2013, alle ore 14:28, Sebastien FLAESCH<sf AT 4js.com> 
>>>>>>>>      ha scritto:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> Context:
>>>>>>>>>
>>>>>>>>> $ uname -a
>>>>>>>>> SunOS s64x100 5.10 Generic_120012-14 i86pc i386 i86pc
>>>>>>>>>
>>>>>>>>> FreeTDS versions tested:
>>>>>>>>> - freetds-0.92.dev.20120312
>>>>>>>>> - freetds-dev.0.92.377
>>>>>>>>>
>>>>>>>>> When fetching a TEXT from the database (SQL Server 2005), using 
>>>>>>>>> SQL_DATA_AT_EXEC,
>>>>>>>>> it appears that the SQLGetData() function returns the size of the 
>>>>>>>>> text data,
>>>>>>>>> minus one...
>>>>>>>>>
>>>>>>>>> I have tested the same FreeTDS version on a Linux 32b, and 
>>>>>>>>> SQLGetData() returns
>>>>>>>>> the correct length as expected...
>>>>>>>>>
>>>>>>>>
>>>>>>>> Strange! How do you read the length?
>>>>>>>>
>>>>>>>>> I will continue to investigate, but I wanted to ask if this is a 
>>>>>>>>> know issue.
>>>>>>>>>
>>>>>>>>> When connecting to SQL Server 2008 R2, this does not occur, because 
>>>>>>>>> SQLGetData()
>>>>>>>>> returns SQL_NO_TOTAL and we fetch text pieces in a different way...
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is done to deal with different encodings.
>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Seb
>>>>>>>>
>>>>>>>> Frediano
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> FreeTDS mailing list
>>>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeTDS mailing list
>>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>> _______________________________________________
>>>>>> FreeTDS mailing list
>>>>>> FreeTDS AT lists.ibiblio.org
>>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> FreeTDS mailing list
>>>>> FreeTDS AT lists.ibiblio.org
>>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>> _______________________________________________
>>>> FreeTDS mailing list
>>>> FreeTDS AT lists.ibiblio.org
>>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>>
>>>
>>> _______________________________________________
>>> FreeTDS mailing list
>>> FreeTDS AT lists.ibiblio.org
>>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>>
>>
>> _______________________________________________
>> FreeTDS mailing list
>> FreeTDS AT lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/freetds
>>
>
> _______________________________________________
> FreeTDS mailing list
> FreeTDS AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/freetds