FreeTDS Development Group

["ZIGLIO, Frediano, VF-IT" <Frediano.Ziglio AT vodafone.com>]

> 
> On Wed, 28 Jan 2004 00:32:43 -0800, Will Lowe <harpo AT thebackrow.net>
> wrote:
> > I'm using FreeTDS 0.62.1 and DBD::Sybase 1.02 on a box 
> running Debian
> > unstable (sid) Linux trying to talk to M$ SQL 2k with 
> whatever patches
> > the Windows Critical Update thing thinks are current. 
> 
> That all sounds good.   
> 

Linux should have a sufficient iconv...

> > I have some a nvarchar columns on the SQL Server side and 
> I'd like to
> > read them in a perl application on the Linux box.  After reading the
> > stuff at http://www.freetds.org/userguide/aboutunicode.htm, I set
> > "client charset = UTF-8" and "tds version = 8.0" in freetds.conf
> > ... but I'm getting a lot of messages like this:
> > 
> > Message String: WARNING! Some character(s) could not be 
> converted into
> > client's character set. Unconverted bytes were changed to question
> > marks ('?').
> 
>       :-( 
> 

This mean that server send some characters that cannot be converted to
UTF-8. This can be caused inserting invalid data into varchar field (not
nvarchar). For example if you insert a 0x80 character in a iso8859-1
field (Latin1, default windows installation). Server insert data just
like you send but FreeTDS say you are doing something wrong... MS
updated copdepage to iso8859-15 but forgot to update it's server...

> > I'm not sure how this can be, since I'm pretty sure UTF-8 should be
> > able to represent any character that can be represented in UCS-2.
> 
> Absolutely correct.  
> 
> > iconv to convert client-side data to the "UTF-8" character set
> > 23:32:00.242427 tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
> > 23:32:00.242922 tds_iconv_info_init: converting 
> "ISO-8859-1"->"UCS-2LE"
> > 
> > Does that last bit mean that FreeTDS is converting data from the
> > server to ISO-8859-1 instead of UTF-8?  
> 
> No, that's normal.  FreeTDS has to be able to convert from 
> something like
> ASCII (of which ISO 8859-1 is a superset) to UCS-2 because 
> FreeTDS's own
> text is ASCII-encoded.  In managing the session with the server, the
> library needs to issue queries "under the hood", and of 
> course with TDS
> 7.0 those queries are encoded as UCS-2.  
> 
> What you're doing should work, afaict.  Could you post a 
> TDSDUMP log of a
> session with one of these errors?  Also, it might help if you 
> could send
> as a binary attachment the data you're selecting.  In theory, I could
> install them on my test server and reproduce your pr.  
> 
> BTW, as you're using UTF-8, you'll want to use the patch I 
> posted here for
> src/tds/write.c.  You can apply the patch, or fetch write.c revision
> 1.57.2.1 from CVS.  It affects queries larger than 128 characters.  
> 

I'm not sure 100% what I wrote... dump will help to discover the
truth...

freddy77