Skip to Content.
Sympa Menu

freetds - RE: [freetds] reading nvarchar columns in UTF-8

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "ZIGLIO, Frediano, VF-IT" <Frediano.Ziglio AT vodafone.com>
  • To: "FreeTDS Development Group" <freetds AT lists.ibiblio.org>
  • Subject: RE: [freetds] reading nvarchar columns in UTF-8
  • Date: Wed, 28 Jan 2004 15:48:04 +0100

>
> On Wed, 28 Jan 2004 00:32:43 -0800, Will Lowe <harpo AT thebackrow.net>
> wrote:
> > I'm using FreeTDS 0.62.1 and DBD::Sybase 1.02 on a box
> running Debian
> > unstable (sid) Linux trying to talk to M$ SQL 2k with
> whatever patches
> > the Windows Critical Update thing thinks are current.
>
> That all sounds good.
>

Linux should have a sufficient iconv...

> > I have some a nvarchar columns on the SQL Server side and
> I'd like to
> > read them in a perl application on the Linux box. After reading the
> > stuff at http://www.freetds.org/userguide/aboutunicode.htm, I set
> > "client charset = UTF-8" and "tds version = 8.0" in freetds.conf
> > ... but I'm getting a lot of messages like this:
> >
> > Message String: WARNING! Some character(s) could not be
> converted into
> > client's character set. Unconverted bytes were changed to question
> > marks ('?').
>
> :-(
>

This mean that server send some characters that cannot be converted to
UTF-8. This can be caused inserting invalid data into varchar field (not
nvarchar). For example if you insert a 0x80 character in a iso8859-1
field (Latin1, default windows installation). Server insert data just
like you send but FreeTDS say you are doing something wrong... MS
updated copdepage to iso8859-15 but forgot to update it's server...

> > I'm not sure how this can be, since I'm pretty sure UTF-8 should be
> > able to represent any character that can be represented in UCS-2.
>
> Absolutely correct.
>
> > iconv to convert client-side data to the "UTF-8" character set
> > 23:32:00.242427 tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
> > 23:32:00.242922 tds_iconv_info_init: converting
> "ISO-8859-1"->"UCS-2LE"
> >
> > Does that last bit mean that FreeTDS is converting data from the
> > server to ISO-8859-1 instead of UTF-8?
>
> No, that's normal. FreeTDS has to be able to convert from
> something like
> ASCII (of which ISO 8859-1 is a superset) to UCS-2 because
> FreeTDS's own
> text is ASCII-encoded. In managing the session with the server, the
> library needs to issue queries "under the hood", and of
> course with TDS
> 7.0 those queries are encoded as UCS-2.
>
> What you're doing should work, afaict. Could you post a
> TDSDUMP log of a
> session with one of these errors? Also, it might help if you
> could send
> as a binary attachment the data you're selecting. In theory, I could
> install them on my test server and reproduce your pr.
>
> BTW, as you're using UTF-8, you'll want to use the patch I
> posted here for
> src/tds/write.c. You can apply the patch, or fetch write.c revision
> 1.57.2.1 from CVS. It affects queries larger than 128 characters.
>

I'm not sure 100% what I wrote... dump will help to discover the
truth...

freddy77




Archive powered by MHonArc 2.6.24.

Top of Page