[freetds] reading nvarchar columns in UTF-8
ZIGLIO, Frediano, VF-IT
Frediano.Ziglio at vodafone.com
Wed Jan 28 09:48:04 EST 2004
> On Wed, 28 Jan 2004 00:32:43 -0800, Will Lowe <harpo at thebackrow.net>
> > I'm using FreeTDS 0.62.1 and DBD::Sybase 1.02 on a box
> running Debian
> > unstable (sid) Linux trying to talk to M$ SQL 2k with
> whatever patches
> > the Windows Critical Update thing thinks are current.
> That all sounds good.
Linux should have a sufficient iconv...
> > I have some a nvarchar columns on the SQL Server side and
> I'd like to
> > read them in a perl application on the Linux box. After reading the
> > stuff at http://www.freetds.org/userguide/aboutunicode.htm, I set
> > "client charset = UTF-8" and "tds version = 8.0" in freetds.conf
> > ... but I'm getting a lot of messages like this:
> > Message String: WARNING! Some character(s) could not be
> converted into
> > client's character set. Unconverted bytes were changed to question
> > marks ('?').
This mean that server send some characters that cannot be converted to
UTF-8. This can be caused inserting invalid data into varchar field (not
nvarchar). For example if you insert a 0x80 character in a iso8859-1
field (Latin1, default windows installation). Server insert data just
like you send but FreeTDS say you are doing something wrong... MS
updated copdepage to iso8859-15 but forgot to update it's server...
> > I'm not sure how this can be, since I'm pretty sure UTF-8 should be
> > able to represent any character that can be represented in UCS-2.
> Absolutely correct.
> > iconv to convert client-side data to the "UTF-8" character set
> > 23:32:00.242427 tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
> > 23:32:00.242922 tds_iconv_info_init: converting
> > Does that last bit mean that FreeTDS is converting data from the
> > server to ISO-8859-1 instead of UTF-8?
> No, that's normal. FreeTDS has to be able to convert from
> something like
> ASCII (of which ISO 8859-1 is a superset) to UCS-2 because
> FreeTDS's own
> text is ASCII-encoded. In managing the session with the server, the
> library needs to issue queries "under the hood", and of
> course with TDS
> 7.0 those queries are encoded as UCS-2.
> What you're doing should work, afaict. Could you post a
> TDSDUMP log of a
> session with one of these errors? Also, it might help if you
> could send
> as a binary attachment the data you're selecting. In theory, I could
> install them on my test server and reproduce your pr.
> BTW, as you're using UTF-8, you'll want to use the patch I
> posted here for
> src/tds/write.c. You can apply the patch, or fetch write.c revision
> 18.104.22.168 from CVS. It affects queries larger than 128 characters.
I'm not sure 100% what I wrote... dump will help to discover the
More information about the FreeTDS