Skip to Content.
Sympa Menu

freetds - Re: [freetds] reading nvarchar columns in UTF-8

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "James K. Lowden" <jklowden AT schemamania.org>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] reading nvarchar columns in UTF-8
  • Date: Wed, 28 Jan 2004 09:25:04 -0500

On Wed, 28 Jan 2004 00:32:43 -0800, Will Lowe <harpo AT thebackrow.net>
wrote:
> I'm using FreeTDS 0.62.1 and DBD::Sybase 1.02 on a box running Debian
> unstable (sid) Linux trying to talk to M$ SQL 2k with whatever patches
> the Windows Critical Update thing thinks are current.

That all sounds good.

> I have some a nvarchar columns on the SQL Server side and I'd like to
> read them in a perl application on the Linux box. After reading the
> stuff at http://www.freetds.org/userguide/aboutunicode.htm, I set
> "client charset = UTF-8" and "tds version = 8.0" in freetds.conf
> ... but I'm getting a lot of messages like this:
>
> Message String: WARNING! Some character(s) could not be converted into
> client's character set. Unconverted bytes were changed to question
> marks ('?').

:-(

> I'm not sure how this can be, since I'm pretty sure UTF-8 should be
> able to represent any character that can be represented in UCS-2.

Absolutely correct.

> iconv to convert client-side data to the "UTF-8" character set
> 23:32:00.242427 tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
> 23:32:00.242922 tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
>
> Does that last bit mean that FreeTDS is converting data from the
> server to ISO-8859-1 instead of UTF-8?

No, that's normal. FreeTDS has to be able to convert from something like
ASCII (of which ISO 8859-1 is a superset) to UCS-2 because FreeTDS's own
text is ASCII-encoded. In managing the session with the server, the
library needs to issue queries "under the hood", and of course with TDS
7.0 those queries are encoded as UCS-2.

What you're doing should work, afaict. Could you post a TDSDUMP log of a
session with one of these errors? Also, it might help if you could send
as a binary attachment the data you're selecting. In theory, I could
install them on my test server and reproduce your pr.

BTW, as you're using UTF-8, you'll want to use the patch I posted here for
src/tds/write.c. You can apply the patch, or fetch write.c revision
1.57.2.1 from CVS. It affects queries larger than 128 characters.

--jkl




Archive powered by MHonArc 2.6.24.

Top of Page