freetds AT lists.ibiblio.org
Subject: FreeTDS Development Group
List archive
- From: "James K. Lowden" <jklowden AT schemamania.org>
- To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
- Subject: Re: [freetds] reading nvarchar columns in UTF-8
- Date: Wed, 28 Jan 2004 09:25:04 -0500
On Wed, 28 Jan 2004 00:32:43 -0800, Will Lowe <harpo AT thebackrow.net>
wrote:
> I'm using FreeTDS 0.62.1 and DBD::Sybase 1.02 on a box running Debian
> unstable (sid) Linux trying to talk to M$ SQL 2k with whatever patches
> the Windows Critical Update thing thinks are current.
That all sounds good.
> I have some a nvarchar columns on the SQL Server side and I'd like to
> read them in a perl application on the Linux box. After reading the
> stuff at http://www.freetds.org/userguide/aboutunicode.htm, I set
> "client charset = UTF-8" and "tds version = 8.0" in freetds.conf
> ... but I'm getting a lot of messages like this:
>
> Message String: WARNING! Some character(s) could not be converted into
> client's character set. Unconverted bytes were changed to question
> marks ('?').
:-(
> I'm not sure how this can be, since I'm pretty sure UTF-8 should be
> able to represent any character that can be represented in UCS-2.
Absolutely correct.
> iconv to convert client-side data to the "UTF-8" character set
> 23:32:00.242427 tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
> 23:32:00.242922 tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
>
> Does that last bit mean that FreeTDS is converting data from the
> server to ISO-8859-1 instead of UTF-8?
No, that's normal. FreeTDS has to be able to convert from something like
ASCII (of which ISO 8859-1 is a superset) to UCS-2 because FreeTDS's own
text is ASCII-encoded. In managing the session with the server, the
library needs to issue queries "under the hood", and of course with TDS
7.0 those queries are encoded as UCS-2.
What you're doing should work, afaict. Could you post a TDSDUMP log of a
session with one of these errors? Also, it might help if you could send
as a binary attachment the data you're selecting. In theory, I could
install them on my test server and reproduce your pr.
BTW, as you're using UTF-8, you'll want to use the patch I posted here for
src/tds/write.c. You can apply the patch, or fetch write.c revision
1.57.2.1 from CVS. It affects queries larger than 128 characters.
--jkl
-
[freetds] reading nvarchar columns in UTF-8,
Will Lowe, 01/28/2004
- Re: [freetds] reading nvarchar columns in UTF-8, James K. Lowden, 01/28/2004
- <Possible follow-up(s)>
- RE: [freetds] reading nvarchar columns in UTF-8, ZIGLIO, Frediano, VF-IT, 01/28/2004
Archive powered by MHonArc 2.6.24.