freetds AT lists.ibiblio.org
Subject: FreeTDS Development Group
List archive
- From: "James K. Lowden" <jklowden AT speakeasy.org>
- To: TDS Development Group <freetds AT franklin.oit.unc.edu>
- Subject: Re: Unicode, UTF-8, and Greek
- Date: Wed, 10 Oct 2001 23:59:55 -0400
Hi Brian,
Brian Bruns wrote:
> I like UTF-8 on the input side alot. I suppose I'll have to match it on
> the bind side too...my main concerns are:
>
> 1) UTF-8 may be larger, applications that make assumptions that a
> varchar(255) can be no longer than 255 bytes will break. I could say we
> just truncate to the set boundary or if it's unlimited, the application
> could core dump. If so, app is broken as designed I suppose.
You're right, it's treacherous.
Try this in isql on NT:
select substring( N'world', 1, 3 )
You will see (from memory; I'm at home):
------
wor
(1 row(s) affected)
SQL Server 7 returned a column six bytes wide, 2 bytes/character.
That's the right answer and the true one; the server and the client
can't predict when the first wide character will arrive, so they reserve
the space always.
I don't know what FreeTDS does with that today, but I'd guess it would
indicate a 3-byte column.
(It's a little maddening, if you spend much time in Query Analyzer
looking at system tables, whose "name" columns are now nvarchar(255),
meaning they take up 512 positions on the screen. That's a lot of
whitespace, even on a 1600x1200 screen. Especially considering there
aren't any *fonts* installed to display non-Latin data, in the unlikely
event any such thing were returned.)
> I think having a default of UTF-8 is perhaps not a bad idea after all but
> let's add a configuration setting to convert to local 8bit where
> applicable. (We have to call iconv anyway, it'll be pretty easy to just
> substitute greek or whatever for UTF-8 where needed).
Thought experiment: You're in the bind business anyway. How about
leaving the Ascii/UTF-8 choice to the bind code? That is, if the client
binds varchar to the column, return ascii. If he binds nvarchar to it,
return UTF-8?
As you've pointed out before, the principle of least surprise should
apply, and we don't know what perl/python/php expect/want/do with
UTF-8. I started looking into the perl side; I'll let you know what I
find out.
Me, I can't wrap my head around BCP. What does Japanese text in a bcp
data file look like on a Unix box? Depends on who's doing the looking,
no doubt. But since Unix files are a stream of bytes, the situation
smells suspiciously like our varchar "client conspiracy", only at the OS
level. It's a puzzlement.
> > What does Sybase support for Unicode?
>
> Not sure, but I'm interested in finding out.
I looked at their online documentation a few months ago. IIRC it looked
like a layered product of some sort that seemed to entail some
new/different APIs. I don't recall anything about UTF-8; I remember
only Unicode.
I don't think we've seen one question ever on this list about Unicode
data on a Sybase system, which I take as zero interest. Just one man's
opinion, of course.
--jkl
-
Re: Unicode, UTF-8, and Greek
, (continued)
- Re: Unicode, UTF-8, and Greek, Lowden, James K, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Bob Kline, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Steve Langasek, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Steve Langasek, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Curt Hagenlocher, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Steve Langasek, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Curt Hagenlocher, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Steve Langasek, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Brian Bruns, 10/10/2001
- Re: Unicode, UTF-8, and Greek, Ed Edgar, 10/10/2001
- Re: Unicode, UTF-8, and Greek, James K. Lowden, 10/11/2001
- Re: Unicode, UTF-8, and Greek, Curt Hagenlocher, 10/11/2001
Archive powered by MHonArc 2.6.24.