Unicode, UTF-8, and Greek
James K. Lowden
jklowden at speakeasy.org
Wed Oct 10 23:59:55 EDT 2001
Brian Bruns wrote:
> I like UTF-8 on the input side alot. I suppose I'll have to match it on
> the bind side too...my main concerns are:
> 1) UTF-8 may be larger, applications that make assumptions that a
> varchar(255) can be no longer than 255 bytes will break. I could say we
> just truncate to the set boundary or if it's unlimited, the application
> could core dump. If so, app is broken as designed I suppose.
You're right, it's treacherous.
Try this in isql on NT:
select substring( N'world', 1, 3 )
You will see (from memory; I'm at home):
(1 row(s) affected)
SQL Server 7 returned a column six bytes wide, 2 bytes/character.
That's the right answer and the true one; the server and the client
can't predict when the first wide character will arrive, so they reserve
the space always.
I don't know what FreeTDS does with that today, but I'd guess it would
indicate a 3-byte column.
(It's a little maddening, if you spend much time in Query Analyzer
looking at system tables, whose "name" columns are now nvarchar(255),
meaning they take up 512 positions on the screen. That's a lot of
whitespace, even on a 1600x1200 screen. Especially considering there
aren't any *fonts* installed to display non-Latin data, in the unlikely
event any such thing were returned.)
> I think having a default of UTF-8 is perhaps not a bad idea after all but
> let's add a configuration setting to convert to local 8bit where
> applicable. (We have to call iconv anyway, it'll be pretty easy to just
> substitute greek or whatever for UTF-8 where needed).
Thought experiment: You're in the bind business anyway. How about
leaving the Ascii/UTF-8 choice to the bind code? That is, if the client
binds varchar to the column, return ascii. If he binds nvarchar to it,
As you've pointed out before, the principle of least surprise should
apply, and we don't know what perl/python/php expect/want/do with
UTF-8. I started looking into the perl side; I'll let you know what I
Me, I can't wrap my head around BCP. What does Japanese text in a bcp
data file look like on a Unix box? Depends on who's doing the looking,
no doubt. But since Unix files are a stream of bytes, the situation
smells suspiciously like our varchar "client conspiracy", only at the OS
level. It's a puzzlement.
> > What does Sybase support for Unicode?
> Not sure, but I'm interested in finding out.
I looked at their online documentation a few months ago. IIRC it looked
like a layered product of some sort that seemed to entail some
new/different APIs. I don't recall anything about UTF-8; I remember
I don't think we've seen one question ever on this list about Unicode
data on a Sybase system, which I take as zero interest. Just one man's
opinion, of course.
More information about the FreeTDS