Skip to Content.
Sympa Menu

freetds - Re: Unicode, UTF-8, and Greek

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "James K. Lowden" <jklowden AT speakeasy.org>
  • To: TDS Development Group <freetds AT franklin.oit.unc.edu>
  • Subject: Re: Unicode, UTF-8, and Greek
  • Date: Wed, 10 Oct 2001 23:59:55 -0400


Hi Brian,

Brian Bruns wrote:
> I like UTF-8 on the input side alot. I suppose I'll have to match it on
> the bind side too...my main concerns are:
>
> 1) UTF-8 may be larger, applications that make assumptions that a
> varchar(255) can be no longer than 255 bytes will break. I could say we
> just truncate to the set boundary or if it's unlimited, the application
> could core dump. If so, app is broken as designed I suppose.

You're right, it's treacherous.

Try this in isql on NT:

select substring( N'world', 1, 3 )

You will see (from memory; I'm at home):

------
wor

(1 row(s) affected)

SQL Server 7 returned a column six bytes wide, 2 bytes/character.
That's the right answer and the true one; the server and the client
can't predict when the first wide character will arrive, so they reserve
the space always.

I don't know what FreeTDS does with that today, but I'd guess it would
indicate a 3-byte column.

(It's a little maddening, if you spend much time in Query Analyzer
looking at system tables, whose "name" columns are now nvarchar(255),
meaning they take up 512 positions on the screen. That's a lot of
whitespace, even on a 1600x1200 screen. Especially considering there
aren't any *fonts* installed to display non-Latin data, in the unlikely
event any such thing were returned.)

> I think having a default of UTF-8 is perhaps not a bad idea after all but
> let's add a configuration setting to convert to local 8bit where
> applicable. (We have to call iconv anyway, it'll be pretty easy to just
> substitute greek or whatever for UTF-8 where needed).

Thought experiment: You're in the bind business anyway. How about
leaving the Ascii/UTF-8 choice to the bind code? That is, if the client
binds varchar to the column, return ascii. If he binds nvarchar to it,
return UTF-8?

As you've pointed out before, the principle of least surprise should
apply, and we don't know what perl/python/php expect/want/do with
UTF-8. I started looking into the perl side; I'll let you know what I
find out.
Me, I can't wrap my head around BCP. What does Japanese text in a bcp
data file look like on a Unix box? Depends on who's doing the looking,
no doubt. But since Unix files are a stream of bytes, the situation
smells suspiciously like our varchar "client conspiracy", only at the OS
level. It's a puzzlement.

> > What does Sybase support for Unicode?
>
> Not sure, but I'm interested in finding out.

I looked at their online documentation a few months ago. IIRC it looked
like a layered product of some sort that seemed to entail some
new/different APIs. I don't recall anything about UTF-8; I remember
only Unicode.

I don't think we've seen one question ever on this list about Unicode
data on a Sybase system, which I take as zero interest. Just one man's
opinion, of course.

--jkl




Archive powered by MHonArc 2.6.24.

Top of Page