FreeTDS Development Group

[Steve Langasek <vorlon AT netexpress.net>]

James,

On Wed, Oct 10, 2001 at 12:23:14PM -0400, Lowden, James K wrote:
> >Ironically, the users who have the most problems with Unicode in general
> >(CJK users, due to incompatibilities with other national encodings) are
> >also the ones most maligned by UCS2 in particular -- but this is not a
> >problem FreeTDS can solve unless we started pushing UTF-16 data into the
> >database instead of UCS2, thus breaking compatibility with all Windows
> >client software.

> Steve, 

> Not just "Windows client software".  The server side, too.  

> Microsoft stores textual data in two forms:  Ascii as char and Unicode
> (UCS2) as nchar data types.  It collates both and pretty much allows
> char/nchar to be used interchangeably.  The only way to store UTF-16 data on
> the server would be either as some kind of blob, or by lying: pretending to
> the server that our varchar field, say, has ascii when in fact it has
> UTF-16.  Such fields could be used in joins (to others like them), but
> there's no way to fashion a "where" clause in the native SQL to search them,
> and no (meaningful) way to sort them.  No tool could display them legibly,
> either.  

I meant to imply that the clients would all be lying to the server about
the kind of data they're storing, yes.  If all clients agree to lie and
store UTF-16 where the server expects UCS-2 (since the server can't tell
these apart, any more than you can tell the difference between Latin1
and UTF-8 without using heuristics), then I believe this would be an
effective way of storing full Unicode UTF-16 text in a UCS2 SQL server.
But it requires that *all* client software be involved in this
conspiracy, so anyone who has any Windows clients would not benefit from
such a hack.

> BTW, do you know of any database, free or proprietary, that understands
> anything besides Ascii and UCS2?  AFAIK, no one does and no one's trying.  

I've heard just today that MySQL will be adding Unicode support.  I'll
read up on it -- and threaten bodily harm if I find them using UCS2. ;)

Steve Langasek
postmodern programmer

Attachment: pgpzk8SZLZvf0.pgp
Description: PGP signature