FreeTDS Development Group

[Steve Langasek <vorlon AT netexpress.net>]

On Wed, Oct 10, 2001 at 12:55:56AM -0400, Brian Bruns wrote:

> TDS 7.0 is completely unicode at the protocol level.  Currently, what we
> do is strip it to ascii (ie ignore high order bytes), so it is broken for
> virtually all non-western languages.  I've basically held off fixing it
> because I felt converting it to UTF-8 may break some applications not
> expecting multibyte characters, but also because we are a bunch of english
> speakers trying to decide what is best for speakers of other languages ;-)

> So, I been basically waiting for someone who is actually going to use TDS
> 7 with a language other than english to tell me what their applications
> require.  Should the text be exported as UTF-8, UCS2, localized into a
> non-ascii 8bit character set, or some other option?

UCS2 is broken, because it truncates Unicode at the 16-bit mark,
and nobody but Microsoft uses it.  Certainly the server implementation
is constrained, but as a consumer of the client libraries, I for one
would rather like to never have to touch that ugliness.  It should be
the library's responsibility to turn this into something Unix-friendly.

That would leave UTF-8, or region-specific character sets.  From a big
picture standpoint, Unix is effectively standardizing on UTF-8.  UTF-8
also represents the most effective use of programming time, since it's
easier to support one character set than it is to support many.
Ironically, the users who have the most problems with Unicode in general
(CJK users, due to incompatibilities with other national encodings) are
also the ones most maligned by UCS2 in particular -- but this is not a
problem FreeTDS can solve unless we started pushing UTF-16 data into the
database instead of UCS2, thus breaking compatibility with all Windows
client software.

So granted, UTF-8 is not a perfect solution, but I don't think there is
a perfect solution at this point, and UTF-8 makes for a good one-off
approximation with relatively little pain of implementation.  And one of
these days, I'm going to put my code where my mouth is and start working
on said implementation...  One of these days.

Steve Langasek
postmodern programmer

Attachment: pgpC1g4rzLTrc.pgp
Description: PGP signature