FreeTDS Development Group

[Steve Langasek <vorlon AT netexpress.net>]

On Fri, 8 Jun 2001, Brian Bruns wrote:

> Ok, so long (and i mean *long*) story short...Microsoft is sending UCS2
> characters to the client. We've got to do something more intelligent with
> them than just skip every other byte.

> The most important goal here is "don't break anything."  UTF-8 retains
> null termination which is pretty cool as far as this goes.  Using iconv to
> do UCS2 -> UTF-8 translation would "just work" for the english speaking
> folks (me).  This strikes me as a good default scheme.

> It would be nice to support UCS2 to other single byte character sets
> such that greek, german, french, etc... could work with existing systems
> that did not have UTF-8 support (variable length characters, bleah!). Is
> UTF-8 compatible with any of iso_1? or are chars > 127 all different?

UTF-8 is more-or-less (I don't remember exactly if it's more, or less)
compatible with ISO-8859-1, because the first set of international characters
that show up in Unicode are (naturally) Latin-1.  However, I'm not sure that
UTF-8 and ISO-8859-1 are binary-compatible, because I think most of the high
bits used to represent printable characters in ISO-8859-1 are used as markers
for multibyte characters in UTF-8.  Truncating the high byte as you currently
do is fully compatbile with ISO-8859-1, so that's one (minor) disadvantage to
switching to UTF-8.

"Don't break anything" certainly applies to existing APIs that are defined to
return (char *).  Are there other APIs defined (perhaps by Microsoft) that
will return UCS2 (or wchar_t *)?  If not, perhaps FreeTDS should be leading
the way...

But probably not until after 0.52? :)

> There should be some mechanism for a straight return of UCS2 to the
> client, assuming it can handle multibyte chars that is, for our CJK
> friends.

The character space available in UCS2 and UTF-8 is equivalent, AFAIUI.  The
differences between the two are primarily in the question of parsing overhead
(UTF-8) versus memory/bandwidth overhead (UCS2).

> This is all well and good as long as we are querying text.  I have no clue
> what happens on the insert (SQL) side of the house.

If the TDS protocol version in use supports Unicode, I think it's safe to
assume text being inserted is also UTF-8, or at least it's safe to handle it
/as if it were/.  I believe that any given null-terminated bytestream has a
vaild interpretation as UTF-8, which means it can also be successfully mapped
to Unicode -- and un-mapped again on the other end.

Steve Langasek
postmodern programmer