FreeTDS Development Group

[Steve Langasek <vorlon AT netexpress.net>]

On Fri, 8 Jun 2001, James K. Lowden wrote:

> Steve Langasek wrote:

>> The problem is that the 16-bit Unicode character space isn't large enough
>> to cover all of the oriental character sets.

> For some value of "all", you're probably right, Steve.  But I think it would
> carry us pretty far.

A good article covering the CJK-Unicode issues was linked to from Slashdot
recently: <http://www.hastingsresearch.com/net/04-unicode-limitations.shtml>.
What it boils down to is that the opinion that 16-bits is big enough for
everybody is a Western-imposed proclamation; and just because it suffices for
Chinese /newspapers/ doesn't mean it's good enough for all of China.  Anything
written in classic Chinese, which encompasses all historical and literary
documents written before the 1940s as well as some modern academic work, is
unrepresentable in Unicode.  Imagine the state we would be in if Japan set a
standard limiting our dictionary to those words that are found in our modern
newspapers...

> From what I understand, Unicode is UCS-2 and is 16-bit.  It can be converted
> to UTF-8, a multibyte counted-string system (no null terminators).

UTF-8 allows for null-termination of strings, if that's what you mean; an
explicit design consideration of UTF-8 was to make it compatible with existing
ascii-oriented string-handling functions that use null-termination.


> All three manuals state Unicode can represent Chinese.  (There's only one
> written Chinese language.  Is there more than one "character set"?)

Manuals written by Westerners... who are not sensitive to the issues the
Chinese face with Unicode. :)  Although awareness of the issues is much higher
now, and hopefully it will only be a matter of time before UCS-4 is adopted.

> There doesn't seem to be any support out there for UCS-4, unless you count 
> Gnu's
> 32-bit wchar_t type (which they say isn't strictly compliant).  So, 
> supporting
> Unicode would put FreeTDS on par with its databases.

Exactly -- we won't be able to support CJK fully using Unicode, but this is
currently not our problem to solve.

Steve Langasek
postmodern programmer