FreeTDS Development Group

[Brian Bruns <camber AT ais.org>]

On Wed, 10 Oct 2001, Curt Hagenlocher wrote:

> Steve Langasek <vorlon AT netexpress.net> writes:
> 
> > FreeTDS is being written today, not
> > five years ago when UCS-2 was being put into NT4.  A Unix widechar
> > implementation done today should be using UCS-4.  Much better still is
> > the multibyte standard, UTF-8, which almost everyone in the Unix world
> > uses as the Unicode encoding of choice.
> 
> As someone else pointed out, you'd still be limited by the server
> implementation even if the client implements the newer, wider spec.
> UTF-8 can, of course, be stored in regular char/varchar fields as
> long as all the clients of the database app agree to use that
> representation.
> 
> Would you return an error if someone passed a valid UCS-4 or UTF-8
> string that was not also a valid UCS-2 string?

All data going *to* the server is in the form of a SQL statement (in
UCS2).  Again, currently we only support ascii at the moment and thunk it
to 16bit before sending it.  UTF-8 would be a very nice fit for sending as
it eliminates differentiating between 8bit codes and unicode for which
there is no API support.

I like UTF-8 on the input side alot.  I suppose I'll have to match it on
the bind side too...my main concerns are:

1) UTF-8 may be larger, applications that make assumptions that a
varchar(255) can be no longer than 255 bytes will break.  I could say we
just truncate to the set boundary or if it's unlimited, the application
could core dump.  If so, app is broken as designed I suppose.

2) Behaviour would differ if the client was configured to use 4.2
(localized character set) or TDS 7.0 (UTF-8).

I think having a default of UTF-8 is perhaps not a bad idea after all but
let's add a configuration setting to convert to local 8bit where
applicable. (We have to call iconv anyway, it'll be pretty easy to just
substitute greek or whatever for UTF-8 where needed).
 
> What does Sybase support for Unicode?

Not sure, but I'm interested in finding out. I think unicode support is
System 12 and later.

Brian