FreeTDS Development Group

[Brian Bruns <camber AT ais.org>]

The double conversion bothers me a bit.  I think we have a couple basic 
situations.

1. Data in the server is contained with in a single character set with 
non-unicode aware application.

2. Data in server is single char set with utf-8 aware application

3. Data in server is multilingual with utf-8 aware application

There is also the case of multibyte non-unicode charsets like big5, but I 
don't have any expertise in cjkv to know where to begin even.

Situation #1 is by far the most common.  English, and all the germanic and 
romance language groups fit into latin_1 afaik.  In any event, this is our 
default high-order byte stripping behaviour.  It is the most common case 
and we should optimize for it.

#2 and #3 will require FreeTDS to become utf-8 clean (if it's not 
already) but otherwise is a simple matter.

By doing utf-8 internally we are optimizing for the least common cases and 
causing two conversions for the common case.  If we are capable of 
handling utf-8 internally then we can handle any single byte set (where 
any means any single byte encoding where 0 is null, all of them afaik) 
just as easily.

I guess my point is that FreeTDS doesn't care what is inside the string 
after it is initially read.  I understand freddy's concern about the 
non-deterministic length of utf-8 strings, and i agree, but considering 
we can't convert the world beyond us to ucs-2 (nor should we, it's very 
limiting for asian languages), it's pay me now or pay me later.

I think pure utf-8 is going to add complexity to the code instead of 
remove it, given our requirements.  We could easily make utf-8 the default 
conversion, and that would be fine for us english speakers (utf-8 being 
compatible in the range 0-127), but I think we'll be pissing off 
any european who wants to use a circumflex, umlaut, or accents when their 
non-unicode aware app prints garbage by default.

We spent a long time on the subject last time around, and I for one was 
pretty happy with the conclusion from both the perspective of how clean to 
the code to implement it was, and the functionality provided by it.

Brian


On Fri, 30 Aug 2002, Lowden, James K wrote:

> > From: Brian Bruns [mailto:camber AT ais.org]
> > Sent: August 30, 2002 6:27 AM
> > 
> > Problem is, if we haul around ucs-2 then conversion becomes 
> > one thing for
> > thing for TDS4/5 and something else for TDS7/8, plus we may 
> > have lost the
> > connection info by that point (a direct call to 
> > dbconvert/cs_convert). 
> > Despite its warts you can still call strlen() on utf8 which 
> > is a huge win in my book.
> 
> Hear, hear!
> 
> I smell an endian.  UCS-2 is CPU-architecture-afflicted.  That, as much as
> anything, is why we should use UTF-8.  Beginning with 0.61.  
> 
> Now, if libtds were written in C++, and we could #include <basic_string>, we
> could circumvent Brian's other argument, said lack of functions.  C++
> strings can be based on wide characters, at least in theory.  As things are,
> though, in silly old C, UTF-8 will be the least painful.  
> 
> In short:
> 
>       client <-> iconv <-> FreeTDS <-> iconv <-> server
>       set-1       <->       UTF-8       <->       set-2
> 
> First thing we do, when we take character data off the wire, is transform it
> into UTF-8.  I say we do this always, regardless of protocol.  Last thing we
> do, when we give it to the client, is transform it into the client's
> character set.  No ascii or unicode anywhere, except at the boundaries
> (client and wire).  
> 
> My suggestion implies iconv is always present.  I think that's fine.  It's a
> rare box today that doesn't have iconv installed, and it's not hard to
> install.  
> 
> Of course I realize that converting ASCII to UTF-8, only to convert it back
> to the same ASCII again is Extra Work, unnecessary when client and server
> share a character set.  But the trend is away from Sybase servers, and we
> want to support non-Latin character sets.  "UTF-8 always" will simplify the
> code and quickly remove every byte==character assumption.  
> 
> Once we put our iconv filters on each end, and ensure that the internal
> representation is always UTF-8, we can add a little short-circuit switch to
> them that skips the conversion when server and client both use the same
> ASCII or UTF-8 character set.  No sneaky
> hack-off-the-top-byte-it-mostly-works though!  Eeww!  Think of the clients
> downwind!  
> 
> Modern Sybase servers, by the way, can deliver UFT-8.  Translation is done
> on the server.  By using UTF-8 within FreeTDS, we'd give the client the
> choice of client- or server-based translation.  At connect-time, no less.
> 
> That's my perspective.  Comments welcome, of course.  
> 
> --jkl
>