Skip to Content.
Sympa Menu

freetds - Re: Ode to UTF-8

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Brian Bruns <camber AT ais.org>
  • To: TDS Development Group <freetds AT franklin.metalab.unc.edu>
  • Subject: Re: Ode to UTF-8
  • Date: Fri, 30 Aug 2002 17:56:10 -0400 (EDT)



The double conversion bothers me a bit. I think we have a couple basic
situations.

1. Data in the server is contained with in a single character set with
non-unicode aware application.

2. Data in server is single char set with utf-8 aware application

3. Data in server is multilingual with utf-8 aware application

There is also the case of multibyte non-unicode charsets like big5, but I
don't have any expertise in cjkv to know where to begin even.

Situation #1 is by far the most common. English, and all the germanic and
romance language groups fit into latin_1 afaik. In any event, this is our
default high-order byte stripping behaviour. It is the most common case
and we should optimize for it.

#2 and #3 will require FreeTDS to become utf-8 clean (if it's not
already) but otherwise is a simple matter.

By doing utf-8 internally we are optimizing for the least common cases and
causing two conversions for the common case. If we are capable of
handling utf-8 internally then we can handle any single byte set (where
any means any single byte encoding where 0 is null, all of them afaik)
just as easily.

I guess my point is that FreeTDS doesn't care what is inside the string
after it is initially read. I understand freddy's concern about the
non-deterministic length of utf-8 strings, and i agree, but considering
we can't convert the world beyond us to ucs-2 (nor should we, it's very
limiting for asian languages), it's pay me now or pay me later.

I think pure utf-8 is going to add complexity to the code instead of
remove it, given our requirements. We could easily make utf-8 the default
conversion, and that would be fine for us english speakers (utf-8 being
compatible in the range 0-127), but I think we'll be pissing off
any european who wants to use a circumflex, umlaut, or accents when their
non-unicode aware app prints garbage by default.

We spent a long time on the subject last time around, and I for one was
pretty happy with the conclusion from both the perspective of how clean to
the code to implement it was, and the functionality provided by it.

Brian


On Fri, 30 Aug 2002, Lowden, James K wrote:

> > From: Brian Bruns [mailto:camber AT ais.org]
> > Sent: August 30, 2002 6:27 AM
> >
> > Problem is, if we haul around ucs-2 then conversion becomes
> > one thing for
> > thing for TDS4/5 and something else for TDS7/8, plus we may
> > have lost the
> > connection info by that point (a direct call to
> > dbconvert/cs_convert).
> > Despite its warts you can still call strlen() on utf8 which
> > is a huge win in my book.
>
> Hear, hear!
>
> I smell an endian. UCS-2 is CPU-architecture-afflicted. That, as much as
> anything, is why we should use UTF-8. Beginning with 0.61.
>
> Now, if libtds were written in C++, and we could #include <basic_string>, we
> could circumvent Brian's other argument, said lack of functions. C++
> strings can be based on wide characters, at least in theory. As things are,
> though, in silly old C, UTF-8 will be the least painful.
>
> In short:
>
> client <-> iconv <-> FreeTDS <-> iconv <-> server
> set-1 <-> UTF-8 <-> set-2
>
> First thing we do, when we take character data off the wire, is transform it
> into UTF-8. I say we do this always, regardless of protocol. Last thing we
> do, when we give it to the client, is transform it into the client's
> character set. No ascii or unicode anywhere, except at the boundaries
> (client and wire).
>
> My suggestion implies iconv is always present. I think that's fine. It's a
> rare box today that doesn't have iconv installed, and it's not hard to
> install.
>
> Of course I realize that converting ASCII to UTF-8, only to convert it back
> to the same ASCII again is Extra Work, unnecessary when client and server
> share a character set. But the trend is away from Sybase servers, and we
> want to support non-Latin character sets. "UTF-8 always" will simplify the
> code and quickly remove every byte==character assumption.
>
> Once we put our iconv filters on each end, and ensure that the internal
> representation is always UTF-8, we can add a little short-circuit switch to
> them that skips the conversion when server and client both use the same
> ASCII or UTF-8 character set. No sneaky
> hack-off-the-top-byte-it-mostly-works though! Eeww! Think of the clients
> downwind!
>
> Modern Sybase servers, by the way, can deliver UFT-8. Translation is done
> on the server. By using UTF-8 within FreeTDS, we'd give the client the
> choice of client- or server-based translation. At connect-time, no less.
>
> That's my perspective. Comments welcome, of course.
>
> --jkl
>




  • Ode to UTF-8, Lowden, James K, 08/30/2002
    • <Possible follow-up(s)>
    • Re: Ode to UTF-8, Brian Bruns, 08/30/2002

Archive powered by MHonArc 2.6.24.

Top of Page