Skip to Content.
Sympa Menu

freetds - Ode to UTF-8

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "Lowden, James K" <LowdenJK AT bernstein.com>
  • To: "'TDS Development Group'" <freetds AT franklin.metalab.unc.edu>
  • Subject: Ode to UTF-8
  • Date: Fri, 30 Aug 2002 16:29:29 -0400


> From: Brian Bruns [mailto:camber AT ais.org]
> Sent: August 30, 2002 6:27 AM
>
> Problem is, if we haul around ucs-2 then conversion becomes
> one thing for
> thing for TDS4/5 and something else for TDS7/8, plus we may
> have lost the
> connection info by that point (a direct call to
> dbconvert/cs_convert).
> Despite its warts you can still call strlen() on utf8 which
> is a huge win in my book.

Hear, hear!

I smell an endian. UCS-2 is CPU-architecture-afflicted. That, as much as
anything, is why we should use UTF-8. Beginning with 0.61.

Now, if libtds were written in C++, and we could #include <basic_string>, we
could circumvent Brian's other argument, said lack of functions. C++
strings can be based on wide characters, at least in theory. As things are,
though, in silly old C, UTF-8 will be the least painful.

In short:

client <-> iconv <-> FreeTDS <-> iconv <-> server
set-1 <-> UTF-8 <-> set-2

First thing we do, when we take character data off the wire, is transform it
into UTF-8. I say we do this always, regardless of protocol. Last thing we
do, when we give it to the client, is transform it into the client's
character set. No ascii or unicode anywhere, except at the boundaries
(client and wire).

My suggestion implies iconv is always present. I think that's fine. It's a
rare box today that doesn't have iconv installed, and it's not hard to
install.

Of course I realize that converting ASCII to UTF-8, only to convert it back
to the same ASCII again is Extra Work, unnecessary when client and server
share a character set. But the trend is away from Sybase servers, and we
want to support non-Latin character sets. "UTF-8 always" will simplify the
code and quickly remove every byte==character assumption.

Once we put our iconv filters on each end, and ensure that the internal
representation is always UTF-8, we can add a little short-circuit switch to
them that skips the conversion when server and client both use the same
ASCII or UTF-8 character set. No sneaky
hack-off-the-top-byte-it-mostly-works though! Eeww! Think of the clients
downwind!

Modern Sybase servers, by the way, can deliver UFT-8. Translation is done
on the server. By using UTF-8 within FreeTDS, we'd give the client the
choice of client- or server-based translation. At connect-time, no less.

That's my perspective. Comments welcome, of course.

--jkl


The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above. If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that for certain accounts we do not accept
orders and/or instructions by e-mail, and for those accounts we will not be
responsible for carrying out such orders and/or instructions. Kindly refrain
from sending orders or instructions by e-mail unless you have confirmed that
we accept such communications for your account. Please also note that to
satisfy regulatory requirements we review the outgoing and incoming e-mail
correspondence of staff members serving certain functions.





  • Ode to UTF-8, Lowden, James K, 08/30/2002

Archive powered by MHonArc 2.6.24.

Top of Page