[freetds] FreeTDS and UTF-8 support
vorlon at dodds.net
Thu Jul 24 14:08:49 EDT 2008
On Thu, Jul 24, 2008 at 09:16:38AM +0200, Sebastien FLAESCH wrote:
> >> We agree; we're just tripping over terminology. By "multibyte characters"
> >> I meant pre-Unicode encodings, such as input to mbtowc(3). I'm happy to
> >> support UTF-8.
> > FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc() from
> > glibc.
> That's true.
> I think FreeTDS could support all single-byte or multi-byte character sets
> 1) The application (main) calls setlocale() to define the current charset
> for libc functions like mbtowc, wctomb, mblen (that would include UTF-8
> 2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the
> size in bytes of the characters (= no byte-based parsers/lexers).
> 3) Before sending on TDS protocol, convert from current charset to
> WideChar (UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses,
> convert from WideChar to current charset with wcstombs().
So mbtowc() and wctomb() only convert between widechar and multibyte
representations *of the same charset*. That makes them insufficient for
converting to "the current charset", because you always want your WC
representation to be UCS-2 but you want your mb representation to be in the
current locale, which may not be a Unicode locale.
> From my understanding, a C application on UNIX can be written in:
> A- Pure ASCII (LC_ALL=POSIX)
> B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
> C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
> D- Wide Char (UTF-16/UCS-2), using the wchar_t type.
I don't think you can be very effective writing a C application on Unix
using a widechar representation internally. All string manipulation
functions in the standard library, and in other common libraries built on
top of them, are implemented using char* as the argument type.
> Maybe FreeTDS could work in wchar_t internally and do charset conversion
> at the ODBC API level? I think this is how MS ODBC drivers work... no?
This is how MS ODBC drivers work, because MS is wchar_t-centric. Unix is
not. Using wchar_t internally and having to either convert to char for
every string manipulation, or reimplement your own wchar-libc, == pain.
> I mean (casts are just there for understanding):
> SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset",
> would be mapped to:
> SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);
> and just use the mbstowcs() standard C lib function to convert current
> charset to wide char...
> WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws
> is 2 bytes.
Right, that means you can't even use mbtowc() to do the conversions, because
your wchar_t internal representation no longer matches the output format of
*either* of your APIs, so... what's the point in using it as the internal
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek at ubuntu.com vorlon at debian.org
More information about the FreeTDS