Skip to Content.
Sympa Menu

freetds - Re: [freetds] FreeTDS and UTF-8 support

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Steve Langasek <vorlon AT dodds.net>
  • To: freetds AT lists.ibiblio.org
  • Subject: Re: [freetds] FreeTDS and UTF-8 support
  • Date: Thu, 24 Jul 2008 11:08:49 -0700

On Thu, Jul 24, 2008 at 09:16:38AM +0200, Sebastien FLAESCH wrote:
> >> We agree; we're just tripping over terminology. By "multibyte
> >> characters"
> >> I meant pre-Unicode encodings, such as input to mbtowc(3). I'm happy to
> >> support UTF-8.

> > FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc()
> > from
> > glibc.

> That's true.

> I think FreeTDS could support all single-byte or multi-byte character sets
> if:
>
> 1) The application (main) calls setlocale() to define the current charset
> for libc functions like mbtowc, wctomb, mblen (that would include UTF-8
> support).

> 2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the
> size in bytes of the characters (= no byte-based parsers/lexers).

> 3) Before sending on TDS protocol, convert from current charset to
> WideChar (UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses,
> convert from WideChar to current charset with wcstombs().

So mbtowc() and wctomb() only convert between widechar and multibyte
representations *of the same charset*. That makes them insufficient for
converting to "the current charset", because you always want your WC
representation to be UCS-2 but you want your mb representation to be in the
current locale, which may not be a Unicode locale.

Hence iconv().

> ...

> From my understanding, a C application on UNIX can be written in:

> A- Pure ASCII (LC_ALL=POSIX)
> B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
> C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
> D- Wide Char (UTF-16/UCS-2), using the wchar_t type.

I don't think you can be very effective writing a C application on Unix
using a widechar representation internally. All string manipulation
functions in the standard library, and in other common libraries built on
top of them, are implemented using char* as the argument type.

> Maybe FreeTDS could work in wchar_t internally and do charset conversion
> at the ODBC API level? I think this is how MS ODBC drivers work... no?

This is how MS ODBC drivers work, because MS is wchar_t-centric. Unix is
not. Using wchar_t internally and having to either convert to char for
every string manipulation, or reimplement your own wchar-libc, == pain.

> I mean (casts are just there for understanding):

> SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset",
> SQL_NTS);

> would be mapped to:

> SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);

> and just use the mbstowcs() standard C lib function to convert current
> charset to wide char...

> WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws
> is 2 bytes.

Right, that means you can't even use mbtowc() to do the conversions, because
your wchar_t internal representation no longer matches the output format of
*either* of your APIs, so... what's the point in using it as the internal
representation?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek AT ubuntu.com vorlon AT debian.org




Archive powered by MHonArc 2.6.24.

Top of Page