[freetds] FreeTDS and UTF-8 support
sf at 4js.com
Fri Jul 25 04:03:33 EDT 2008
Basically we agree, some comments below.
Steve Langasek wrote:
> On Thu, Jul 24, 2008 at 09:16:38AM +0200, Sebastien FLAESCH wrote:
>>>> We agree; we're just tripping over terminology. By "multibyte characters"
>>>> I meant pre-Unicode encodings, such as input to mbtowc(3). I'm happy to
>>>> support UTF-8.
>>> FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc() from
>> That's true.
>> I think FreeTDS could support all single-byte or multi-byte character sets
>> 1) The application (main) calls setlocale() to define the current charset
>> for libc functions like mbtowc, wctomb, mblen (that would include UTF-8
>> 2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the
>> size in bytes of the characters (= no byte-based parsers/lexers).
>> 3) Before sending on TDS protocol, convert from current charset to
>> WideChar (UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses,
>> convert from WideChar to current charset with wcstombs().
> So mbtowc() and wctomb() only convert between widechar and multibyte
> representations *of the same charset*. That makes them insufficient for
> converting to "the current charset", because you always want your WC
> representation to be UCS-2 but you want your mb representation to be in the
> current locale, which may not be a Unicode locale.
mbtowc(), wctomb(), mbstowcs() and wcstombs() can convert Wide Char to/from
*the current* SB or MB charset defined by a setlocale() call.
From the man pages of wctomb:
"The behavior of wctomb depends on the LC_CTYPE category of the current locale."
> Hence iconv().
From my understanding iconv will be used, but if only WC to MB/SB conversion
is concerned, the standard libc functions are sufficient IMHO.
We use that in our VM to convert from the current charset to wchar_t in order
to implement upshift/downshift functions with towupper / towlower (you cannot
use toupper/tolower when current charset is MB ...
>> From my understanding, a C application on UNIX can be written in:
>> A- Pure ASCII (LC_ALL=POSIX)
>> B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
>> C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
>> D- Wide Char (UTF-16/UCS-2), using the wchar_t type.
> I don't think you can be very effective writing a C application on Unix
> using a widechar representation internally. All string manipulation
> functions in the standard library, and in other common libraries built on
> top of them, are implemented using char* as the argument type.
Honestly I have no idea...
I was just guessing there are WC apps out there on UNIX because I know
about some WC libc functions, you have for example fwprintf(), getwchar().
Note I found no equivalent for fopen()... probably because UNIX file
systems don't have an explicit charset? (filenames are just a set of bytes,
you can have one file name in UTF-8 and another in ISO-8859-1, AFAIK).
I known also that in ou VM we have to convert to WC in some cases because
functions like isalpha(), toupper() work only for SBCS.
>> Maybe FreeTDS could work in wchar_t internally and do charset conversion
>> at the ODBC API level? I think this is how MS ODBC drivers work... no?
> This is how MS ODBC drivers work, because MS is wchar_t-centric. Unix is
> not. Using wchar_t internally and having to either convert to char for
> every string manipulation, or reimplement your own wchar-libc, == pain.
Ok I understand.
>> I mean (casts are just there for understanding):
>> SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset",
>> would be mapped to:
>> SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);
>> and just use the mbstowcs() standard C lib function to convert current
>> charset to wide char...
>> WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws
>> is 2 bytes.
> Right, that means you can't even use mbtowc() to do the conversions, because
> your wchar_t internal representation no longer matches the output format of
> *either* of your APIs, so... what's the point in using it as the internal
Ok forget it.
More information about the FreeTDS