[freetds] FreeTDS and UTF-8 support
Sebastien FLAESCH
sf at 4js.com
Thu Jul 24 03:16:38 EDT 2008
Steve Langasek wrote:
> On Wed, Jul 23, 2008 at 01:59:54PM -0400, James K. Lowden wrote:
>> Sebastien FLAESCH wrote:
>>>> I see no reason support multibyte characters. I have no problem
>>>> converting to UCS-2, UTF-8, etc. But I see no value in using
>>>> pre-iconv functions or supporting weird encodings. Unicode is making
>>>> them obsolete.
>> ...
>>> So the idea is to make FreeTDS support multi-byte characters sets, even
>>> if it's just UTF-8:
>
>> Hi Sebastien,
>
>> We agree; we're just tripping over terminology. By "multibyte characters"
>> I meant pre-Unicode encodings, such as input to mbtowc(3). I'm happy to
>> support UTF-8.
>
> FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc() from
> glibc.
>
That's true.
I think FreeTDS could support all single-byte or multi-byte character sets if:
1) The application (main) calls setlocale() to define the current charset for
libc functions like mbtowc, wctomb, mblen (that would include UTF-8 support).
2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the size
in bytes of the characters (= no byte-based parsers/lexers).
3) Before sending on TDS protocol, convert from current charset to WideChar
(UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses, convert from
WideChar to current charset with wcstombs().
...
From my understanding, a C application on UNIX can be written in:
A- Pure ASCII (LC_ALL=POSIX)
B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
D- Wide Char (UTF-16/UCS-2), using the wchar_t type.
Ideally, FreeTDS should allow all of that... no?
In cases A, B, C, FreeTDS can determine the current charset with nl_langinfo().
Maybe FreeTDS could work in wchar_t internally and do charset conversion
at the ODBC API level? I think this is how MS ODBC drivers work... no?
I mean (casts are just there for understanding):
SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset", SQL_NTS);
would be mapped to:
SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);
and just use the mbstowcs() standard C lib function to convert current charset
to wide char...
WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws is 2 bytes.
?
Cheers,
Seb
More information about the FreeTDS
mailing list