Skip to Content.
Sympa Menu

freetds - Re: [freetds] FreeTDS and UTF-8 support

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Sebastien FLAESCH <sf AT 4js.com>
  • To: freetds AT lists.ibiblio.org
  • Subject: Re: [freetds] FreeTDS and UTF-8 support
  • Date: Thu, 24 Jul 2008 09:16:38 +0200

Steve Langasek wrote:
On Wed, Jul 23, 2008 at 01:59:54PM -0400, James K. Lowden wrote:
Sebastien FLAESCH wrote:
I see no reason support multibyte characters. I have no problem
converting to UCS-2, UTF-8, etc. But I see no value in using
pre-iconv functions or supporting weird encodings. Unicode is making
them obsolete.
...
So the idea is to make FreeTDS support multi-byte characters sets, even
if it's just UTF-8:

Hi Sebastien,

We agree; we're just tripping over terminology. By "multibyte characters"
I meant pre-Unicode encodings, such as input to mbtowc(3). I'm happy to
support UTF-8.

FWIW, UTF-8 is a multibyte character set, and is understood by mbtowc() from
glibc.


That's true.

I think FreeTDS could support all single-byte or multi-byte character sets if:

1) The application (main) calls setlocale() to define the current charset for
libc functions like mbtowc, wctomb, mblen (that would include UTF-8 support).

2) Any (char*) string manipulation in FreeTDS lib uses mblen() to know the
size
in bytes of the characters (= no byte-based parsers/lexers).

3) Before sending on TDS protocol, convert from current charset to WideChar
(UTF-16/UCS-2) with mbstowcs() and when receiving TDS responses, convert from
WideChar to current charset with wcstombs().

...

From my understanding, a C application on UNIX can be written in:

A- Pure ASCII (LC_ALL=POSIX)
B- A specific single-byte charset (LC_ALL=fr_FR.iso88591)
C- A specific multi-byte charset (LC_ALL=ja_JP.eucjp or en_US.utf8)
D- Wide Char (UTF-16/UCS-2), using the wchar_t type.

Ideally, FreeTDS should allow all of that... no?

In cases A, B, C, FreeTDS can determine the current charset with
nl_langinfo().

Maybe FreeTDS could work in wchar_t internally and do charset conversion
at the ODBC API level? I think this is how MS ODBC drivers work... no?

I mean (casts are just there for understanding):

SQLPrepare(hp, (char *) "my SQL string in current SBCD or MBCS charset",
SQL_NTS);

would be mapped to:

SQLPrepareW(hp, (wchar_t *) "my SQL string in UCS-2", SQL_NTS);

and just use the mbstowcs() standard C lib function to convert current charset
to wide char...

WARNING: wchar_t on UNIX is 4 bytes while UCS-2 and wchar_t on Windonws is 2
bytes.

?

Cheers,
Seb




Archive powered by MHonArc 2.6.24.

Top of Page