Skip to Content.
Sympa Menu

freetds - Re: [freetds] FreeTDS and UTF-8 support

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Sebastien FLAESCH <sf AT 4js.com>
  • To: freetds AT lists.ibiblio.org
  • Subject: Re: [freetds] FreeTDS and UTF-8 support
  • Date: Wed, 23 Jul 2008 18:12:05 +0200

Hello James and thanks for your reply.
Comments below.
Seb

James K. Lowden wrote:
ZIGLIO, Frediano, VF-IT wrote:
From a ANSI/POSIX C programming point of view, the program *must* set the current
locale with the setlocale() function. This defines the locale for the whole program,
for functions such as mblen(), wcwidth(), mbtowc(), wctomb()....

This is the standard, from our point of view.

I see no reason support multibyte characters. I have no problem
converting to UCS-2, UTF-8, etc. But I see no value in using pre-iconv
functions or supporting weird encodings. Unicode is making them obsolete.

Actually we are Informix 4gl compatible. We don't write final applications:
Our product is a compiler and runtime system (VM), and we have customers
writing database applications in 4gl.

In the Informix 4gl language (and Informix database BTW), you can choose
the SBCS or MBCS you want: It will be ISO-8859-1 for most SBCS apps, and
can be UTF-8 for recent UNICODE apps, because Informix supports UTF-8.

There is a huge amount of legacy code out their we want to target, using
this or that charset, and we cant' ask customers to move to another
charset... except of course if it's really needed for I18N. But as long
as the customer is happy with ISO-8859-1, we go with that.

We could actually support the conversion to leave 4gl sources in charset X
and use UTF-16 internally, but this is a major re-design for us.

So the idea is to make FreeTDS support multi-byte characters sets, even
if it's just UTF-8:

Maybe you know that UTF-8 is the default locale for recent UNIX/Linux
versions...

So if one writes an ODBC application on a recent Linux, if the app uses
non-ASCII chars, you get UTF-8 strings in SQL text or CHAR/VARCHAR params.

If TDS protocol supports UCS-2, I believe it's really not a big deal to
convert UTF-8 to/from UCS-2 in FreeTDS, according to nl_langinfo().

You can find some conversion libs under GNU license like libutf8 ...

Just ask yourself:

"I am on Unix/Linux. I want to write a UNICODE application, so I use
UTF-8 because this seems to be the standard on UNIX now, (and I am free
to choose that one anyway). What options do I have if I want to use
FreeTDS + SQL Server/Sybase?"


In supporting wide characters, I'm concerned about e.g. strlen(3). I
don't want to see code like:

if (!wide_encoding) {
memcpy(dest, src, len);
} else {
wmemcpy(dest, src, len);
}

Instead, we need a data structure like this for all internal
representations:

typedef struct string_data_t {
union {
wchar_t *ws;
char *s;
}
size_t size;
mumble encoding;
} TDS_STRING;

Then we can define:

TDS_STRING * tds_strcpy( TDS_STRING *dest, const TDS_STRING *src);

If we're going to continue this discussion, I prefer to use the mailing
list where others can contribute and learn.
Regards,
--jkl






Archive powered by MHonArc 2.6.24.

Top of Page