[freetds] FreeTDS and UTF-8 support

Sebastien FLAESCH sf at 4js.com
Wed Jul 23 12:12:05 EDT 2008

Hello James and thanks for your reply.
Comments below.

James K. Lowden wrote:
> ZIGLIO, Frediano, VF-IT wrote:
>>>  From a ANSI/POSIX C programming point of view, the program 
>>> *must* set the current
>>> locale with the setlocale() function. This defines the locale 
>>> for the whole program,
>>> for functions such as mblen(), wcwidth(), mbtowc(), wctomb()....
>>> This is the standard, from our point of view.
> I see no reason support multibyte characters.  I have no problem
> converting to UCS-2, UTF-8, etc.  But I see no value in using pre-iconv
> functions or supporting weird encodings.  Unicode is making them obsolete.

Actually we are Informix 4gl compatible. We don't write final applications:
Our product is a compiler and runtime system (VM), and we have customers
writing database applications in 4gl.

In the Informix 4gl language (and Informix database BTW), you can choose
the SBCS or MBCS you want: It will be ISO-8859-1 for most SBCS apps, and
can be UTF-8 for recent UNICODE apps, because Informix supports UTF-8.

There is a huge amount of legacy code out their we want to target, using
this or that charset, and we cant' ask customers to move to another
charset... except of course if it's really needed for I18N. But as long
as the customer is happy with ISO-8859-1, we go with that.

We could actually support the conversion to leave 4gl sources in charset X
and use UTF-16 internally, but this is a major re-design for us.

So the idea is to make FreeTDS support multi-byte characters sets, even
if it's just UTF-8:

Maybe you know that UTF-8 is the default locale for recent UNIX/Linux

So if one writes an ODBC application on a recent Linux, if the app uses
non-ASCII chars, you get UTF-8 strings in SQL text or CHAR/VARCHAR params.

If TDS protocol supports UCS-2, I believe it's really not a big deal to
convert UTF-8 to/from UCS-2 in FreeTDS, according to nl_langinfo().

You can find some conversion libs under GNU license like libutf8 ...

Just ask yourself:

"I am on Unix/Linux. I want to write a UNICODE application, so I use
UTF-8 because this seems to be the standard on UNIX now, (and I am free
to choose that one anyway). What options do I have if I want to use
FreeTDS + SQL Server/Sybase?"

> In supporting wide characters, I'm concerned about e.g. strlen(3).  I
> don't want to see code like:
> 	if (!wide_encoding) {
> 		memcpy(dest, src, len);
> 	} else {
> 		wmemcpy(dest, src, len);
> 	}
> Instead, we need a data structure like this for all internal
> representations:
> typedef struct string_data_t {
> 	union {
> 		wchar_t *ws;
> 		char *s;
> 	}
> 	size_t size;
> 	mumble encoding;
> Then we can define:
> 	TDS_STRING * tds_strcpy( TDS_STRING *dest, 
> 	                         const TDS_STRING *src);
> If we're going to continue this discussion, I prefer to use the mailing
> list where others can contribute and learn.  
> Regards, 
> --jkl

More information about the FreeTDS mailing list