Skip to Content.
Sympa Menu

freetds - Re: Unicode, UTF-8, and Greek

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Steve Langasek <vorlon AT netexpress.net>
  • To: TDS Development Group <freetds AT franklin.oit.unc.edu>
  • Subject: Re: Unicode, UTF-8, and Greek
  • Date: Mon, 16 Jul 2001 17:36:43 -0500 (CDT)


On Sat, 14 Jul 2001, Brian Bruns wrote:

> That was very insightful. I think we have three major issues here:

> 1) TDS 7/8. Everything is unicode, so we should just be able to read the
> field in freetds.conf and use iconv to convert between UCS2 and
> that. If there is no entry in freetds.conf, we could try to grab the
> value of $LANG, although I'm not sure how that maps to the iconv
> labels. Failing there we simply use the present 7bit ASCII conversion

I rather favor defaulting to UTF-8 instead of ASCII when no charset has been
specified. If there's non-ascii data in a database, it's because someone put
it there; doesn't it make more sense to treat this data losslessly by default,
passing it through for the client app to deal with?

I also wonder if it's necessary to support language settings within
freetds.conf itself. If we're going to make use of any sort of a 'default', I
think either C or an appropriate UTF-8 locale are the best choices; and
anything other than a default can be specified using either the $LANG
setting or the appropriate bits of the dblib API.

Choosing default charsets for a given language can be done using a
charset.map, as with Sybase. This seems to me to be the only portable way of
detecting per-session charset preferences, in any case; my pokings into glibc,
at least, show that glibc doesn't expose information to applications about its
own language->charset mappings. This would also apply below.

> 2) TDS 4.2/5. This gets a little complicated as FreeTDS needs to
> communicate to the server what character set to use. Problem is the label
> for this (iso_1, cp850) are not the same as the iconv labels (do
> a iconv --list to see them). I would suggest we create a 'charset.map'
> file to map sybase style charset names to the iconv ones, perhaps also
> dealing with $LANG similarly. Sybase does something similar with the
> locales.dat file.

> 3) multibyte char sets. I'm afraid I just don't know enough about usage
> of these to make a good attempt at it. We'll shelve it until a native
> user comes along. ;-)

For the most part, I think iconv will be doing all the hard work. If we have
working conversion from UCS2->UTF8 (wchar->mbyte) and UCS2->ISO-8859-x
(wchar->char), I expect we'll get working conversion to other multibyte
charsets for free.


ObApology: Debian packages for freetds 0.52 are coming along. Didn't have as
much time to start on them before the wedding as I had hoped. :)

Steve Langasek
postmodern programmer





Archive powered by MHonArc 2.6.24.

Top of Page