Skip to Content.
Sympa Menu

freetds - Re: [freetds] charset selection [was: 0.64rc2 Debian prerelease packages available]

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "James K. Lowden" <jklowden AT freetds.org>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] charset selection [was: 0.64rc2 Debian prerelease packages available]
  • Date: Sun, 2 Apr 2006 15:25:17 -0400

Steve Langasek wrote:
> On Sat, Apr 01, 2006 at 01:32:13PM -0500, James K. Lowden wrote:
>
> > + if ((s = nl_langinfo(CODESET))) {
>
> > Will s always be valid/useful when it isn't NULL? What if the string
> > is 'CCCCCCCC', or somesuch, as it often is? Should that then
> > supersede locales.conf?
>
> SUSv2 defines nl_langinfo() to return:
>
> In a locale where langinfo data is not defined, nl_langinfo() returns
> a pointer to the corresponding string in the POSIX locale. In all
> locales, nl_langinfo() returns a pointer to an empty string if item
> contains an invalid setting.
>
> <http://www.opengroup.org/onlinepubs/007908799/xsh/nl_langinfo.html>
>
> Under what circumstances/on what platform do you see
> nl_langinfo(CODESET) returning 'CCCCCCCC'? That seems an unusual name
> for a codeset.

tsql displays locale and codeset strings because I wanted to know what
people would see. Of 14,371 messages, here's a report that might interest
you, too.

Locales:
$ find . | xargs grep 'locale is ' |perl -ne'/("[[:alnum:] ._-]+")/; print
$1, $/' | sort | grep -Ev '^$|locale' | uniq -c | sort -n
1 "646"
1 "Swedish_Sweden.1252"
1 "en_US.iso885915"
1 "it_IT.UTF-8"
1 "pt_BR"
2 "it_IT"
3 "cs_CZ"
3 "de_DE.UTF-8"
3 "el_GR"
4 "no_NO"
7 "C C C C C C"
7 "pt_PT.UTF-8"
36 "en_US.UTF-8"
61 "en_US"
156 "C"

Codesets:
$ find . | xargs grep 'charset is ' |perl -ne'/("[[:alnum:] ._-]+")/;
print $1, $/' | sort | grep -Ev '^$|locale' | uniq -c | sort -n
1 "C C C C C C"
1 "en_US"
1 "pt_PT.UTF-8"
2 "C"
3 "ISO-8859-2"
4 "ISO-8859-7"
4 "ISO8859-5"
7 "roman8"
10 "ASCII"
13 "US-ASCII"
15 "ISO-8859-15"
17 "ISO8859-1"
54 "UTF-8"
58 "ANSI_X3.4-1968"
65 "ISO-8859-1"
72 "646"

Lots of systems are set up for ASCII or similar. I would bet some of them
need to override that setting.

> It's true
> that SUSv2 doesn't seem to require that nl_langinfo(CODESET) be a valid
> encoding name recognized by iconv_open(); I'm certain this is the case
> with glibc, but it doesn't seem to be guaranteed by any standards.

(On NetBSD a few years ago, we had no iconv support in libc. The string
returned by nl_langinfo(3) didn't match what GNU's iconv_open() expected.
We have a similar problem with Solaris and OS X.)

Sometimes the host OS lacks a native iconv library of nl_charset(3). Even
if available, it might not be correctly configured. Even if correctly
configured, it might not be appropriate to an application. Imagine a
webserver, for instance, configured for POSIX or ISO 8859-1, but that
wants its web pages encoded as UTF-8.

So the string I get from nl_langinfo, even if it meets all technical
standards, might not be all that useful. So we need a way to say "ignore
whatever nl_langinfo() says". We can do that either by pointing to
another source (i.e. locales.conf) or, as now, by specifying the charset
directly. The latter is easier to understand and more generally useful,
so I agree we should stick with it. The Right Thing to do with a system
with a broken/misconfigured nl_langinfo() is to fix the system, not rely
on elaborate overrides in FreeTDS.

> But this is precisely why the latest version of my patch allows you to
> specify an alternate charset for a given locale, overriding the one
> derived from the environment. Any value retrieved from nl_langinfo()
> takes precedence over a client charset specified in a [default] section
> of locales.conf, and any per-locale setting specified in locales.conf
> takes precedence in turn over the system-provided value.
...
> The
> actual order implemented by the patch is:
>
> - per-database client charset setting from /etc/freetds/freetds.conf
> - per-locale client charset setting from /etc/freetds/locales.conf
> - locale-derived client charset setting
> - default client charset setting from /etc/freetds/locales.conf
>
> Do you consider this reasonable? I'm not bothered if locale-derived
> settings end up at the bottom, I'll just keep an empty locales.conf in
> that case anyway. :)

Yes, on further consideration, I think it's reasonable.

I don't want the user to putz with locales.conf except to extend it; I
want it to be a reference database, something we consult to infer a
charset from a locale string when we have no better guidance i.e., when
nl_langinfo(3) is absent or fails. I would not want it to be the case
that you could *improve* behavior by deleting or emptying) it.

> I think in general, overriding the client charset on a
> per-database level is going to be more important than overriding on a
> per-locale level anyway, since people are likely to connect to lots of
> databases using, e.g., a single apache process with a locale of C and
> want to have different client charset handling for each.

You say you understand why someone would want per-server encodings, but I
actually don't. After all, it's the same client connecting to every
database, still configured the same way, using the same fonts, etc. I
don't see how the client's charset requirements depend on the server it's
connected to.

All to say: I think I agree with you. Whether or not we need per-server
overrides, we have them. Basically, these things should hold:

1. freetds.conf overrides all else.
2. If nl_langinfo(CODESET) is available and working, we use that.
3. Otherwise, we look to locales.conf.

To use your terms:

- per-database client charset setting from /etc/freetds/freetds.conf
- [global] client charset setting from /etc/freetds/freetds.conf
- locale-derived client charset setting
- per-locale client charset setting from /etc/freetds/locales.conf
- default client charset setting from /etc/freetds/locales.conf

I like that. If that's what you patch does, we should use it. Thank you
for providing it.

One problem remains: whether or not nl_langinfo works depends on whether
or not the application called setlocale(3), but we have no per-application
or per-invocation control. (Or should libtds call it?? Sometimes??)

Frediano said:
> I think we all agree to have client_charset in TDSLOCALE.

IMO, no. To override the system's locale settings, use freetds.conf. To
change settings per application, keep several versions of freetds.conf and
use $FREETDS to select the right one.

Are we all on one page? Does the patch do what we want?

Regards,

--jkl




Archive powered by MHonArc 2.6.24.

Top of Page