FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

Steve Langasek wrote:
> On Sat, Apr 01, 2006 at 01:32:13PM -0500, James K. Lowden wrote:
> 
> > +   if ((s = nl_langinfo(CODESET))) {
> 
> > Will s always be valid/useful when it isn't NULL?  What if the string
> > is 'CCCCCCCC', or somesuch, as it often is?  Should that then
> > supersede locales.conf?  
> 
> SUSv2 defines nl_langinfo() to return:
> 
>   In a locale where langinfo data is not defined, nl_langinfo() returns
>   a pointer to the corresponding string in the POSIX locale. In all
>   locales, nl_langinfo() returns a pointer to an empty string if item
>   contains an invalid setting.
> 
>   <http://www.opengroup.org/onlinepubs/007908799/xsh/nl_langinfo.html>
> 
> Under what circumstances/on what platform do you see
> nl_langinfo(CODESET) returning 'CCCCCCCC'?  That seems an unusual name
> for a codeset.  

tsql displays locale and codeset strings because I wanted to know what
people would see.  Of 14,371 messages, here's a report that might interest
you, too.

Locales:
$ find . | xargs grep 'locale is ' |perl -ne'/("[[:alnum:] ._-]+")/; print
$1, $/' | sort | grep -Ev '^$|locale' | uniq -c | sort -n
   1 "646"
   1 "Swedish_Sweden.1252"
   1 "en_US.iso885915"
   1 "it_IT.UTF-8"
   1 "pt_BR"
   2 "it_IT"
   3 "cs_CZ"
   3 "de_DE.UTF-8"
   3 "el_GR"
   4 "no_NO"
   7 "C C C C C C"
   7 "pt_PT.UTF-8"
  36 "en_US.UTF-8"
  61 "en_US"
 156 "C"

Codesets:
$ find . | xargs grep 'charset is ' |perl -ne'/("[[:alnum:] ._-]+")/;
print $1, $/' | sort | grep -Ev '^$|locale' | uniq -c | sort -n
   1 "C C C C C C"
   1 "en_US"
   1 "pt_PT.UTF-8"
   2 "C"
   3 "ISO-8859-2"
   4 "ISO-8859-7"
   4 "ISO8859-5"
   7 "roman8"
  10 "ASCII"
  13 "US-ASCII"
  15 "ISO-8859-15"
  17 "ISO8859-1"
  54 "UTF-8"
  58 "ANSI_X3.4-1968"
  65 "ISO-8859-1"
  72 "646"

Lots of systems are set up for ASCII or similar.  I would bet some of them
need to override that setting.  

> It's true
> that SUSv2 doesn't seem to require that nl_langinfo(CODESET) be a valid
> encoding name recognized by iconv_open(); I'm certain this is the case
> with glibc, but it doesn't seem to be guaranteed by any standards.

(On NetBSD a few years ago, we had no iconv support in libc.  The string
returned by nl_langinfo(3) didn't match what GNU's iconv_open() expected. 
We have a similar problem with Solaris and OS X.)  

Sometimes the host OS lacks a native iconv library of nl_charset(3).  Even
if available, it might not be correctly configured.  Even if correctly
configured, it might not be appropriate to an application.  Imagine a
webserver, for instance, configured for POSIX or ISO 8859-1, but that
wants its web pages encoded as UTF-8.  

So the string I get from nl_langinfo, even if it meets all technical
standards, might not be all that useful.  So we need a way to say "ignore
whatever nl_langinfo() says".  We can do that either by pointing to
another source (i.e. locales.conf) or, as now, by specifying the charset
directly.  The latter is easier to understand and more generally useful,
so I agree we should stick with it.  The Right Thing to do with a system
with a broken/misconfigured nl_langinfo() is to fix the system, not rely
on elaborate overrides in FreeTDS.  

> But this is precisely why the latest version of my patch allows you to
> specify an alternate charset for a given locale, overriding the one
> derived from the environment.  Any value retrieved from nl_langinfo()
> takes precedence over a client charset specified in a [default] section
> of locales.conf, and any per-locale setting specified in locales.conf
> takes precedence in turn over the system-provided value.
...
> The
> actual order implemented by the patch is:
> 
> - per-database client charset setting from /etc/freetds/freetds.conf
> - per-locale client charset setting from /etc/freetds/locales.conf
> - locale-derived client charset setting
> - default client charset setting from /etc/freetds/locales.conf
> 
> Do you consider this reasonable?  I'm not bothered if locale-derived
> settings end up at the bottom, I'll just keep an empty locales.conf in
> that case anyway. :)  

Yes, on further consideration, I think it's reasonable.  

I don't want the user to putz with locales.conf except to extend it; I
want it to be a reference database, something we consult to infer a
charset from a locale string when we have no better guidance i.e., when
nl_langinfo(3) is absent or fails.  I would not want it to be the case
that you could *improve* behavior by deleting or emptying) it. 

> I think in general, overriding the client charset on a
> per-database level is going to be more important than overriding on a
> per-locale level anyway, since people are likely to connect to lots of
> databases using, e.g., a single apache process with a locale of C and
> want to have different client charset handling for each.

You say you understand why someone would want per-server encodings, but I
actually don't.  After all, it's the same client connecting to every
database, still configured the same way, using the same fonts, etc.  I
don't see how the client's charset requirements depend on the server it's
connected to.  

All to say: I think I agree with you.  Whether or not we need per-server
overrides, we have them.  Basically, these things should hold:

1.  freetds.conf overrides all else.
2.  If nl_langinfo(CODESET) is available and working, we use that.
3.  Otherwise, we look to locales.conf. 

To use your terms:

 - per-database client charset setting from /etc/freetds/freetds.conf
 - [global] client charset setting from /etc/freetds/freetds.conf
 - locale-derived client charset setting
 - per-locale client charset setting from /etc/freetds/locales.conf
 - default client charset setting from /etc/freetds/locales.conf

I like that.  If that's what you patch does, we should use it.  Thank you
for providing it.  

One problem remains: whether or not nl_langinfo works depends on whether
or not the application called setlocale(3), but we have no per-application
or per-invocation control.  (Or should libtds call it?? Sometimes??)

Frediano said:
> I think we all agree to have client_charset in TDSLOCALE.

IMO, no. To override the system's locale settings, use freetds.conf. To
change settings per application, keep several versions of freetds.conf and
use $FREETDS to select the right one. 

Are we all on one page?  Does the patch do what we want?  

Regards, 

--jkl