FreeTDS Development Group

[jklowden AT schemamania.org]

On Thu, Apr 29, 2010 at 11:02:36PM -0500, Craig A. Berry wrote:
> 
> On Apr 20, 2010, at 2:33 PM, jklowden AT schemamania.org wrote:
> 
> >On Tue, Apr 20, 2010 at 01:06:38PM -0700, Vinh Nguyen wrote:
> >>On Tue, Apr 20, 2010 at 11:00 AM,  <jklowden AT schemamania.org> wrote:
> >>>>So what exactly is the issue again? ?Is there a difference  
> >>>>betweenOn Tue, Apr 20, 2010 at 01:06:38PM -0700, Vinh Nguyen wrote:
> >>Issuing your command with tds version 8.0 specified I get:
> >>config.c:202:         server_charset = iso_1
> >>config.c:216:         client_charset = ISO-8859-1
> >>
> >>When I specify "client charset = UTF-8" in my .freetds.conf file, I  
> >>get:
> >>
> >>config.c:479:   client charset = 'UTF-8'
> >>config.c:526:tds_config_login: client charset is UTF-8.
> >>config.c:202:         server_charset = iso_1
> >>config.c:216:         client_charset = UTF-8
> >>
> >>However, with this, when I use tsql I get:
> >>
> >>locale is "en_US.utf-8"
> >>locale charset is "utf-8"
> >>Default database being set to IBADataDictionary
> >>Msg 20017, Level 9, State -1, Server OpenClient, Line -1
> >>Unexpected EOF from the server
> >
> >I think there might be a bug.  I have a Mac but I won't be able
> >to look at it for the next few days and I don't want you to
> >wonder what happened.
>
> There are two or three bugs.  It's reporting 'client charset name ""  
> unrecognized' because it's pulling the name from a structure where the  
> name doesn't exist in the case of not being able to find a canonical  
> name.  That's an easy fix -- patch attached.
> 
> But the real damage is already done during the configuration phase and  
> easily reproducible by setting LANG.  
...
> 
> The relevant bit of tsql.c is:
> 
>       /* process all the command line args into the login structure */
>       populate_login(login, argc, argv);
> 
>       /* Try to open a connection */
>       tds = tds_alloc_socket(context, 512);
>       assert(tds);
>       tds_set_parent(tds, NULL);
>       connection = tds_read_config_info(tds, login, context->locale);
> 
> where populate_login() retrieves the locale and puts the charset in  
> the login structure.  When we pass that structure to  
> tds_read_config_info(), it correctly finds the value of the charset in  
> freetds.conf, but then the last two things it does before printing its  
> status are:
> 
>         /* Override config file settings with environment variables. */
>         tds_fix_connection(connection);
> 
>         /* And finally apply anything from the login structure */
>         tds_config_login(connection, login);
> 
> So in our case what's in the login structure that we passed in trumps  
> what's in the configuration file.  It looks like an environment  
> variable would do the same thing.  That seems to be by design and  
> probably best left alone.

Hi Craig, 

First of all, thanks for looking at this and for the
patches.  I'm sure there are more thanks out there, even if you
can't hear them.  

Second I want to clarify what we want to happen.  Setting tsql
aside for the moment, the library should find its client
encoding from the following, in declining order of precedence:

1.  API call e.g. DBSETLCHARSET().
2.  freetds.conf. 
3.  nl_langinfo(3) /* query using CODESET parameter */
4.  setlocale(3) /* query using NULL parameter, split on "." */
5.  ISO 8859-1

None of this works today, as you know.  Not everyone who's worked
on this project has understood the issues as clearly I as do now.
 Including me.  

tsql does what any application should do to set the C standard
library's locale:

        setlocale(LC_ALL, "");

(I don't think FreeTDS should call setlocale(3) automagically
in e.g. dbinit().  If the application calls setlocale(3) before
dbinit(), we don't want dbinit() to override the prior call.)  

That's all tsql should need to do.  To glean the encoding, libtds
should (but does not) 

        #if HAVE_NL_LANGINFO && defined(CODESET)
                if (!charset)
                        charset = nl_langinfo(CODESET);
        #else
                locale = setlocale(LC_ALL, NULL);
                /* ... and strtok(3) etc. to set the
                 * charset because the string is likely 
                 * to be in the form
                 *      lang_region.encoding, e.g. en_US.UTF-8.
                 */
        #endif

On the assumption that most applications would not call
setlocale(3) first, tds_get_locale() interrogates $LANG directly.
 I now think that is a mistake.  If the application cannot be
changed, the admin can (er, should be able to) set the encoding
in freetds.conf.  

So, you see, I don't think there's a problem with tsql; the
problem lies in the encoding defaults and overrides.  

The order of initialization should be something like the above
list in reverse, with later calls overwriting prior results:

1.  ISO 8859-1 in tds_alloc_locale()
2.  setlocale(3) next in tds_alloc_locale()
3.  nl_langinfo(3) next in tds_alloc_locale()
4.  freetds.conf in tds_parse_conf_section()
5.  accept e.g. DBSETLCHARSET() in tds_config_login()

> I think the right fix is going to be to reorder things in tsql.c so it  
> only puts a charset in the login structure when the user specifies one  
> explicitly on the command line or, as a last chance, puts the charset  
> in the connection structure if it's empty after the call to  
> tds_read_config_info().

I think your -J should just set the login structure
the way DBSETLCHARSET() does.  Then we need some TLC for
tds_alloc_locale(), and some debugging to verify the assignment
sequence.  

BTW, any work I've done with this has been in the blind because
the version of NetBSD I used had no locale support to speak of. 
Latterly it does and I haven't upgraded, being an ASCII kind of
guy.  But my RHEL box is set up with UTF-8 and I suppose its
outputs are pretty typical.  

$ cat locale.C && make locale && ./locale
#include <locale.h>
#include <iostream>
#include <fstream>
using namespace std;

int
main( int argc, char *argv[] ) 
{
        setlocale(LC_ALL, "");
        
        cout << "setlocale: " << setlocale(LC_ALL, NULL) << endl;
        cout << "nl_langinfo: " << nl_langinfo(CODESET) << endl;

        return 0;
}
make: `locale' is up to date.
setlocale: en_US.UTF-8
nl_langinfo: UTF-8

Regards, 

--jkl