FreeTDS Development Group

["James K. Lowden" <jklowden AT speakeasy.org>]

Hi Brian, 

I'm very interested in seeing FreeTDS correctly deal with other
character sets.  I don't know why.  I don't read any non-latin
scripts, and I think the only application I've got that can display
non-latin scripts is Mozilla.  If did know why, I suppose I could put
that therapy money to better use.  Anyway, here goes nothing.  

My thanks to Michael Peppler for pointing out Sybase's capabilities.  I
haven't programmed to Sybase servers since System 10.  They've moved
on.  

First, let me see if I can frame the problem from a functional point of
view.  That way we'll know if I know what I'm talking about, and when
I respond to your particular points, you'll know where I'm coming from.  

I think the whole question boils down to four things:

1.      What is the server's character set?  
2.      What is the client's character set?
3.      Will the server convert the character set?
4.      Will tdslib convert the character set?  

All these things have to be known to tdslib for it to provide/pass
conversion services.  It doesn't matter whether the client uses db-lib,
ct-lib, or odbc, yea?  

I think tdslib should do (with iconv) for Microsoft servers some of the
same
sorts of conversions Sybase servers do.  When an API call arrives from
the client library, tdslib should either pass it on (to a Sybase
server) or deal with it (in front of a Microsoft server). 

The *protocol* isn't precisely at issue, IMHO.  True, if you're
talking TDS 7 these days, you can bet the server has no character set
conversion capacity, and tdslib will have to do something to
get from UCS2 to anything else.  But that's an artifact of a particular
implementation at a point in time, not an attribute of the protocol.  

The answers for a FreeTDS client talking TDS7 today are:

1.      UCS2
2.      ISO-8859-1 
3.      No
4.      sort of

Microsoft punts on the whole question.  You can have any character set
you want, as long as it's UCS2.  tdslib currently whacks off a byte and
a
bit to do a rude conversion, a very clever way to provide for an
upgrade path, I must say.  ;-)

We have to emulate Sybase's API throughout, because it addresses
character set conversion, including in particular UCS2->something. 
Via the Sybase API, the client can ascertain what conversion services
are available, and provide the information needed to engage them.
With iconv, FreeTDS can improve on what services are available,
especially when connected to a dumb server.  

Put another way: the answer to dbcharsetconv() is a logical OR of the
answers to questions 3  and 4.  

By emulating Sybase's conversion services in tdslib for Microsoft
servers, we'd give FreeTDS clients something they can't get anywhere
else at any price.  If we were smart, we'd charge Microsoft. I guess
we're not smart, though.  

We don't have to completely reproduce what Sybase does on their
server, by the way.  The first requirement is simply not to lie.  The
second is to handle USC2<->ISO-8859-1.  Third is USC2<->UTF-8.  

+++

I had been thinking about this problem in terms of protocols, vendors,
Unicode, and client libraries.  That was a mishmash, clouding the what I
now
believe are the four basic questions.  

If you agree with this characterization of the problem, then what
remains
is:

*       fully implement the Sybase API 
        (even if dbcharsetconv() returns mostly FALSE for now)
*       add character set information to tdslib
*       figure out where to get the defaults from 
*       add iconv

I can be of some use to some of this.  

Now [whew], let me answer your message.  

> The only problem with supporting UTF-8 against a sybase (12.x) server
> would be the more general problem of multibyte sets, specifically
> ct_describe() returning sane values back to the client app. Otherwise, I
> could continue to receive iso_1, greek, or whatever format I specify via
> charset login field.  I really don't think FreeTDS should have any problem
> with supporting this *today*.

Understood.  Comes under "fully implement the Sybase API".

> The TDS 7 issue is different though.  There is no charset field. 
> Everything is unicode. 

Right.  Server character set is USC2.  

> So we must convert it to something usable by the
> client app.  Today that means 7bit ascii by a simple high byte strip
> mechanism.  But, obviously, that is only a valid option for english and a
> few other languages.  (Even most western european language need the
> 128-255 range and don't work with 7.0 currently).  

Yes, iconv, of course, as we've been saying.  

> The safest thing would
> be to default to latin_1 with a freetds.conf field allowing other
> character sets.  I was leaning towards UTF-8, but that's an easy conf file
> change for non-western langs and gives a lot of flexibility to those folks
> to choose from a variety of charsets and not be pinned into UTF-8 if the
> app is not unicode-aware.  Therefore I propose a 'tds7 client charset'
> field in freetds.conf to have a value of any iconv charset label (iconv
> --list on linux to view).

I lost you there.  Sybase infers the client's character set from the
LANG and LC_ALL variables.  I'd say if they're not set you get
ISO-8859-1, else you get something reasonable based on them. 

I admit there's a *huge* failure of standardization when it comes to
just what string you'll find in LC_ALL.  But in the scheme of things,
that's not the greatest evil.  

> Another
> question becomes what to do with DBSETLCHARSET() and friends under tds
> 7.0?  one could use it as an iconv set, but then you have a problem of
> people passing in legal 4.2 sets (iso_1) and not having them work on 7.0. 

What I'm suggesting is that DBSETLCHARSET() be handled intelligently
and transparently by tdslib.  If the server handles conversion, fine,
pass it along.  If the server doesn't (TDS 7, so far), fine, make a
note and perform the conversion with iconv locally in tdslib.  

> I know OpenClient uses the locales.dat file to map en_US for instance to a
> language/charset pairing.  Maybe we need something similar for Sybase
> charset -> iconv charset mapping.

I think so.  

--jkl