FreeTDS Development Group

[Brian Bruns <camber AT ais.org>]

On Sun, 1 Sep 2002, James K. Lowden wrote:

> On Fri, 30 Aug 2002 17:56:10 -0400 (EDT), "Brian Bruns" <camber AT ais.org>
> wrote:
> > 
> > The double conversion bothers me a bit.  I think we have a couple basic 
> > situations.
> 
> Me too.  As a practical matter, there are hundred of entry points from the
> client side; every API call will need some sort of client->utf8
> prolog/epilog.  But let me noodle it around with you, in case we can learn
> something.  

Sounds fair.
 
> [accurate 123 analysis ommitted...]
> 
> > I guess my point is that FreeTDS doesn't care what is inside the string 
> > after it is initially read.  I understand freddy's concern about the 
> > non-deterministic length of utf-8 strings, and i agree, but considering 
> > we can't convert the world beyond us to ucs-2 (nor should we, it's very 
> > limiting for asian languages), it's pay me now or pay me later.
> > 
> > I think pure utf-8 is going to add complexity to the code instead of 
> > remove it, given our requirements.  
> 
> It will add steps to the processing (to do the conversions), but the steps
> will have fewer branches (testing the character set).  

Not sure if I agree here.  Consider what branches exist in the code today.

1. We receive coded charset in TDS 4.2/5.0 and do no conversion.  We 
support the supplying of a character set by the application or config 
file.  We have to maintain this behaviour, plus we have to figure out how 
iso_1 for instance maps to an iconv defined set via a mapping file 
probably, convert, then convert back.

2. We receive UCS-2 in TDS 7/8.  The extra branching here is a single 
conf file option and one localized conversion in read.c.  Pretty minimal 
from my standpoint.

> > We could easily make utf-8 the default 
> > conversion, and that would be fine for us english speakers (utf-8 being 
> > compatible in the range 0-127), but I think we'll be pissing off 
> > any european who wants to use a circumflex, umlaut, or accents when
> > their non-unicode aware app prints garbage by default.
> 
> See, that's the thing.  It all depends on what you want to offer the
> client, and what work you're willing to put yourself through.  
> 
> Single (or no) conversions work, as long as the client accepts a single
> byte character set or UTF-8.  Because, as you say, the FreeTDS code can
> easily be made safe for UTF-8.  And, as you say, FreeTDS doesn't care what
> glyphs you associate with what bytes, as long as you stop at null.  
> 
> I find myself asking, Do we want to support Unicode clients, or any other
> kind of multibyte character set client?  I think your answer is No, and
> for the time being that might be good enough.  We should recognize,
> though, that what we're really doing is imposing UTF8->UCS2 (or whatever)
> on the client; he can have any character set he likes from us, as long as
> it's black^W single-byte.  

I think perhaps the best solution would be to support multi-byte char 
sets, period.  Be it big5 or ucs-2 or whatever.  This is particularly nice 
for our asian users as their non-unicode aware applications are 
multi-byte.  UCS-2 could be converted to big5, and then bound to the 
application in that format.  The bigest thing here would simply be to 
remove all reference to strlen() as it is applied to items within the row 
buffer, and some fixups to conversion code (int is an int but the string 
representaion in multibyte is tricky), most likely we convert to ascii (as 
all numbers, and decimal point are ascii) using the standard unix 
functions and convert from there using iconv.  Don't get me started on 
datetimes.

> There *are* Unicode clients out there.  They don't work with FreeTDS very
> well yet, so they're not on our radar.  Java is one, right?  XML can
> definitely be in Unicode, http://www.w3.org/TR/REC-xml#charencoding.  
> 
> I also remember a fellow named Jan, a Czech, I think, who ran afoul of our
> UTF-8 shortcomings (I suspect; the problems were never resolved.)  If we
> implemented dual conversion, *everyone* could be run UTF-8 internally, and
> we'd have no problem supporting it.  No current developer afaik has tools
> to exercise the UTF-8 work.  

What's needed? unicode columns in SQL Server and a terminal that support 
utf-8 with sqsh.

> > We spent a long time on the subject last time around, and I for one was 
> > pretty happy with the conclusion from both the perspective of how clean
> > to the code to implement it was, and the functionality provided by it.
> 
> Me too.  I'm better informed now than I was then, which I hope will mean
> this discussion is shorter. :)  Let me propose another arrangement that I
> think is both feasible and good.  

Shorter, I hope so ;-)
 
> There are three encodings: wire, internal, client.  Therefore, two
> possible conversions: wire->internal, and internal->client.  
> 
> 1.    Step 1 is where we almost are: to convert Unicode wire to any
> single-byte encoding, for internal & client use.  In practice, that means
> ASCII, ISO-8859-x, or UTF-8.  
>
> 2.    Introduce the concept of internal->client conversion.  If
> internal==client, the conversion is a no-op.  This allows gradual
> introduction of tds_iconv() calls in the client libraries.  Maybe just do
> ODBC.  
> 
> 3.    When ODBC, say, is completely ready (every API call prepared for
> internal->client conversion), announce FreeTDS supports multibyte client
> character encodings.  Listen very carefully for applause.  

Like me step outside the box for a minute.  Why have *an* internal format?  
What to we gain?  FreeTDS now is flexible enough to handle a large number 
of char sets internally with no real problem, if we were to extend this to 
multibyte sets, what do we have to lose?  The code as is it fairly clean 
and maintainable wrt to conversions, picking the one true format for 
internal use seems to add more complexity than it removes, (IMNSHO of 
course ;-)
  
> Does that strike you as at least unharmful?  I think that if Frediano 
> wants to have Unicode clients, that's one way he could do it.  
> 
> There are a lot of useful things we can do with FreeTDS, and I agree with
> you that UFT-8 to the client is high on the list.  I'd just like to kick
> around this framework for a bit, if you dont' mind, to see where it takes
> us.  

UTF-8 is first on my list, but fixed length multibyte is probably not that 
much harder to handle.

> OK, it's back to the appendices for me.  
> 
> Regards, 
> 
> --jkl
> 

Brian