FreeTDS Development Group

[James K. Lowden <jklowden AT speakeasy.org>]

On Fri, 30 Aug 2002 17:56:10 -0400 (EDT), "Brian Bruns" <camber AT ais.org>
wrote:
> 
> The double conversion bothers me a bit.  I think we have a couple basic 
> situations.

Me too.  As a practical matter, there are hundred of entry points from the
client side; every API call will need some sort of client->utf8
prolog/epilog.  But let me noodle it around with you, in case we can learn
something.  

[accurate 123 analysis ommitted...]

> I guess my point is that FreeTDS doesn't care what is inside the string 
> after it is initially read.  I understand freddy's concern about the 
> non-deterministic length of utf-8 strings, and i agree, but considering 
> we can't convert the world beyond us to ucs-2 (nor should we, it's very 
> limiting for asian languages), it's pay me now or pay me later.
> 
> I think pure utf-8 is going to add complexity to the code instead of 
> remove it, given our requirements.  

It will add steps to the processing (to do the conversions), but the steps
will have fewer branches (testing the character set).  

> We could easily make utf-8 the default 
> conversion, and that would be fine for us english speakers (utf-8 being 
> compatible in the range 0-127), but I think we'll be pissing off 
> any european who wants to use a circumflex, umlaut, or accents when
> their non-unicode aware app prints garbage by default.

See, that's the thing.  It all depends on what you want to offer the
client, and what work you're willing to put yourself through.  

Single (or no) conversions work, as long as the client accepts a single
byte character set or UTF-8.  Because, as you say, the FreeTDS code can
easily be made safe for UTF-8.  And, as you say, FreeTDS doesn't care what
glyphs you associate with what bytes, as long as you stop at null.  

I find myself asking, Do we want to support Unicode clients, or any other
kind of multibyte character set client?  I think your answer is No, and
for the time being that might be good enough.  We should recognize,
though, that what we're really doing is imposing UTF8->UCS2 (or whatever)
on the client; he can have any character set he likes from us, as long as
it's black^W single-byte.  

There *are* Unicode clients out there.  They don't work with FreeTDS very
well yet, so they're not on our radar.  Java is one, right?  XML can
definitely be in Unicode, http://www.w3.org/TR/REC-xml#charencoding.  

I also remember a fellow named Jan, a Czech, I think, who ran afoul of our
UTF-8 shortcomings (I suspect; the problems were never resolved.)  If we
implemented dual conversion, *everyone* could be run UTF-8 internally, and
we'd have no problem supporting it.  No current developer afaik has tools
to exercise the UTF-8 work.  

> We spent a long time on the subject last time around, and I for one was 
> pretty happy with the conclusion from both the perspective of how clean
> to the code to implement it was, and the functionality provided by it.

Me too.  I'm better informed now than I was then, which I hope will mean
this discussion is shorter. :)  Let me propose another arrangement that I
think is both feasible and good.  

There are three encodings: wire, internal, client.  Therefore, two
possible conversions: wire->internal, and internal->client.  

1.      Step 1 is where we almost are: to convert Unicode wire to any
single-byte encoding, for internal & client use.  In practice, that means
ASCII, ISO-8859-x, or UTF-8.  

2.      Introduce the concept of internal->client conversion.  If
internal==client, the conversion is a no-op.  This allows gradual
introduction of tds_iconv() calls in the client libraries.  Maybe just do
ODBC.  

3.      When ODBC, say, is completely ready (every API call prepared for
internal->client conversion), announce FreeTDS supports multibyte client
character encodings.  Listen very carefully for applause.  

Does that strike you as at least unharmful?  I think that if Frediano
wants to have Unicode clients, that's one way he could do it.  

There are a lot of useful things we can do with FreeTDS, and I agree with
you that UFT-8 to the client is high on the list.  I'd just like to kick
around this framework for a bit, if you dont' mind, to see where it takes
us.  

OK, it's back to the appendices for me.  

Regards, 

--jkl