FreeTDS Development Group

[Brian Bruns <camber AT ais.org>]

On Mon, 2 Sep 2002, James K. Lowden wrote:

> On 02 Sep 2002 21:04:25 +0200, "Frediano Ziglio" <freddyz77 AT tin.it> wrote:
> 
> > Mac OS X use ucs4 ?? What waste of space...
> 
> Frediano, it's only a "waste" if it's wasted.  UCS-2 is insufficient for,
> say, ancient Chinese texts.  UCS-4 exists to address that kind of problem.
>  I admit, the typical Chinese scholar is not connecting his Mac to Sybase
> to view his texts....

UTF-8 is absolutely beautiful in this regard, single byte for the common 
case (admitedly a western view), and extends to cover the full range.  
But you give any correlation between size and characters.  

UCS-2 is like the worst of both worlds, it's two bytes all the time and if 
you happen to be chinese and want to express yourself with language 
exceeding the vocabulary of the average newspaper, tough luck.

UCS-4 balloons up the size by a whopping factor of 4, but has room for all 
comers.
 
Pick your poison!

> > > Another thing for me is that I'd prefer to avoid any conversion in 
> > > freeTDS, or at least have it at the client level. Right now, it comes 
> > > on the wire as unicode, iconv converts it to latin1, and then my 
> > > application converts it back to unicode.
> > > 
> > > If conversion were at the client level, I could get the unicode via 
> > > dbdata(). As it is, that doesn't work since the iconv support is at 
> > > the tds layer. Utf8 or ucs2 support would be better, of course.
> > > 
> > 
> > Disagree. Too work for even a simple program that want just to use
> > ascii...
> > No conversion at all mean that client must expect data in
> > single/multiple bytes with dozen of character sets...
> 
> No conversion might be what the client wants.  
> 
> I think Brian's direction (and yours, I think) makes the most sense: 
> Leave iconv() in libtds, so that 8859-x and UTF-8 clients can communicate
> with UCS-2 servers.  Also, make its use optional (in 0.61): if the client
> wants exactly what the server is sending, give it to him!  Instead of dual
> conversion with an internal format, have zero or one conversions, as
> requested by the client. 
> 
> That means making FreeTDS capable of passing UCS-2 (or even UCS-4, for
> that matter).  I was going to try to avoid that, but I was wrong.  :)  In
> so doing, there's one thing I would like us to avoid:  Any "if Unicode X
> else Y" sort of code.  I'm pretty sure you agree with me on that point. 
> As data are plucked off the wire, the byte and character counts should be
> set.  After that, there's no reason to re-examine the encoding.  

There are two basic places where we care about unicode vs. none. 

The first is int tds_get_string().  While this is dependant on the 
protocol version and not on column type there is a branch here.

Secondly, if we are going to support character sets in which 0-9 and . are 
not in their ascii positions the convert code will have to know whether to 
call iconv or not.  I don't see anyway around this....open to suggestions 
though.

> Regards, 
> 
> --jkl
> 

Brian