FreeTDS Development Group

["Brian Bruns" <camber AT ais.org>]

Problem is, if we haul around ucs-2 then conversion becomes one thing for
thing for TDS4/5 and something else for TDS7/8, plus we may have lost the
connection info by that point (a direct call to dbconvert/cs_convert). 
Despite it's worts you can still call strlen() on utf8 which is a huge win
in my book.

I think the size of allocation argument is a non-starter.  You eventually
have to do the conversion somewhere, and therefore you're merely moving
the problem.
I think we should optimize for the common case which is single byte
charset at the moment.  Besides the same argument of alloc'ing 12000 vs
8000, could be used to say with ucs-2 vs ascii it's 8000 vs. 4000, but I
don't think that gets us anywhere.

Upfront conversion to single byte works for probably 95% of the people out
there, it's fast and clean.  UCS-2 does the other 5% no good anyway since
they really want utf-8 in the end. (Well, there's probably some small
contingent that wants ucs-2...).  I just don't see where keeping ucs-2
longer than we have to is  beneficial for us.

Sorry for the long-winded nature of this, but it's pretty core to the
system, and I think it needs lots of discussion before we make any moves. 
I'd love to be corrected if any of my assumptions are wrong.

Brian 

> > > This is true. But is not the same question...
> > > I don't like de-unicoding data. Data may lost... as I 
> > suggest in a previous
> > > mail late conversion is better.
> > > If I write (in mssql7+) a column name in chinese (well, I 
> > don't still know
> > > chinese..) there is no way to simply extract only this 
> > field from table (for
> > > example) in our current implementation...
> > 
> > I think UTF-8 is the way forward on this.  The basic problem 
> > is what do 
> > the unix apps support?  It sure isn't UCS-2.  If they support unicode
> > at all it's utf-8 and ucs-2 to utf-8 can be done losslessly.  
> > And then 
> > there are crippled clients that need single byte char sets, I 
> > can't see 
> > removing this for some years to come.
> > 
> 
> The problem with utf8 is that conversion from ucs2 -> utf8 require at max 3
> bytes per 2 bytes. So if I return a nchar(8000) we should allocate 12000
> bytes to store this type. If we store just ucs2 coding we can allocate only
> 8000 bytes. Another solution should use variable buffer for nchar/nvarchar,
> calculate required uft8 length and than convert...
> Samba solution is to use always utf8 coding for internal strings...
> 
> ...
> > > > 
> > > > I just need to know what I should do with the tds_convert_ntext()
> > > > function....
> > 
> > tds_convert_ntext should in theory be exactly like tds_convert_text.
> > 
> 
> unixODBC have function to support wide character (I suppose they use ucs2).
> I know that gnome2 and kde3 support unicode much better than before. What
> coding they use for unicode? utf8 (usually unix), ucs2 (windows) or
> ucs4/utf4 (c wide char in linux) ?
> 
> ...
> > 
> > Brian
> > 
> freddy77
> 
> =================================
> "STRICTLY PERSONAL AND CONFIDENTIAL
> 
> This message may contain confidential and proprietary material for the sole
> use of the intended recipient. Any review or distribution by others is
> strictly prohibited. If you are not the intended recipient please contact
> the sender and delete all copies.
> The contents of this message that do not relate to the official business of
> our company shall be understood as neither given nor endorsed by it."
> 
> =================================