FreeTDS Development Group

["James K. Lowden" <jklowden AT schemamania.org>]

On 30 Mar 2003 22:44:00 +0200, Frediano Ziglio <freddyz77 AT tin.it> wrote:
> Il dom, 2003-03-30 alle 22:17, James K. Lowden ha scritto:
> > > 
> > > Yes, but wire can be unicode and single-byte, this is the reason for
> > > 4 iconv_t. 
> > 
> > It doesn't matter.  Don't think in terms of single/double byte, think
> > in terms of client/server.  For any session, there are only two
> > character sets: client and server.  We should not care which one needs
> > more bytes/char, except for buffer allocation.  
>
>   create table #tmp (a varchar(40), b nvarchar(40))
>   ... fill table ...
>   select * from #tmp
> this last select return both single-byte and unicode
> 
> mssql2k
>   create table #tmp (a varchar(40) collate SQL_Latin1_General_CP1_CI_AS,
> b varchar(40) collate Japanese_CI_AI_KS_WS, c nvarchar(40)
>   ... fill table ...
>   select * from #tmp
> this return 2 single-byte with different encoding and a unicode column

Acknowledged.  You're saying that, even though SQL Server delivers
nvarchar columns as USC-2, a given result set may combine single- and
multi-byte character sets, and even several single-byte encodings.  OK.  

If, for a result set, we want to convert all varchar/nvarchar columns to
the client charset, we need a TDSICONVINFO structure per column.  Or, at
least, we need a pair of iconv conversion descriptors per server-client
charset pair.  

[Breathe.  That was a lot to pack into two sentences.]

It's not obvious to me that we *should* convert single-byte character sets
from SQL Server 2000.  If the server is going to send several columns
using different charsets, it seems to me there must be reason, and it's
well possible we'd defeat the purpose by reducing them all to a single
client character set.  

Is there any way in ODBC to convey the encoding of a column to the client?
 AFAIK, it's not possible in db-lib.  Perhaps the choice should be
API-dependent.  

> > If !HAVE_ICONV or iconv_open fails, we initialize the iconv_t members
> > to NULL.  
> 
> You discovered a bug...

Not afaik.  I was describing my proposed behavior....

> In tds7_unicode2ascii we have 
> 
>                       /* skip one UCS-2 sequence */
>                       in_ptr += 2;
>                       in_bytes -= 2;
> 
> this is not so easy to implement for all charset ...

Understood.  

> > > > If we need to estimate how big a buffer to use in conversion, 4
> > > > bytes/char is always safe for the foreseeable future.  
> > > 
> > > Yes, but it's a waste of space. Every column in row occupy 4 times. 
> > 
> > OK, granted.  Here's my proposed function prototype:
> > 
> >     int bytes_per_char(const char charset_name[]);
> > 
> > That will require strcmp(3) on a lookup table, which isn't very
> > efficient.
> >  We should probably cache the results in TDSICONVINFO for *both*
> >  server
> > and client charsets.  
> > 
> 
> You can compute a digest of charset and do a switch...

Wonderful idea.  

> > A different approach would be to cache the client:server byte/char
> > ratio. IOW, if client and server are both 1 byte/char, the ratio is
> > 1:1 === 1.0. If the client in Latin1 and the server is UCS-2, the
> > ratio is 1:2 === 0.50.  Etc.  That way, given a buffer and knowing
> > which direction you're going, you know how much to multiply its size
> > by to get the output buffer size.  
> > 
> 
> This is the reason for bytes_per_char in tdsiconvinfo structure...

Yes, except bytes_per_char is only one side of the equation.  We assume
UCS-2 on the server side if iconv is involved.  My notion divides client
char/byte by server char/byte to arrive at a generalized ratio.  I'm not
sure this is better or handier than keeping two ratios, one each for the
client and server.  

--jkl