FreeTDS Development Group

[Frediano Ziglio <freddyz77 AT tin.it>]

Il dom, 2003-03-30 alle 22:17, James K. Lowden ha scritto:
> Hi Freddy, 
> 
> On 30 Mar 2003 21:40:44 +0200, Frediano Ziglio <freddyz77 AT tin.it> wrote:
> > Il dom, 2003-03-30 alle 21:06, James K. Lowden ha scritto:
> > > typedef struct tdsiconvinfo
> > > {
> > >   char client_charset[64];
> > >   char server_charset[64];
> > >   iconv_t to_wire;   /* conversion from client charset to server's
> > >   format
> > > */
> > >   iconv_t from_wire; /* conversion from server's format to client
> > >   charset
> > > */
> > > } TDSICONVINFO;
> > > 
> > 
> > Yes, but wire can be unicode and single-byte, this is the reason for 4
> > iconv_t. 
> 
> It doesn't matter.  Don't think in terms of single/double byte, think in
> terms of client/server.  For any session, there are only two character
> sets: client and server.  We should not care which one needs more
> bytes/char, except for buffer allocation.  
> 

Well
mssql7
  create table #tmp (a varchar(40), b nvarchar(40))
  ... fill table ...
  select * from #tmp
this last select return both single-byte and unicode

mssql2k
  create table #tmp (a varchar(40) collate SQL_Latin1_General_CP1_CI_AS,
b varchar(40) collate Japanese_CI_AI_KS_WS, c nvarchar(40)
  ... fill table ...
  select * from #tmp
this return 2 single-byte with different encoding and a unicode column

> > Also bytes_per_char is not still used but required for
> > allocating row...
> 
> I'll write a function that takes an iconv charset name and returns
> bytes/char pessimisitically (i.e., 4 is the default).  
> 

good!

> > > If either iconv_t is NULL, that's the equivalent of use_iconv == 0.  
> > 
> > iconv_t is not a pointer, use (iconv_t)-1 (like file handle...)
> 
> If !HAVE_ICONV or iconv_open fails, we initialize the iconv_t members to
> NULL.  
> 

You discovered a bug...

> > > Then, instead of 
> > > 
> > > char* tds7_ascii2unicode(TDSSOCKET * tds, const char *in_string, 
> > >                    char *out_string, int maxlen);
> > > 
> > > and its complement, I propose:
> > > 
> > > size_t tds_iconv (iconv_t cd, const char *input, size_t * input_size,
> > >                    char *out_string, size_t maxlen);
> > > 
> > 
> > Yes and not. To skip a character (required to handle inconvertible
> > characters without truncating) you should know how to skip it. 
> 
> tds_iconv requires a loop, just as tds7_ascii2unicode() & Co. do now.  Is
> that what you mean?  
> 

In tds7_unicode2ascii we have 

                        /* skip one UCS-2 sequence */
                        in_ptr += 2;
                        in_bytes -= 2;

this is not so easy to implement for all charset ...

> > > If we need to estimate how big a buffer to use in conversion, 4
> > > bytes/char is always safe for the foreseeable future.  
> > 
> > Yes, but it's a waste of space. Every column in row occupy 4 times. 
> 
> OK, granted.  Here's my proposed function prototype:
> 
>       int bytes_per_char(const char charset_name[]);
> 
> That will require strcmp(3) on a lookup table, which isn't very efficient.
>  We should probably cache the results in TDSICONVINFO for *both* server
> and client charsets.  
> 

You can compute a digest of charset and do a switch...

> A different approach would be to cache the client:server byte/char ratio. 
> IOW, if client and server are both 1 byte/char, the ratio is 1:1 === 1.0. 
> If the client in Latin1 and the server is UCS-2, the ratio is 1:2 ===
> 0.50.  Etc.  That way, given a buffer and knowing which direction you're
> going, you know how much to multiply its size by to get the output buffer
> size.  
> 

This is the reason for bytes_per_char in tdsiconvinfo structure...

> Regards, 
> 
> --jkl

bye
  freddy77