Skip to Content.
Sympa Menu

freetds - Re: [freetds] iconv structure and function names

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Frediano Ziglio <freddyz77 AT tin.it>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] iconv structure and function names
  • Date: 30 Mar 2003 22:44:00 +0200

Il dom, 2003-03-30 alle 22:17, James K. Lowden ha scritto:
> Hi Freddy,
>
> On 30 Mar 2003 21:40:44 +0200, Frediano Ziglio <freddyz77 AT tin.it> wrote:
> > Il dom, 2003-03-30 alle 21:06, James K. Lowden ha scritto:
> > > typedef struct tdsiconvinfo
> > > {
> > > char client_charset[64];
> > > char server_charset[64];
> > > iconv_t to_wire; /* conversion from client charset to server's
> > > format
> > > */
> > > iconv_t from_wire; /* conversion from server's format to client
> > > charset
> > > */
> > > } TDSICONVINFO;
> > >
> >
> > Yes, but wire can be unicode and single-byte, this is the reason for 4
> > iconv_t.
>
> It doesn't matter. Don't think in terms of single/double byte, think in
> terms of client/server. For any session, there are only two character
> sets: client and server. We should not care which one needs more
> bytes/char, except for buffer allocation.
>

Well
mssql7
create table #tmp (a varchar(40), b nvarchar(40))
... fill table ...
select * from #tmp
this last select return both single-byte and unicode

mssql2k
create table #tmp (a varchar(40) collate SQL_Latin1_General_CP1_CI_AS,
b varchar(40) collate Japanese_CI_AI_KS_WS, c nvarchar(40)
... fill table ...
select * from #tmp
this return 2 single-byte with different encoding and a unicode column

> > Also bytes_per_char is not still used but required for
> > allocating row...
>
> I'll write a function that takes an iconv charset name and returns
> bytes/char pessimisitically (i.e., 4 is the default).
>

good!

> > > If either iconv_t is NULL, that's the equivalent of use_iconv == 0.
> >
> > iconv_t is not a pointer, use (iconv_t)-1 (like file handle...)
>
> If !HAVE_ICONV or iconv_open fails, we initialize the iconv_t members to
> NULL.
>

You discovered a bug...

> > > Then, instead of
> > >
> > > char* tds7_ascii2unicode(TDSSOCKET * tds, const char *in_string,
> > > char *out_string, int maxlen);
> > >
> > > and its complement, I propose:
> > >
> > > size_t tds_iconv (iconv_t cd, const char *input, size_t * input_size,
> > > char *out_string, size_t maxlen);
> > >
> >
> > Yes and not. To skip a character (required to handle inconvertible
> > characters without truncating) you should know how to skip it.
>
> tds_iconv requires a loop, just as tds7_ascii2unicode() & Co. do now. Is
> that what you mean?
>

In tds7_unicode2ascii we have

/* skip one UCS-2 sequence */
in_ptr += 2;
in_bytes -= 2;

this is not so easy to implement for all charset ...

> > > If we need to estimate how big a buffer to use in conversion, 4
> > > bytes/char is always safe for the foreseeable future.
> >
> > Yes, but it's a waste of space. Every column in row occupy 4 times.
>
> OK, granted. Here's my proposed function prototype:
>
> int bytes_per_char(const char charset_name[]);
>
> That will require strcmp(3) on a lookup table, which isn't very efficient.
> We should probably cache the results in TDSICONVINFO for *both* server
> and client charsets.
>

You can compute a digest of charset and do a switch...

> A different approach would be to cache the client:server byte/char ratio.
> IOW, if client and server are both 1 byte/char, the ratio is 1:1 === 1.0.
> If the client in Latin1 and the server is UCS-2, the ratio is 1:2 ===
> 0.50. Etc. That way, given a buffer and knowing which direction you're
> going, you know how much to multiply its size by to get the output buffer
> size.
>

This is the reason for bytes_per_char in tdsiconvinfo structure...

> Regards,
>
> --jkl

bye
freddy77





Archive powered by MHonArc 2.6.24.

Top of Page