FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Craig A. Berry [mailto:craigberry AT mac.com]
> Sent: May 29, 2003 3:15 PM
> 
> In other words, shouldn't something like what 
> tds_iconv_init does for ISO-8859-1 and UTF-8 be done more generally 
> at configuration time, i.e., permute through all of the aliases until 
> we find the one that works?

Frediano hit on a general solution that works by trial and error.  The notion 
is to cache the iconv (not necessarily GNU's) name at runtime.  

How will it work?  src/tds/alternative_character_sets.h is our database of 
character set names.  On receipt of a name, look up the canonical name and 
try iconv_open(3).  If that one fails, try each alternative in turn.  If one 
succeeds, cache it for future use; it becomes the canonical name 
thenceforward.  If none succeeds, well, it's curtains for iconv.  

The static array src/tds/iconv.c::iconv_names is the cache.  It's up to 
tds_iconv_open() to find the right iconv name, using logic similar to (and 
more general than) that found in tds_iconv_init().  As you can see, that's 
not done yet.  

Other concerns in this area:

1.  To support UCS-2 clients (e.g. OS X) we have to reserve a forth member of 
the socket's iconv_info array, to hold ASCII->client conversions.  This will 
be needed for error messages.  

2.  iconv_info[client2ucs2] need to be initialized when the server tells us 
its single-byte encoding.  

3.  Some iconv() implementations emit a byte order mark (BOM) when converting 
to UCS-2.  Some don't.  Some let you control endianism.  Some don't.  I don't 
know how if or when to control or record these things.  We always want the 
same thing: little endian, no BOM.  

4.  iconv-less compilation is currently broken, pending a naive 
LATIN1<->UCS-2 implementation in src/replacements/iconv.c.  

Anyone who wants to take on any of these things is more than welcome.  

I myself got caught in the La Brea error message morass while recoding 
tds_iconv to handle unconvertible sequences reasonably.  I think I'll punt -- 
use tds_client_message() for now, even though it's patently seriously broken 
for ct-lib clients -- in order to move forward with UTF-8 clients (and 
servers).  I hope Bill Thompson decides to unify our client-side message 
system as part of his ct-lib renewal.  However, it's not clear he'll have 
that particular itch.  

I hope that answers your question. :-)

--jkl
-----------------------------------------
The information contained in this transmission may contain privileged and 
confidential information and is intended only for the use of the person(s) 
named above. If you are not the intended recipient, or an employee or agent 
responsible for delivering this message to the intended recipient, any 
review, dissemination, distribution or duplication of this communication is 
strictly prohibited. If you are not the intended recipient, please contact 
the sender immediately by reply e-mail and destroy all copies of the original 
message. Please note that we do not accept account orders and/or instructions 
by e-mail, and therefore will not be responsible for carrying out such orders 
and/or instructions.