[freetds] tds_canonical_charset_name futures?

Lowden, James K LowdenJK at bernstein.com
Thu May 29 16:24:03 EDT 2003

> From: Craig A. Berry [mailto:craigberry at mac.com]
> Sent: May 29, 2003 3:15 PM
> In other words, shouldn't something like what 
> tds_iconv_init does for ISO-8859-1 and UTF-8 be done more generally 
> at configuration time, i.e., permute through all of the aliases until 
> we find the one that works?

Frediano hit on a general solution that works by trial and error.  The notion is to cache the iconv (not necessarily GNU's) name at runtime.  

How will it work?  src/tds/alternative_character_sets.h is our database of character set names.  On receipt of a name, look up the canonical name and try iconv_open(3).  If that one fails, try each alternative in turn.  If one succeeds, cache it for future use; it becomes the canonical name thenceforward.  If none succeeds, well, it's curtains for iconv.  

The static array src/tds/iconv.c::iconv_names is the cache.  It's up to tds_iconv_open() to find the right iconv name, using logic similar to (and more general than) that found in tds_iconv_init().  As you can see, that's not done yet.  

Other concerns in this area:

1.  To support UCS-2 clients (e.g. OS X) we have to reserve a forth member of the socket's iconv_info array, to hold ASCII->client conversions.  This will be needed for error messages.  

2.  iconv_info[client2ucs2] need to be initialized when the server tells us its single-byte encoding.  

3.  Some iconv() implementations emit a byte order mark (BOM) when converting to UCS-2.  Some don't.  Some let you control endianism.  Some don't.  I don't know how if or when to control or record these things.  We always want the same thing: little endian, no BOM.  

4.  iconv-less compilation is currently broken, pending a naive LATIN1<->UCS-2 implementation in src/replacements/iconv.c.  

Anyone who wants to take on any of these things is more than welcome.  

I myself got caught in the La Brea error message morass while recoding tds_iconv to handle unconvertible sequences reasonably.  I think I'll punt -- use tds_client_message() for now, even though it's patently seriously broken for ct-lib clients -- in order to move forward with UTF-8 clients (and servers).  I hope Bill Thompson decides to unify our client-side message system as part of his ct-lib renewal.  However, it's not clear he'll have that particular itch.  

I hope that answers your question. :-)

The information contained in this transmission may contain privileged and confidential information and is intended only for the use of the person(s) named above. If you are not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender immediately by reply e-mail and destroy all copies of the original message. Please note that we do not accept account orders and/or instructions by e-mail, and therefore will not be responsible for carrying out such orders and/or instructions.

More information about the FreeTDS mailing list