[freetds] tds_get_char_data and iconv

Lowden, James K LowdenJK at bernstein.com
Fri May 2 11:55:59 EDT 2003

> From: ZIGLIO Frediano [mailto:Frediano.Ziglio at vodafoneomnitel.it]
> Sent: May 2, 2003 4:56 AM
> tds_get_char_data need some work. The caller know the 
> "wire_size", so there's no reason to compute (there is also 
> no good way to compute it...). tds_iconv call lead to buffer 
> overflow (destination size is never decremented...). If we 
> pass curcol there is no need to pass dest. IMHO a good prototype is:
>   int tds_get_char_data(TDSSOCKET * tds, const TDSCOLINFO * 
> curcol, int wire_size)
> (if we want to discard data just call tds_get_n with NULL 
> destination...)

Thanks.  Good suggestions.  It's taken me some time to get clear in my head
how I want these things to work.  

I still mean to change tds_iconv() to look like iconv(3).  I think its only
value now will be to emit an error message when the conversion fails, and to
copy input->output if the conversion descriptor is invalid.  

> So there are no solution for all platform. IMHO configure 
> should check all standard names using iconv command (not 
> iconv function so it should work on cross compiler too).. 

I don't see the need.  configure should test for locale_charset().  That
tells us if we're using GNU iconv().  [Actually, it's a proxy: a machine
could have the GNU iconv.h but link the native iconv library.  I'd say
that's a local configuration problem.]  If we have locale_charset(), we use
it to get the iconv client charset name, else we rely on nl_langinfo(3).  We
will probably encounter some broken OS somewhere, where that doesn't work.
We'll cross that bridge when we come to it.  I consider client-side charset
names solved, in theory.  

Server side is a dual challenge and not solved.
tds_canonical_charset_name() will currently map Sybase names and GNU aliases
to the GNU iconv names.  It can be extended (by adding to
alternative_character_sets.h) to include more aliases, but it will need
heavy editing to support non-GNU iconv names.  That is, the canonical names
are currently GNU's; we have no mapping of Sybase names to those used by the
iconv that comes with, say, Solaris.  

To support non-GNU iconv, someone is going to have to pony up.  Someone's
going to have to volunteer the OS<->server name map.  I'll do GNU's names,
and I'll provide the framework for other names, but I'm not going to do the
legwork for nonfree systems.  

There is an avenue open to the ambitious person who'd like to do this: the
underlying data can be found in the data files that support
locale_charset().  The information's all there; all that's needed is to
expose the function that maps the OS's name to GNU's (and perhaps

If presented with a useful map for another OS, we should consider moving the
name map out of a header file, and into a dbm database of some kind.  I
don't know the best standard to use, but the list of names is quite long,
and it seems wasteful to me to retain them in statically allocated memory
when they're only needed occasionally (mostly at initialization).  Another
alternative would be to put the names in a table on the *server* (!) but
we'd have to be careful of a chicken-and-egg problem.  

> Should we define some required charset ??

Not unless we have no alternative.  The base assumption is that the client
and server share an encoding in common, iconv not required.  

> Another problem is conversions. GNU iconv accept conversion 
> from any charset to any charset. Many implementation accept a 
> limited conversion (for example in HP/UX I can convert 
> iso88591 -> ucs2 and ucs2 -> iso88592 but it can't convert 
> iso88591 -> iso88592). Samba solution is: if a direct 
> conversion is not present convert to ucs2 first (GNU iconv 
> internally convert chars to ucs4 and than ucs4 to destination 
> charset)...

Everyone is free to use GNU iconv.  If the OS's iconv() doesn't pass muster,
GNU's will.  I'm willing to say that FreeTDS's conversion capacity is a
direct function of the underlying iconv's capacity.  Why should we jump
through any hoops?  



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.

More information about the FreeTDS mailing list