[freetds] Running FreeTDS under VMS

Lowden, James K LowdenJK at bernstein.com
Thu May 1 13:46:32 EDT 2003

> From: Craig A. Berry [mailto:craigberry at mac.com]
> Sent: May 1, 2003 11:28 AM
> One workaround that got me over one obstacle was adding the following
> line to the global section of my freetds.conf file:
>    client charset = ISO8859-1
> However, the GNUish "ISO-8859-1" variant is hard-coded various places
> in the sources.  

Craig, thanks for trying this out.  I know about the hard-coding, but could
you elaborate on why "ISO8859-1" helps you?  I think I know.  

To convert character sets, we use iconv.  To tell iconv what we'd like it to
convert from/to, we pass charset names.  These names are not standardized
across iconv implementations (or operating systems).  

There are two kinds of charset names.  Well, three, sort of:

1.  Server names, from master..syscharsets.  
2.  OS names, from nl_langinfo(CODESET).
3.  iconv names.  

I suppose it's obvious that server charset names need to be mapped to iconv
names.  That mapping is provided by src/tds/sybase_character_sets.h.  

Many OS's provide both nl_langinfo() and iconv().  In that case, there's no
mapping of OS name to iconv name; that should be 1:1 unless the vendor is
internally inconsistent.  GNU/Linux of course uses GNU iconv; no mapping
issue there, either.  

If the OS provides nl_langinfo() but not iconv(), or if FreeTDS was
configured to use GNU's iconv instead, the need arises to map the OS's
charset name to GNU's name.  Lucky for us, that map exists already, as part
of the GNU iconv distribution, in locale_charset().  That's handy for folks
like me whose OS (NetBSD) lacks iconv(3).  

On the server side, sybase_character_sets.h maps Sybase's names to GNU
names.  (GNU didn't choose silly names; they relied on well-established
standards.  There are competing and sometimes conflicting standards.)
Anyone who wants to use a non-GNU iconv will need to patch that file (or its
Perl source), using #if to insinuate the overrides.  We're accepting such
patches now; I don't see that mechanism changing.  And, even if the
mechanism does change, we'll still need the names, regardless.  

On the client side, returning to your observation, we have to allow for both
the need and the non-need of locale_charset().  My plan was: 

	/* query OS, get GNU name */
	name = locale_charset();

but that works only for GNU iconv.  Bummer.  Not all systems will have
locale_charset(), of course, notably those not running GNU iconv.  The logic

		/* query OS, get GNU charset name */
		name = locale_charset();
		/* query OS, get OS charset name */
		name = nl_langinfo(CODESET);

Which I think will Just Work on any rightly configured system.  For
misconfigured systems, we still have "client charset" in freetds.conf, which
we'll accept verbatim.  

Status: we're between "knowing the answer" and "doing the work".
Initialization is also shaky.  

Status: we're shaking the tree to get one thing right: UTF-8 clients with
any server.  We're starting with GNU iconv because that's what I've got.
Generalization isn't my focus right now, but patches are welcome.  If you
see an error or an opportunity, carpe diem et codem transit.  I'll apply the
patch before proceeding if it's any use.  :-)

As to the internal hard-coding of "ISO-8859-1": it's the same problem in new
garb.  Occasionally, we need to convert static, compiled-in strings to the
server's (and one day the client's) character set.  Those strings are ASCII.
We can't rely on the client's encoding; the client might for example be
UCS-2 (UTF-8 and IS0-8859-x would be OK, though).  We need to tell iconv,
"Hey baby, like ASCII, you know?"  But, you may have heard the rumor that
iconv names are not standardized across iconv implementations.  It's true.  

Perhaps for ASCII we can find a single string that all iconv implementations
recognize.  GNU's recognizes these:

$ iconv -l |grep -i ascii |sed 's/ /\
        /g' |sort

Is "ASCII" perchance on your list, too?  I like the idea of calling ASCII,

I know, "ASCII silly question, get a silly ANSI".  

For more background, see my similarly long missives on 15 April 2003.
There's an explanation there of how we came to know locale_charset() and why
we're abandoning src/tds/alternative_character_sets.h.  



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.

More information about the FreeTDS mailing list