[freetds] UTF-8 update

James K. Lowden jklowden at schemamania.org
Tue Apr 8 03:41:17 EDT 2003

On Mon, 7 Apr 2003 18:05:04 -0400 , "Lowden, James K"
<LowdenJK at bernstein.com> wrote:
> I'm adding tds_get_char_data(), which takes a const TDSICONVINFO*,
> allowing conversion to be more intelligent (including per-column).  If
> the pointer is NULL, no conversion is done.  

Just committed to CVS a version that runs sp_who under tsql.  I think
we're now about back to where we were before I started mucking with
things.  I missed the snapshot cutoff, though, so if you're looking to try
out the new version, you'll have to wait until it comes around again on
the guitar.  

We are *not* ready for UTF-8.  There are still some shortcuts in the code
that have to be, um, lengthened.  

I intend two more changes in the near future:

1.  Change tds_iconv to mimic iconv(3).  All tds_iconv really does is
handle unconvertible characters until it exhausts its input.  The current
version is needlessly hard to work with.  And, yeah, that's my fault.  

2.  Add a TDSICONVINFO* to every column.  If it can use the "global"
to_client descriptor, fine, point to it.  If not, allocate a new one for
the conversion.  On deallocation, close the descriptor and free the
stucture if it doesn't match the global one.  (Perhaps better would be
linked list of TDSICONVINFO* for every conversion we run across?  Most
databases won't spawn new conversion pairs very often, and will tend to
need the same conversions in query after query.)

As far as I can tell, we really don't need to keep track of char/byte
ratios and don't need to provide them to tds_iconv.  The real question is
always: What descriptor do I hand to iconv?  That's all that really

Once those changes are made, we can hunt down the shortcuts and begin
paving the way for UTF-8 clients and servers (and UCS-2 clients). 

Discussion welcome.  That's why I post these rambling messages, to sort
out the brilliance from the lunacy.  It's not always as obvious as it
might seem, to know which is which.  


More information about the FreeTDS mailing list