FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Varley, David (CBorn at Alcoa) 
> [mailto:David.Varley AT alcoa.com.au]
> Sent: May 13, 2003 9:09 PM
> 
> as you now claim some stability I thought I'd give it another try! 

Hi David, 

Thanks for the report.  I've never seen it expressed this way, but istm the
development cycle has two phases: getting it to work correctly when
everything's OK, and getting it to cope successfully when they're not.  I
hope we're in Phase II.  ;-)

> As it
> stands, the package builds without problems, but bombs out on 
> the first
> test, with freetds.log containing:
> 
> Starting log file for FreeTDS 0.62.dev.
>       on 2003-05-14 08:28:29 with debug level 99.
> iconv to convert client-side data to the "ISO-8859-1" character set
> 08:28:29.631897 tds_iconv_info_init: cannot convert "CP819"->"UCS-2LE"
> 08:28:29.631897 Connecting to 192.168.221.5 port 1433, TDS 7.0.
> 08:28:29.632873 tds_put_string converting 6 bytes of "myhost"
> 08:28:29.632873 tds_put_string converting 6 bytes of "myhost"
> [...]
> 
> which continues until I get a "device full" message.
> 
> This can be traced to tds_put_string() in tds/write.c which 
> loops forever
> waiting for tds_iconv() to decrement 'len' below zero, which 
> never happens.
> Now while the problem is doubtless in iconv due to different 
> character set
> naming on Tru64, having the code loop forever because 
> tds_iconv() doesn't convert anything is not good.

Right.  :-( It's important on any call to tds_iconv() to check if no
conversions were made.  Passing it the same inputs over and over is not a
recipe for progress.  That I know how to fix.  

As things stand, we have effectively no support for non-GNU iconv.  However
we decide to address the issue technically, this much is certain:  We need a
list of charset names for any iconv we're going to support, mapped to our
canonical name (or at least a GNU name).  If you'll provide that list, I'll
work it into the system.  

The basic problem is, you see, we need to know how many bytes to alot for
each character.  We maintain a lookup table, in src/tds/character_sets.h,
with this information.  Its key is a string, our canonical name for the
character set.  That's why we need to map proprietary names to  our
canonical ones.  

The fact that there's no way to obtain this information from the iconv
system leads me to think that other applications just use a worst-case, 4
byte/char, assumption.  A database library doesn't have that luxury.  We'd
be allocating 32 KB for every 8000 byte varchar, of which half would
typically be wasted.  

> 08:28:29.631897 tds_iconv_info_init: cannot convert "CP819"->"UCS-2LE"

CP819 is in our table, and UCS-2LE has always been the name we passed to
iconv_open, going back at least to the 0.61 release.  My guess is that Tru64
iconv doesn't recognize "UCS-2LE".  Can you confirm?  

In the 0.61 release, if iconv failed to open, we reverted to our "strip the
high byte" algorithm.  That would be very complex to support in the new
framework, and not much use besides.  If we're going to support Tru64, we
might as well figure out how to engage its iconv.  

Regards, David, and thanks for your continued support.   

--jkl


-----------------------------------------
The information contained in this transmission may contain privileged and 
confidential information and is intended only for the use of the person(s) 
named above. If you are not the intended recipient, or an employee or agent 
responsible for delivering this message to the intended recipient, any 
review, dissemination, distribution or duplication of this communication is 
strictly prohibited. If you are not the intended recipient, please contact 
the sender immediately by reply e-mail and destroy all copies of the original 
message. Please note that we do not accept account orders and/or instructions 
by e-mail, and therefore will not be responsible for carrying out such orders 
and/or instructions.