FreeTDS Development Group

["James K. Lowden" <jklowden AT schemamania.org>]

On Sun, 25 Jan 2004, "Marco Brinkmann" <brinkmann AT meacon.de> wrote:
> 
> > Hmm.  When you set up your RH8 box, did you get a newer version of
> > PHP, too?  
> > 
> The system configuration is still the same. It is PHP 4.3.3 and FreeTDS 
> 0.62.1. We just made a new installation of Red Hat with version 8.0. 
> freetds.conf and php.ini were transferred from the old system. The only
> real difference is that in RedHat 7.0 the locale settings were "en_US"
> and in RedHat 8.0 it is "de_DE.UTF-8". But if freetds 0.62.1 really
> takes the client charset in the conf it shouldn't matter. 

I don't see why the locale would matter, you're right.  For now, it's just
a data point.  

> > I'm a bit disadvantaged, because I don't speak Japanese.  
> > 
> Me, too! What I did is just compare those hieroglyphs ;) I created a
> very small script that just owns a textarea field (I filled it up with
> data from our japanese database). After the post of the UTF-8 data, it
> writes this content to the database. And then I compared those two
> strings...

Clever.  I'm not going to install Japanese fonts for now, but I might
employ your strategy later.  

> > When you configured FreeTDS, did it detect and use your system's
> > libiconv? You can check in config.log.  
> > 
> I am not very familiar with compiler statements under Linux (a collegue
> did the compiling), but as far as I can see everything seems to work
> (checking for iconv... result: yes)

Then you're using RH's included GNU iconv.  That's good.  (Your log would
also indicate if you're using our built-in iconv replacement.  It doesn't,
so you're not.)

> I cannot use tsql because the shell does not accept the Japanese 
> signs (they are converted to question marks).

Ah.  I see we need a better test bed.  Maybe you can use redirection to
get around that, e.g.:
        tsql -S server -U user -P pass  < query.utf8

> I also ran the SQL profiler and it shows no insertation code.
> SQL:BatchStarting and SQL:BatchCompleted ran an empty string.

Right.  Confirmed by the log, here:

> > 19:20:23.342561 tds_put_string converting 128 bytes of "USE xxxx_xxx;
> > INSERT INTO test (test) VALUES (N'ãLF-X1ãã¯ã12.1Våã®æ
> > æã¢ãã¿ã¼ã¨æä½é¨ï¼ãã¼ã¹ã¹>ãã¼ã·ã§ã³ï¼ã§æ§æãããåæ¹ã®éä¿¡ã"ã¯ã5GHz帯ã
> > ã¤ã¤ãã¹è¦æ ¼ã®ç¡ç·æè¡ï¼IEEE802.11aï¼ã¨ã2.4GHz帯ã®ã¯ã¤ã¤ãã¹è¦æ ¼ã®ç¡ç
> > æè¡ï¼IEEE802.11b /  ^M')"
> > 19:20:23.342580 tds_put_string wrote 0 bytes
> > Sending packet @ 19:20:23.342597
> > 0000 01 01 00 08 00 00 01 00-                        |........|

That's bad, and I think see the problem.  tds_put_string() processes its
input in chunks, and uses a temporary buffer of 256 bytes for its output. 
It evaluates the input and output character sets, and arranges to convert
no more input at a time than can (conservatively) fit in its output
buffer.  In this case, the conversion is UTF8->UCS2, a maximum possible
expansion ratio of 1:2 (because UTF-8 characters in the ASCII range are 1
byte of input and 2 bytes of output).  tds_put_string() thus processes
your input 128 bytes at a time.  

Now, your UTF-8 input is a mixture of ASCII and Japanese characters; the
characters vary in width from 1 to 3 bytes.  When tds_put_string() grabs
128 bytes, it's very possible that it will grab a partial character;
input[127] may not be the last (or only) byte of a character. 
tds_put_string() can't avoid that possibility without interpreting the
input stream; it relies on tds_iconv() instead.  

(You can see that inserting ASCII characters into the input buffer would
sometimes cause the 128th byte to be the last of a complete character
sequence, neatly avoiding this partial-character situation.)

If tds_iconv() receives a partial character at the end of its input, it
emits an error message to that effect, unless the message is suppressed. 
However, the message IS suppressed, because we know there's more to come,
and we know we sliced the input buffer at an arbitrary (not character)
boundary.  Unfortunately, when tds_iconv() returns -1 (indicating some
kind of error), tds_put_string() quits instead of handling the situation
intelligently.  :-((

Thanks for finding our bug.  Please try the attached patch.  If it works,
I'll apply it to the 0.62 branch.  At the very least, it will make your
log bigger.  

N.B.:  src/tds/unittests/utf8_1 does not exercise this loop effectively. 
We need a longer Japanese input string, and larger buffers.  

Regards, 

--jkl

Attachment: write.diff
Description: Binary data