FreeTDS Development Group

["Marco Brinkmann" <brinkmann AT meacon.de>]

Hi James,

Hmm.  When you set up your RH8 box, did you get a newer version of PHP,
too?  

> The system configuration is still the same. It is PHP 4.3.3 and FreeTDS 
> 0.62.1. We just made a new installation of Red Hat with version 8.0. 
> freetds.conf and php.ini were transferred from the old system. The only
> real difference is that in RedHat 7.0 the locale settings were "en_US" and
> in RedHat 8.0 it is "de_DE.UTF-8". But if freetds 0.62.1 really takes the
> client charset in the conf it shouldn't matter. 

That's not good.  :-(  I'm happy to fix it if I can; just please
understand that I'm a bit disadvantaged, because I don't speak Japanese.  

> Me, too! What I did is just compare those hieroglyphs ;) I created a very
> small script that just owns a textarea field (I filled it up with data
> from our japanese database). After the post of the UTF-8 data, it writes 
> this content to the database. And then I compared those two strings...

There shouldn't be any limit, but there may be some confusion regarding
character count and buffer size.  Perhaps we've just been lucky so far. 
:-/  Or maybe there's an implementation problem we haven't been aware of. 

> What I figured is as follows: The error only happens when there are more 
> then 128 Bytes (UTF-8 code) are attached to each other. If I BREAK UP this 
> > UTF-8 stream with ONE single ascii sign, the insert works!

When you configured FreeTDS, did it detect and use your system's libiconv?
You can check in config.log.  

> I am not very familiar with compiler statements under Linux (a collegue
> did the compiling), but as far as I can see everything seems to work
> (checking for iconv... result: yes)

Can you force the failure using tsql, and can you post a TDSDUMP log of
the session?  That will give us something to work with.  

> I attached the freetds.log for my php script (does just one insert with 
> 267 bytes (UTF-8) for round about 140 japanese signs mixed with normal
> ascii). I  cannot use tsql because the shell does not accept the Japanese 
> signs (they are converted to question marks).

> I also ran the SQL profiler and it shows no insertation code.
> SQL:BatchStarting and SQL:BatchCompleted ran an empty string.

Thanks and best regards,

Marco Brinkmann





Freetds.log
--------------------------------------

Starting log file for FreeTDS 0.62.1
        on 2004-01-25 19:20:23 with debug level 99.
iconv to convert client-side data to the "UTF-8" character set
19:20:23.338634 tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
19:20:23.338664 tds_iconv_info_init: converting "ISO-8859-1"->"UCS-2LE"
19:20:23.338685 Connecting to xxx.xxx.xxx.xx port xxxx, TDS 8.0.
19:20:23.339121 tds_put_string converting 17 bytes of "xxx"
19:20:23.339145 tds_put_string wrote 34 bytes
19:20:23.339161 tds_put_string converting 2 bytes of "xxx"
19:20:23.339178 tds_put_string wrote 4 bytes
19:20:23.339194 tds_put_string converting 7 bytes of "PHP 4.0"
19:20:23.339213 tds_put_string wrote 14 bytes
19:20:23.339229 tds_put_string converting 5 bytes of "xxx"
19:20:23.339245 tds_put_string wrote 10 bytes
19:20:23.339261 tds_put_string converting 10 bytes of "DB-Library"
19:20:23.339278 tds_put_string wrote 20 bytes
19:20:23.339293 tds_put_string converting 10 bytes of "us_english"
19:20:23.339310 tds_put_string wrote 20 bytes
19:20:23.339325 tds_put_string wrote 0 bytes
19:20:23.339355 tds_process_login_tokens()
Received header @ 19:20:23.340102
0000 04 01 01 85 00 37 01 00-                        |.....7..|


Received packet @ 19:20:23.340129
0000 e3 1b 00 01 06 6d 00 61-00 73 00 74 00 65 00 72 |.....m.a .s.t.e.r|
0010 00 06 6d 00 61 00 73 00-74 00 65 00 72 00 ab 78 |..m.a.s. t.e.r..x|
0020 00 45 16 00 00 02 00 31-00 44 00 65 00 72 00 20 |.E.....1 .D.e.r. |
0030 00 44 00 61 00 74 00 65-00 6e 00 62 00 61 00 6e |.D.a.t.e .n.b.a.n|
0040 00 6b 00 6b 00 6f 00 6e-00 74 00 65 00 78 00 74 |.k.k.o.n .t.e.x.t|
0050 00 20 00 77 00 75 00 72-00 64 00 65 00 20 00 61 |. .w.u.r .d.e. .a|
0060 00 75 00 66 00 20 00 27-00 6d 00 61 00 73 00 74 |.u.f. .' .m.a.s.t|
0070 00 65 00 72 00 27 00 20-00 67 00 65 00 e4 00 6e |.e.r.'.  .g.e...n|
0080 00 64 00 65 00 72 00 74-00 2e 00 05 4d 00 00 00 |.d.e.r.t ........|
0090 00 00 00 00 00 00 00 00-00 e3 08 00 07 05 09 04 |........ ........|
00a0 50 00 00 00 e3 17 00 02-0a 75 00 73 00 5f 00 65 |P....... .u.s._.e|
00b0 00 6e 00 67 00 6c 00 69-00 73 00 68 00 00 ab 64 |.n.g.l.i .s.h...d|
00c0 00 47 16 00 00 01 00 27-00 43 00 68 00 61 00 6e |.G.....' .C.h.a.n|
00d0 00 67 00 65 00 64 00 20-00 6c 00 61 00 6e 00 67 |.g.e.d.  .l.a.n.g|
00e0 00 75 00 61 00 67 00 65-00 20 00 73 00 65 00 74 |.u.a.g.e . .s.e.t|
00f0 00 74 00 69 00 6e 00 67-00 20 00 74 00 6f 00 20 |.t.i.n.g . .t.o. |
0100 00 75 00 73 00 5f 00 65-00 6e 00 67 00 6c 00 69 |.u.s._.e .n.g.l.i|
0110 00 73 00 68 00 2e 00 05-4d 00 00 00 00 00 00 00 |.s.h.... ........|
0120 00 00 00 00 00 ad 36 00-01 71 00 00 01 16 4d 00 |......6. .q....M.|
0130 69 00 63 00 72 00 6f 00-73 00 6f 00 66 00 74 00 |i.c.r.o. s.o.f.t.|
0140 20 00 53 00 51 00 4c 00-20 00 53 00 65 00 72 00 | .S.Q.L.  .S.e.r.|
0150 76 00 65 00 72 00 00 00-00 00 08 00 02 f8 e3 13 |v.e.r... ........|
0160 00 04 04 34 00 30 00 39-00 36 00 04 34 00 30 00 |...4.0.9 .6..4.0.|
0170 39 00 36 00 fd 00 00 00-00 00 00 00 00          |9.6..... .....|


19:20:23.340250 looking for login token, got  e3(ENVCHANGE)
19:20:23.340266 tds_process_default_tokens() marker is e3(ENVCHANGE)
tds_get_string: reading 12 from wire to give 6 to client.
tds_get_string: reading 12 from wire to give 6 to client.
19:20:23.340313 looking for login token, got  ab(INFO)
19:20:23.340329 tds_process_default_tokens() marker is ab(INFO)
tds_get_string: reading 98 from wire to give 49 to client.
tds_get_string: reading 10 from wire to give 5 to client.
19:20:23.340379 looking for login token, got  e3(ENVCHANGE)
19:20:23.340396 tds_process_default_tokens() marker is e3(ENVCHANGE)
19:20:23.340428 tds_iconv_info_init: converting "UTF-8"->"CP1252"
19:20:23.340445 looking for login token, got  e3(ENVCHANGE)
19:20:23.340461 tds_process_default_tokens() marker is e3(ENVCHANGE)
tds_get_string: reading 20 from wire to give 10 to client.
19:20:23.340492 looking for login token, got  ab(INFO)
19:20:23.340508 tds_process_default_tokens() marker is ab(INFO)
tds_get_string: reading 78 from wire to give 39 to client.
tds_get_string: reading 10 from wire to give 5 to client.
19:20:23.340554 looking for login token, got  ad(LOGINACK)
tds_get_string: reading 44 from wire to give 22 to client.
19:20:23.340607 looking for login token, got  e3(ENVCHANGE)
19:20:23.340623 tds_process_default_tokens() marker is e3(ENVCHANGE)

tds_get_string: reading 8 from wire to give 4 to client.
tds_get_string: reading 8 from wire to give 4 to client.
19:20:23.340667 increasing block size from 4096 to 4096
19:20:23.340685 looking for login token, got  fd(DONE)
19:20:23.340701 tds_process_default_tokens() marker is fd(DONE)
19:20:23.340717 tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 0
19:20:23.340735 tds_process_end() state set to TDS_IDLE
19:20:23.340750 leaving tds_process_login_tokens() returning 1
19:20:23.340770 tds_put_string converting 19 bytes of "set textsize 64512 "
19:20:23.340787 tds_put_string wrote 38 bytes
Sending packet @ 19:20:23.340803
0000 01 01 00 2e 00 00 01 00-73 00 65 00 74 00 20 00 |........ s.e.t. .|
0010 74 00 65 00 78 00 74 00-73 00 69 00 7a 00 65 00 |t.e.x.t. s.i.z.e.|
0020 20 00 36 00 34 00 35 00-31 00 32 00 20 00       | .6.4.5. 1.2. .|


Received header @ 19:20:23.341223
0000 04 01 00 11 00 37 01 00-                        |.....7..|


Received packet @ 19:20:23.341249
0000 fd 00 00 be 00 00 00 00-00                      |........ .|


19:20:23.341268 processing result tokens.  marker is  fd(DONE)
19:20:23.341285 tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 0
19:20:23.341302 tds_process_end() state set to TDS_IDLE
19:20:23.341316 tds_process_result_tokens() state is COMPLETED
19:20:23.341363 dbuse()
19:20:23.341380 dbcmd() bufsz = 0
19:20:23.341396 in dbsqlexec()
19:20:23.341411 in dbsqlsend()
19:20:23.341427 tds_put_string converting 13 bytes of "use xxx"
19:20:23.341444 tds_put_string wrote 26 bytes
Sending packet @ 19:20:23.341460
0000 01 01 00 22 00 00 01 00-75 00 73 00 65 00 20 00 |...".... u.s.e. .|
0010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 |......_. ........|
0020 .. 00                  -                        |..|


19:20:23.341497 in dbsqlok()
Received header @ 19:20:23.341971
0000 04 01 00 a9 00 37 01 00-                        |.....7..|


Received packet @ 19:20:23.341996


0000 e3 21 00 01 09 00 00 00-00 00 00 00 00 00 00 00 |.!...... ........|
0010 00 00 00 00 00 00 00 06-6d 00 61 00 73 00 74 00 |........ m.a.s.t.|
0020 65 00 72 00 ab 66 00 45-16 00 00 01 00 28 00 43 |e.r..f.E .....(.C|
0030 00 68 00 61 00 6e 00 67-00 65 00 64 00 20 00 64 |.h.a.n.g .e.d. .d|
0040 00 61 00 74 00 61 00 62-00 61 00 73 00 65 00 20 |.a.t.a.b .a.s.e. |
0050 00 63 00 6f 00 6e 00 74-00 65 00 78 00 74 00 20 |.c.o.n.t .e.x.t. |
0060 00 74 00 6f 00 20 00 27-00 00 00 00 00 00 00 00 |.t.o. .' ........|
0070 00 00 00 00 00 00 00 00-00 00 00 27 00 2e 00 05 |........ ...'....|
0080 00 00 00 00 00 00 00 00-00 00 00 01 00 e3 08 00 |........ ........|
0090 07 05 09 04 50 00 00 00-fd 00 00 e2 00 00 00 00 |....P... ........|
00a0 00                     -                        |.|


19:20:23.342055 dbsqlok() marker is e3
19:20:23.342071 processing result tokens.  marker is  e3(ENVCHANGE)
19:20:23.342087 tds_process_default_tokens() marker is e3(ENVCHANGE)
tds_get_string: reading 18 from wire to give 9 to client.
tds_get_string: reading 12 from wire to give 6 to client.
19:20:23.342131 processing result tokens.  marker is  ab(INFO)
19:20:23.342147 tds_process_default_tokens() marker is ab(INFO)
tds_get_string: reading 80 from wire to give 40 to client.
tds_get_string: reading 10 from wire to give 5 to client.
19:20:23.342194 processing result tokens.  marker is  e3(ENVCHANGE)
19:20:23.342210 tds_process_default_tokens() marker is e3(ENVCHANGE)
19:20:23.342228 processing result tokens.  marker is  fd(DONE)
19:20:23.342244 tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 0
19:20:23.342262 tds_process_end() state set to TDS_IDLE
19:20:23.342278 dbsqlok() end status was success
19:20:23.342293 dbresults()
19:20:23.342307 dbresults_r()
19:20:23.342320 tds_process_result_tokens() state is COMPLETED
19:20:23.342356 leaving dbresults() returning 1
19:20:23.342372 tds_process_row_tokens() state is COMPLETED
19:20:23.342495 dbcmd() bufsz = 14
19:20:23.342513 in dbfreebuf()
19:20:23.342530 in dbsqlexec()
19:20:23.342545 in dbsqlsend()
19:20:23.342561 tds_put_string converting 128 bytes of "USE xxxx_xxx; INSERT 
INTO test (test) VALUES (N'ãLF-X1ãã¯ã12.1Våã®æ²æã¢ãã¿ã¼ã¨æä½é¨ï¼ãã¼ã¹ã¹>
ãã¼ã·ã§ã³ï¼ã§æ§æãããåæ¹ã®éä¿¡ã"ã¯ã5GHz帯ã¯ã¤ã¤ãã¹è¦æ 
¼ã®ç¡ç·æè¡ï¼IEEE802.11aï¼ã¨ã2.4GHz帯ã®ã¯ã¤ã¤ãã¹è¦æ ¼ã®ç¡ç·æè¡ï¼IEEE802.11b / 
 ^M
')"
19:20:23.342580 tds_put_string wrote 0 bytes
Sending packet @ 19:20:23.342597
0000 01 01 00 08 00 00 01 00-                        |........|


19:20:23.342630 in dbsqlok()
Received header @ 19:20:23.343099
0000 04 01 00 11 00 37 01 00-                        |.....7..|


Received packet @ 19:20:23.343124
0000 fd 00 00 fd 00 00 00 00-00                      |........ .|


19:20:23.343143 dbsqlok() marker is fd
19:20:23.343158 processing result tokens.  marker is  fd(DONE)
19:20:23.343174 tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 0
19:20:23.343192 tds_process_end() state set to TDS_IDLE
19:20:23.343207 dbsqlok() end status was success
19:20:23.343222 dbresults()
19:20:23.343236 dbresults_r()
19:20:23.343249 tds_process_result_tokens() state is COMPLETED
19:20:23.343265 leaving dbresults() returning 1
19:20:23.343279 dbresults()
19:20:23.343294 dbresults_r()
19:20:23.343308 tds_process_result_tokens() state is COMPLETED
19:20:23.343322 leaving dbresults() returning 2
19:20:23.343435 in dbfreebuf()

---------------------------------
End of freetds.log