FreeTDS Development Group

["Marco Brinkmann" <brinkmann AT meacon.de>]

Hi everyone,

we have the following system environment: RedHat 7.1, PHP 4.3.3,
MSSQL2000 and FreeTDS 0.62.1. TDS-Version is set to 8.0 and Client
charset is UTF-8 (tested also version 7.0).

Situation is as follows: We have a table with NTEXT and NVARCHAR
columns. Data is Japanese and German. If we try to read 10 double-bytes,
the conversion process is ok and the read data in the TDS-protocols
looks also quite good, but we will receive only 10 UTF-8 bytes in PHP.

Our UTF-8 stream is always truncated, no matter what we do. And it is
truncated to the original size of the Unicode field.

TDS-protocol looks like (with my unexperienced knowledge about TDS):

2004-01-23 17:13:34.610924 in dbsqlok()
Received header @ 2004-01-23 17:13:34.613831
0000  04 01 01 97 00 38 01 00                            |.....8..|


Received packet @ 2004-01-23 17:13:34.613904
0000  e3 19 00 01 08 66 00 64  00 69 00 64 00 32 00 30   |.....f.d
.i.d.2.0|
0010  00 30 00 33 00 03 6f 00  76 00 33 00 ab 64 00 45   |.0.3..o.
v.3..d.E|
0020  16 00 00 01 00 27 00 43  00 68 00 61 00 6e 00 67   |.....'.C
.h.a.n.g|
0030  00 65 00 64 00 20 00 64  00 61 00 74 00 61 00 62   |.e.d. .d
.a.t.a.b|
0040  00 61 00 73 00 65 00 20  00 63 00 6f 00 6e 00 74   |.a.s.e.
.c.o.n.t|
0050  00 65 00 78 00 74 00 20  00 74 00 6f 00 20 00 27   |.e.x.t.  .t.o.
.'|
0060  00 66 00 64 00 69 00 64  00 32 00 30 00 30 00 33   |.f.d.i.d
.2.0.0.3|
0070  00 27 00 2e 00 05 00 00  00 00 00 00 00 00 00 00   |.'......
........|
0080  00 01 00 e3 08 00 07 05  09 04 d0 00 34 00 fd 01   |........
....4...|
0090  00 e2 00 00 00 00 00 81  04 00 00 00 08 00 e7 40   |........
.......@|
00a0  00 09 04 d0 00 34 0c 66  00 64 00 69 00 64 00 5f   |.....4.f
.d.i.d._|
00b0  00 63 00 6f 00 6e 00 74  00 65 00 78 00 74 00 00   |.c.o.n.t
.e.x.t..|
00c0  00 08 00 e7 00 01 09 04  d0 00 34 07 66 00 64 00   |........
..4.f.d.|
00d0  69 00 64 00 5f 00 69 00  64 00 00 00 08 00 e7 06   |i.d._.i.
d.......|
00e0  00 09 04 d0 00 34 09 66  00 64 00 69 00 64 00 5f   |.....4.f
.d.i.d._|
00f0  00 6c 00 61 00 6e 00 67  00 00 00 08 00 63 00 fc   |.l.a.n.g
.....c..|
0100  00 00 09 04 d0 00 34 08  00 66 00 64 00 69 00 64   |......4.
.f.d.i.d|
0110  00 5f 00 67 00 75 00 69  00 09 66 00 64 00 69 00   |._.g.u.i
..f.d.i.|
0120  64 00 5f 00 74 00 65 00  78 00 74 00 d1 0c 00 63   |d._.t.e.
x.t....c|
0130  00 6f 00 6d 00 6d 00 6f  00 6e 00 12 00 6f 00 6e   |.o.m.m.o
.n...o.n|
0140  00 72 00 65 00 71 00 75  00 65 00 73 00 74 00 04   |.r.e.q.u
.e.s.t..|
0150  00 6a 00 70 00 10 fc ff  95 00 00 00 00 00 70 00   |.j.p....
......p.|
0160  00 00 01 00 30 00 64 75  6d 6d 79 54 53 00 14 00   |....0.du
mmyTS...|
0170  00 00 4a 30 4f 55 44 30  08 54 8f 30 5b 30 0b 4e   |..J0OUD0
.T.0[0.N|
0180  55 30 44 30 02 30 fd 10  00 c1 00 01 00 00 00      |U0D0.0..
.......|

> The data seems to be correct. TDS gets 20 Bytes for the SELECT which
is
> absolutely correct (there are 10 japanese signs in the database for
that
> request). At least if iconv has not run at this point.

2004-01-23 17:13:34.614360 dbsqlok() marker is e3
2004-01-23 17:13:34.614383 dbsqlok() found throwaway token
2004-01-23 17:13:34.614406 inside tds_process_default_tokens() marker is
e3(ENVCHANGE)
2004-01-23 17:13:34.614451 dbsqlok() marker is ab
2004-01-23 17:13:34.614474 dbsqlok() found throwaway token
2004-01-23 17:13:34.614497 inside tds_process_default_tokens() marker is
ab(INFO)
2004-01-23 17:13:34.614538 dbsqlok() marker is e3
2004-01-23 17:13:34.614561 dbsqlok() found throwaway token
2004-01-23 17:13:34.614583 inside tds_process_default_tokens() marker is
e3(ENVCHANGE)
2004-01-23 17:13:34.614608 dbsqlok() marker is fd
2004-01-23 17:13:34.614630 dbsqlok() found end token
2004-01-23 17:13:34.614652 inside tds_process_end() more_results = 1,
was_cancelled = 0
2004-01-23 17:13:34.614679 dbsqlok() end status was success
2004-01-23 17:13:34.614702 inside dbresults()
2004-01-23 17:13:34.614724 inside dbresults_r()
2004-01-23 17:13:34.614746 processing result tokens.  marker is
81(TDS7_RESULT)
2004-01-23 17:13:34.614790 tds7_get_data_info:1083:
        type = 39 (varchar)
        column_varint_size = 2
        colname = fdid_context
        colnamelen = 12
2004-01-23 17:13:34.614825 tds7_get_data_info:1083:
        type = 39 (varchar)
        column_varint_size = 2
        colname = fdid_id
        colnamelen = 7
2004-01-23 17:13:34.614859 tds7_get_data_info:1083:
        type = 39 (varchar)
        column_varint_size = 2
        colname = fdid_lang
        colnamelen = 9
2004-01-23 17:13:34.614892 tds7_get_data_info:1083:
        type = 35 (text)
        column_varint_size = 4
        colname = fdid_text
        colnamelen = 9
2004-01-23 17:13:34.614926 inside dbresults_r() result_type = 4049
retcode = 1
2004-01-23 17:13:34.614952 processing result tokens.  marker is  d1(ROW)
2004-01-23 17:13:34.614976 inside dbresults_r() result_type = 4040
retcode = 1
2004-01-23 17:13:34.615002 leaving dbresults() returning 1
2004-01-23 17:13:34.615028 inside dbnextrow()
2004-01-23 17:13:34.615051 processing row tokens.  marker is  d1(ROW)
2004-01-23 17:13:34.615075 processing row.  column is 0 varint size = 2
2004-01-23 17:13:34.615135 processing row.  column size is 12
2004-01-23 17:13:34.615160 clearing column 0 NULL bit
2004-01-23 17:13:34.615185 processing row.  column is 1 varint size = 2
2004-01-23 17:13:34.615210 processing row.  column size is 18
2004-01-23 17:13:34.615233 clearing column 1 NULL bit
2004-01-23 17:13:34.615257 processing row.  column is 2 varint size = 2
2004-01-23 17:13:34.615283 processing row.  column size is 4
2004-01-23 17:13:34.615305 clearing column 2 NULL bit
2004-01-23 17:13:34.615329 processing row.  column is 3 varint size = 4
2004-01-23 17:13:34.615355 processing row.  column size is 20
2004-01-23 17:13:34.615378 clearing column 3 NULL bit

> Also, the calculated length is correct.

2004-01-23 17:13:34.615411 leaving dbnextrow() returning -1
2004-01-23 17:13:34.615454 dbdatlen() type = 39
2004-01-23 17:13:34.615478 leaving dbdatlen() returning 6
2004-01-23 17:13:34.615501 dbdatlen() type = 39
2004-01-23 17:13:34.615523 leaving dbdatlen() returning 6
2004-01-23 17:13:34.615547 dbdatlen() type = 39
2004-01-23 17:13:34.615568 leaving dbdatlen() returning 9
2004-01-23 17:13:34.615591 dbdatlen() type = 39
2004-01-23 17:13:34.615613 leaving dbdatlen() returning 9
2004-01-23 17:13:34.615636 dbdatlen() type = 39
2004-01-23 17:13:34.615658 leaving dbdatlen() returning 2
2004-01-23 17:13:34.615681 dbdatlen() type = 39
2004-01-23 17:13:34.615702 leaving dbdatlen() returning 2
2004-01-23 17:13:34.615725 dbdatlen() type = 35
2004-01-23 17:13:34.615747 leaving dbdatlen() returning 10
2004-01-23 17:13:34.615770 dbdatlen() type = 35
2004-01-23 17:13:34.615791 leaving dbdatlen() returning 10
2004-01-23 17:13:34.615818 inside dbnextrow()
2004-01-23 17:13:34.615840 processing row tokens.  marker is  fd(DONE)
2004-01-23 17:13:34.615864 inside tds_process_end() more_results = 0,
was_cancelled = 0
2004-01-23 17:13:34.615891 leaving dbnextrow() returning -2

> What quite strange is, that freetds returns 10 bytes for that request,
but
> it should return 20. That's at least what I understand!

The received binary data in PHP is "e3818ae5958fe3818400". These are 3
japanese signs, which are totally correct according to the database
content (and printed out on the web browser correctly, too). But the 7
other signs are not transferred.

If you have any clues or hints where I could have a look (or is it a
bug?), let me know...

Thanks for your support,

Marco Brinkmann