Skip to Content.
Sympa Menu

freetds - [freetds] UTF-8 truncated

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "Marco Brinkmann" <brinkmann AT meacon.de>
  • To: <freetds AT lists.ibiblio.org>
  • Subject: [freetds] UTF-8 truncated
  • Date: Sat, 24 Jan 2004 10:40:48 +0100


Hi everyone,

we have the following system environment: RedHat 7.1, PHP 4.3.3,
MSSQL2000 and FreeTDS 0.62.1. TDS-Version is set to 8.0 and Client
charset is UTF-8 (tested also version 7.0).

Situation is as follows: We have a table with NTEXT and NVARCHAR
columns. Data is Japanese and German. If we try to read 10 double-bytes,
the conversion process is ok and the read data in the TDS-protocols
looks also quite good, but we will receive only 10 UTF-8 bytes in PHP.

Our UTF-8 stream is always truncated, no matter what we do. And it is
truncated to the original size of the Unicode field.

TDS-protocol looks like (with my unexperienced knowledge about TDS):

2004-01-23 17:13:34.610924 in dbsqlok()
Received header @ 2004-01-23 17:13:34.613831
0000 04 01 01 97 00 38 01 00 |.....8..|


Received packet @ 2004-01-23 17:13:34.613904
0000 e3 19 00 01 08 66 00 64 00 69 00 64 00 32 00 30 |.....f.d
.i.d.2.0|
0010 00 30 00 33 00 03 6f 00 76 00 33 00 ab 64 00 45 |.0.3..o.
v.3..d.E|
0020 16 00 00 01 00 27 00 43 00 68 00 61 00 6e 00 67 |.....'.C
.h.a.n.g|
0030 00 65 00 64 00 20 00 64 00 61 00 74 00 61 00 62 |.e.d. .d
.a.t.a.b|
0040 00 61 00 73 00 65 00 20 00 63 00 6f 00 6e 00 74 |.a.s.e.
.c.o.n.t|
0050 00 65 00 78 00 74 00 20 00 74 00 6f 00 20 00 27 |.e.x.t. .t.o.
.'|
0060 00 66 00 64 00 69 00 64 00 32 00 30 00 30 00 33 |.f.d.i.d
.2.0.0.3|
0070 00 27 00 2e 00 05 00 00 00 00 00 00 00 00 00 00 |.'......
........|
0080 00 01 00 e3 08 00 07 05 09 04 d0 00 34 00 fd 01 |........
....4...|
0090 00 e2 00 00 00 00 00 81 04 00 00 00 08 00 e7 40 |........
.......@|
00a0 00 09 04 d0 00 34 0c 66 00 64 00 69 00 64 00 5f |.....4.f
.d.i.d._|
00b0 00 63 00 6f 00 6e 00 74 00 65 00 78 00 74 00 00 |.c.o.n.t
.e.x.t..|
00c0 00 08 00 e7 00 01 09 04 d0 00 34 07 66 00 64 00 |........
..4.f.d.|
00d0 69 00 64 00 5f 00 69 00 64 00 00 00 08 00 e7 06 |i.d._.i.
d.......|
00e0 00 09 04 d0 00 34 09 66 00 64 00 69 00 64 00 5f |.....4.f
.d.i.d._|
00f0 00 6c 00 61 00 6e 00 67 00 00 00 08 00 63 00 fc |.l.a.n.g
.....c..|
0100 00 00 09 04 d0 00 34 08 00 66 00 64 00 69 00 64 |......4.
.f.d.i.d|
0110 00 5f 00 67 00 75 00 69 00 09 66 00 64 00 69 00 |._.g.u.i
..f.d.i.|
0120 64 00 5f 00 74 00 65 00 78 00 74 00 d1 0c 00 63 |d._.t.e.
x.t....c|
0130 00 6f 00 6d 00 6d 00 6f 00 6e 00 12 00 6f 00 6e |.o.m.m.o
.n...o.n|
0140 00 72 00 65 00 71 00 75 00 65 00 73 00 74 00 04 |.r.e.q.u
.e.s.t..|
0150 00 6a 00 70 00 10 fc ff 95 00 00 00 00 00 70 00 |.j.p....
......p.|
0160 00 00 01 00 30 00 64 75 6d 6d 79 54 53 00 14 00 |....0.du
mmyTS...|
0170 00 00 4a 30 4f 55 44 30 08 54 8f 30 5b 30 0b 4e |..J0OUD0
.T.0[0.N|
0180 55 30 44 30 02 30 fd 10 00 c1 00 01 00 00 00 |U0D0.0..
.......|

> The data seems to be correct. TDS gets 20 Bytes for the SELECT which
> is absolutely correct (there are 10 japanese signs in the database for

> that request). At least if iconv has not run at this point.

2004-01-23 17:13:34.614360 dbsqlok() marker is e3
2004-01-23 17:13:34.614383 dbsqlok() found throwaway token 2004-01-23
17:13:34.614406 inside tds_process_default_tokens() marker is
e3(ENVCHANGE) 2004-01-23 17:13:34.614451 dbsqlok() marker is ab
2004-01-23 17:13:34.614474 dbsqlok() found throwaway token 2004-01-23
17:13:34.614497 inside tds_process_default_tokens() marker is ab(INFO)
2004-01-23 17:13:34.614538 dbsqlok() marker is e3 2004-01-23
17:13:34.614561 dbsqlok() found throwaway token 2004-01-23
17:13:34.614583 inside tds_process_default_tokens() marker is
e3(ENVCHANGE) 2004-01-23 17:13:34.614608 dbsqlok() marker is fd
2004-01-23 17:13:34.614630 dbsqlok() found end token 2004-01-23
17:13:34.614652 inside tds_process_end() more_results = 1, was_cancelled
= 0 2004-01-23 17:13:34.614679 dbsqlok() end status was success
2004-01-23 17:13:34.614702 inside dbresults() 2004-01-23 17:13:34.614724
inside dbresults_r() 2004-01-23 17:13:34.614746 processing result
tokens. marker is 81(TDS7_RESULT) 2004-01-23 17:13:34.614790
tds7_get_data_info:1083:
type = 39 (varchar)
column_varint_size = 2
colname = fdid_context
colnamelen = 12
2004-01-23 17:13:34.614825 tds7_get_data_info:1083:
type = 39 (varchar)
column_varint_size = 2
colname = fdid_id
colnamelen = 7
2004-01-23 17:13:34.614859 tds7_get_data_info:1083:
type = 39 (varchar)
column_varint_size = 2
colname = fdid_lang
colnamelen = 9
2004-01-23 17:13:34.614892 tds7_get_data_info:1083:
type = 35 (text)
column_varint_size = 4
colname = fdid_text
colnamelen = 9
2004-01-23 17:13:34.614926 inside dbresults_r() result_type = 4049
retcode = 1 2004-01-23 17:13:34.614952 processing result tokens. marker
is d1(ROW) 2004-01-23 17:13:34.614976 inside dbresults_r() result_type
= 4040 retcode = 1 2004-01-23 17:13:34.615002 leaving dbresults()
returning 1 2004-01-23 17:13:34.615028 inside dbnextrow() 2004-01-23
17:13:34.615051 processing row tokens. marker is d1(ROW) 2004-01-23
17:13:34.615075 processing row. column is 0 varint size = 2 2004-01-23
17:13:34.615135 processing row. column size is 12 2004-01-23
17:13:34.615160 clearing column 0 NULL bit 2004-01-23 17:13:34.615185
processing row. column is 1 varint size = 2 2004-01-23 17:13:34.615210
processing row. column size is 18 2004-01-23 17:13:34.615233 clearing
column 1 NULL bit 2004-01-23 17:13:34.615257 processing row. column is
2 varint size = 2 2004-01-23 17:13:34.615283 processing row. column
size is 4 2004-01-23 17:13:34.615305 clearing column 2 NULL bit
2004-01-23 17:13:34.615329 processing row. column is 3 varint size = 4
2004-01-23 17:13:34.615355 processing row. column size is 20 2004-01-23
17:13:34.615378 clearing column 3 NULL bit

> Also, the calculated length is correct.

2004-01-23 17:13:34.615411 leaving dbnextrow() returning -1 2004-01-23
17:13:34.615454 dbdatlen() type = 39 2004-01-23 17:13:34.615478 leaving
dbdatlen() returning 6 2004-01-23 17:13:34.615501 dbdatlen() type = 39
2004-01-23 17:13:34.615523 leaving dbdatlen() returning 6 2004-01-23
17:13:34.615547 dbdatlen() type = 39 2004-01-23 17:13:34.615568 leaving
dbdatlen() returning 9 2004-01-23 17:13:34.615591 dbdatlen() type = 39
2004-01-23 17:13:34.615613 leaving dbdatlen() returning 9 2004-01-23
17:13:34.615636 dbdatlen() type = 39 2004-01-23 17:13:34.615658 leaving
dbdatlen() returning 2 2004-01-23 17:13:34.615681 dbdatlen() type = 39
2004-01-23 17:13:34.615702 leaving dbdatlen() returning 2 2004-01-23
17:13:34.615725 dbdatlen() type = 35 2004-01-23 17:13:34.615747 leaving
dbdatlen() returning 10 2004-01-23 17:13:34.615770 dbdatlen() type = 35
2004-01-23 17:13:34.615791 leaving dbdatlen() returning 10 2004-01-23
17:13:34.615818 inside dbnextrow() 2004-01-23 17:13:34.615840 processing
row tokens. marker is fd(DONE) 2004-01-23 17:13:34.615864 inside
tds_process_end() more_results = 0, was_cancelled = 0 2004-01-23
17:13:34.615891 leaving dbnextrow() returning -2

> What quite strange is, that freetds returns 10 bytes for that request,

> but it should return quite more. That's at least what I understand!

The received binary data in PHP is "e3818ae5958fe3818400". These are 3
japanese signs, which are totally correct according to the database
content (and printed out on the web browser correctly, too). But the 7
other signs are not transferred.

If you have any clues or hints where I could have a look (or is it a
bug?), let me know...

Thanks for your support,

Marco Brinkmann




  • [freetds] UTF-8 truncated, Marco Brinkmann, 01/23/2004
    • <Possible follow-up(s)>
    • [freetds] UTF-8 truncated, Marco Brinkmann, 01/24/2004

Archive powered by MHonArc 2.6.24.

Top of Page