Skip to Content.
Sympa Menu

freetds - Re: [freetds] Sybase: Charset conversion

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "James K. Lowden" <jklowden AT freetds.org>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] Sybase: Charset conversion
  • Date: Sat, 5 Dec 2009 16:53:56 -0500

Sebastian Podjasek wrote:
>
> Below are few interesting lines from TDSDUMP
>
> log.c:190:Starting log file for FreeTDS 0.82
> on 2009-11-24 17:09:44 with debug flags 0x4fff.
> iconv.c:197:names for ISO-8859-1: ISO-8859-1
> iconv.c:197:names for UTF-8: UTF-8
> iconv.c:197:names for UCS-2LE: UCS-2LE
> iconv.c:197:names for UCS-2BE: UCS-2BE
> iconv.c:363:iconv to convert client-side data to the "UTF-8" character
> set
> iconv.c:516:tds_iconv_info_init: converting "UTF-8"->"UCS-2LE"
...
> token.c:316:looking for login token, got e3(ENVCHANGE)
> token.c:108:tds_process_default_tokens() marker is e3(ENVCHANGE)
> token.c:2356:server indicated charset change to "utf8"
> iconv.c:985:setting server single-byte charset to "UTF-8"
> iconv.c:516:tds_iconv_info_init: converting "ISO-8859-1"->"UTF-8"

Line 516 looks like a problem: the server said it will send UTF-8, and
iconv.c says it will convert "ISO-8859-1" to "UTF-8". If so, FreeTDS
didn't interpret the message correctly.

When we look at the data, we find

> 0070 00 00 e1 80 02 00 00 00-d1 08 c5 9c 00 00 60 cf |........ ......`.|
> 0080 c0 00 3e 00 00 00 00 50-00 6f 01 42 01 05 00 63 |..>....P .o.B...c|
> 0090 00 7a 00 65 00 6e 00 69-00 65 00 20 00 7a 00 20 |.z.e.n.i .e. .z. |
> 00a0 00 54 00 43 00 47 00 20-00 6a 00 65 00 73 00 74 |.T.C.G. .j.e.s.t|
> 00b0 00 20 00 7a 00 61 00 6b-01 42 00 75 00 63 00 6f |. .z.a.k .B.u.c.o|
> 00c0 00 6e 00 65 fd 10 00 01-00 01 00 00 00 |.n.e.... .....|
>
> token.c:510:processing result tokens. marker is 61(ROWFMT2)
> token.c:1713:tds5_process_result
> mem.c:563:tds_free_all_results()
> token.c:1737:num_cols=2
> token.c:1840:col 0:
> token.c:1841: column_name=[dt_made]
> token.c:1847: flags=20 utype=80 type=111 varint=1
> token.c:1850: colsize=8 prec=0 scale=0
> token.c:3294:adjust_character_column_size:
> Server charset: UCS-2LE
> Server column_size: 640
> Client charset: UTF-8
> Client column_size: 1280
> token.c:1840:col 1:
> token.c:1841: column_name=[c_text]
> token.c:1847: flags=20 utype=34 type=225 varint=5
> token.c:1850: colsize=1280 prec=0 scale=0

Problem is, that's neither UTF-8 nor ISO 8859-1. I think it's utf-16be.
If so, the data would read something like:

Po??czenie z TCG jest zak?ucone

(where I have no glyphs for some Polish characters.) Does that look
right?

If so, I see two problems:

1. The server is announcing UTF-8 but the client thinks it's getting ISO
8859-1.
2. The server isn't sending UTF-8 but UTF-16. That would be because the
data are incorrectly encoded on the server. (The DBA tells the server the
encoding of the data. When bulk-loading, it's up the to the DBA to ensure
any data added use the column's encoding, because the server can't know.

HTH.

--jkl








  • Re: [freetds] Sybase: Charset conversion, James K. Lowden, 12/05/2009

Archive powered by MHonArc 2.6.24.

Top of Page