FreeTDS Development Group

[Niklas Lampén <Niklas.Lampen AT eventusmedia.fi>]

> You might want to set TDSDUMPCONFIG and TDSDUMP in your environment
> and review the resulting logs to see what client character set you're
> actually getting.  You might also try applying the patch from:
> 
> <https://lists.ibiblio.org/sympa/arc/freetds/2010q2/026008.html>
> 
> which I think should do a better job than the existing code at
> allowing freetds.conf to override the character set, though no
> guarantees it will work in your situation (however, it would be nice
> to have the confirmation if it does).

I'll do that and post you the results. Here's some more info about this issue 
without the patch. I tried to insert UTF-8 chars from shell using tsql, but 
it fails too. I do not get any errors, but result is not expected:

# tsql -S AavaDb2 -U ***** -P ***** -D *****
locale is "en_US.UTF-8"
locale charset is "UTF-8"
Default database being set to *****
1> UPDATE Mod_Texts SET ContentData='Найти' WHERE ContentId=6824
2> go
1> SELECT ContentData FROM Mod_Texts WHERE ContentId=6824
2> go
ContentData
?????
(1 row affected)

The string is inserted as question marks to database. If I set text like 'foo 
Найти', it will be 'foo ?????'.

By analyzing the log it seems that FreeTDS thinks it should send data as 
CP1252. Or is it just for single byte strings and UCS-2 is used otherwise?

Hope someone can dig something out of this.

And this is what the log says:

log.c:196:Starting log file for FreeTDS 0.83.dev.20100522
        on 2010-06-02 15:17:08 with debug flags 0x4fff.
iconv.c:337:tds_iconv_open(0x80869e8, UTF-8)
iconv.c:197:local name for ISO-8859-1 is ISO-8859-1
iconv.c:197:local name for UTF-8 is UTF-8
iconv.c:197:local name for UCS-2LE is UCS-2LE
iconv.c:197:local name for UCS-2BE is UCS-2BE
iconv.c:365:setting up conversions for client charset "UTF-8"
iconv.c:367:preparing iconv for "UTF-8" <-> "UCS-2LE" conversion
iconv.c:408:preparing iconv for "ISO-8859-1" <-> "UCS-2LE" conversion
iconv.c:411:tds_iconv_open: done
net.c:225:Connecting to ****** port 1433 (TDS version 7.1)
net.c:281:tds_open_socket: connect(2) returned "Operation now in progress"
net.c:321:tds_open_socket() succeeded
util.c:162:Changed query state from DEAD to IDLE
net.c:792:Sending packet
0000 12 01 00 34 00 00 00 00-00 00 15 00 06 01 00 1b |...4.... ........|
0010 00 01 02 00 1c 00 0c 03-00 28 00 04 ff 08 00 01 |........ .(......|
0020 55 00 00 02 4d 53 53 51-4c 53 65 72 76 65 72 00 |U...MSSQ LServer.|
0030 9d 51 00 00            -                        |.Q..|

net.c:608:Received header
0000 04 01 00 25 00 00 01 00-                        |...%....|

net.c:662:Received packet
0000 04 01 00 25 00 00 01 00-00 00 15 00 06 01 00 1b |...%.... ........|
0010 00 01 02 00 1c 00 01 03-00 1d 00 00 ff 0a 00 09 |........ ........|
0020 e3 00 00 02 00         -                        |.....|

login.c:1044:detected flag 2
login.c:771:quietly sending TDS 7+ login packet
token.c:328:tds_process_login_tokens()
net.c:608:Received header
0000 04 01 01 a5 00 38 01 00-                        |.....8..|

net.c:662:Received packet
<CLIP>

token.c:337:looking for login token, got  e3(ENVCHANGE)
token.c:122:tds_process_default_tokens() marker is e3(ENVCHANGE)
token.c:337:looking for login token, got  ab(INFO)
token.c:122:tds_process_default_tokens() marker is ab(INFO)
token.c:2708:tds_process_msg() reading message 5701 from server
token.c:2781:tds_process_msg() calling client msg handler
token.c:2794:tds_process_msg() returning TDS_SUCCEED
token.c:337:looking for login token, got  e3(ENVCHANGE)
token.c:122:tds_process_default_tokens() marker is e3(ENVCHANGE)
token.c:2539:tds_process_env_chg(): 5 bytes of collation data received
token.c:2540:tds->collation was
0000 00 00 00 00 00         -                        |.....|

iconv.c:1025:setting server single-byte charset to "CP1252"
token.c:2550:tds->collation now
0000 0b 04 f0 20 00         -                        |... .|

token.c:337:looking for login token, got  e3(ENVCHANGE)
token.c:122:tds_process_default_tokens() marker is e3(ENVCHANGE)
token.c:337:looking for login token, got  ab(INFO)
token.c:122:tds_process_default_tokens() marker is ab(INFO)
token.c:2708:tds_process_msg() reading message 5703 from server
token.c:2781:tds_process_msg() calling client msg handler
token.c:2794:tds_process_msg() returning TDS_SUCCEED
token.c:337:looking for login token, got  ad(LOGINACK)
token.c:372:server reports TDS version 71.0.0.1
token.c:374:Product name for 0x71000001 is 2000 SP1
token.c:337:looking for login token, got  e3(ENVCHANGE)
token.c:122:tds_process_default_tokens() marker is e3(ENVCHANGE)
token.c:2597:changing block size from 4096 to 4096
token.c:337:looking for login token, got  fd(DONE)
token.c:122:tds_process_default_tokens() marker is fd(DONE)
token.c:2459:tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 0
token.c:2475:tds_process_end() state set to TDS_IDLE
token.c:2490:                rows_affected = 0
token.c:437:tds_process_login_tokens() returning TDS_SUCCEED
mem.c:615:tds_free_all_results()
util.c:162:Changed query state from IDLE to QUERYING
write.c:140:tds_put_string converting 19 bytes of "set textsize 64512 "
write.c:168:tds_put_string wrote 38 bytes
util.c:162:Changed query state from QUERYING to PENDING
net.c:792:Sending packet
0000 01 01 00 2e 00 00 00 00-73 00 65 00 74 00 20 00 |........ s.e.t. .|
0010 74 00 65 00 78 00 74 00-73 00 69 00 7a 00 65 00 |t.e.x.t. s.i.z.e.|
0020 20 00 36 00 34 00 35 00-31 00 32 00 20 00       | .6.4.5. 1.2. .|

token.c:539:tds_process_tokens(0x80869e8, 0xffffbf9c, 0xffffbf98, 0x100)
util.c:162:Changed query state from PENDING to READING
net.c:608:Received header
0000 04 01 00 11 00 38 01 00-                        |.....8..|

net.c:662:Received packet
0000 04 01 00 11 00 38 01 00-fd 00 00 be 00 00 00 00 |.....8.. ........|
0010 00                     -                        |.|

token.c:554:processing result tokens.  marker is  fd(DONE)
token.c:2459:tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 0
token.c:2475:tds_process_end() state set to TDS_IDLE
util.c:162:Changed query state from READING to IDLE
token.c:2490:                rows_affected = 0
util.c:110:logic error: cannot change query state from IDLE to PENDING
token.c:539:tds_process_tokens(0x80869e8, 0xffffbf9c, 0xffffbf98, 0x100)
token.c:542:tds_process_tokens() state is COMPLETED
mem.c:615:tds_free_all_results()
util.c:162:Changed query state from IDLE to QUERYING
write.c:140:tds_put_string converting 67 bytes of "UPDATE Mod_Texts SET 
ContentData='Найти' WHERE ContentId=6824
"
write.c:168:tds_put_string wrote 124 bytes
util.c:162:Changed query state from QUERYING to PENDING
net.c:792:Sending packet
0000 01 01 00 84 00 00 01 00-55 00 50 00 44 00 41 00 |........ U.P.D.A.|
0010 54 00 45 00 20 00 4d 00-6f 00 64 00 5f 00 54 00 |T.E. .M. o.d._.T.|
0020 65 00 78 00 74 00 73 00-20 00 53 00 45 00 54 00 |e.x.t.s.  .S.E.T.|
0030 20 00 43 00 6f 00 6e 00-74 00 65 00 6e 00 74 00 | .C.o.n. t.e.n.t.|
0040 44 00 61 00 74 00 61 00-3d 00 27 00 1d 04 30 04 |D.a.t.a. =.'...0.|
0050 39 04 42 04 38 04 27 00-20 00 57 00 48 00 45 00 |9.B.8.'.  .W.H.E.|
0060 52 00 45 00 20 00 43 00-6f 00 6e 00 74 00 65 00 |R.E. .C. o.n.t.e.|
0070 6e 00 74 00 49 00 64 00-3d 00 36 00 38 00 32 00 |n.t.I.d. =.6.8.2.|
0080 34 00 0a 00            -                        |4...|

token.c:539:tds_process_tokens(0x80869e8, 0xffffc4c0, (nil), 0x6914)
util.c:162:Changed query state from PENDING to READING
net.c:608:Received header
0000 04 01 00 11 00 38 01 00-                        |.....8..|

net.c:662:Received packet
0000 04 01 00 11 00 38 01 00-fd 10 00 c5 00 01 00 00 |.....8.. ........|
0010 00                     -                        |.|

token.c:554:processing result tokens.  marker is  fd(DONE)
token.c:2459:tds_process_end: more_results = 0
                was_cancelled = 0
                error = 0
                done_count_valid = 1
token.c:2475:tds_process_end() state set to TDS_IDLE
util.c:162:Changed query state from READING to IDLE
token.c:2490:                rows_affected = 1
util.c:110:logic error: cannot change query state from IDLE to PENDING
token.c:539:tds_process_tokens(0x80869e8, 0xffffc4c0, (nil), 0x6914)
token.c:542:tds_process_tokens() state is COMPLETED
mem.c:615:tds_free_all_results()
util.c:162:Changed query state from IDLE to DEAD



Niklas