[freetds] Character set problem

Lowden, James K LowdenJK at bernstein.com
Wed Sep 15 11:59:30 EDT 2004


-----Original Message-----
From: Hannu Niemi
Sent: Wednesday, September 15, 2004 5:12 AM

In short, the problem is our national characters (with umlauts) that 
are converted to question marks (in SELECT queries at least). When 
debugging for example 'a umlaut' is hex 84 (CP850?). The server 
collation is SQL_Scandinavian_CP850_CI_AS.
---End Original Message---

Hej Hannu! 

I assume you're using TDS 0.62 or later.  

Show us your freetds.conf and a short TDSDUMP log, just to make sure things are set up right.  It would be good if the log included a query with a failed conversion.  

What is the column's datatype?  Unicode datatypes (nchar, nvarchar) have seen more testing than single-byte charsets.  Are both broken for you?  

http://www.microsoft.com/globaldev/nlsweb/default.asp?submitted=40b
http://www.microsoft.com/globaldev/reference/oem/850.htm
http://www.microsoft.com/globaldev/reference/sbcs/1252.htm


As a Finnish speaker, is your server configured for Locale ID 0x40B?  In src/tds/iconv.c::lcid2charset(), that's interpreted as code page 1252.  That may not be right.  You said your 'a umlaut' is hex 84, which implies CP850; for CP1252, it's 0xE4.  0x84 is a DOUBLE LOW-9 QUOTATION MARK, which quite likely isn't in your client character set, causing FreeTDS to render it as '?'.  

IOW, if your server uses Locale ID 0x40B, I think FreeTDS may be mapping your server's char/varchar data according to the wrong charset.  It's using charset 1252, when it should use charset 850.  

What do these queries return?

1> sp_helpsort
2> select char(0xE4), char(0x84)
3> go
 Server default collation
 ------------------------------
 Latin1-General, binary sort

 - -
 ä ,

I don't see a conversion to any 850 charset, meaning FreeTDS won't tell iconv(3) that char/varchar data arriving from the server are encoded as CP850.  If that's the problem, your TDSDUMP log should provide enough information for us to generate a patch.  

By the way, as long as it's broken ;-) you shouldn't be able to insert 'a umlaut', either.  The INSERT statement should fail (not convert your data to '?').  

Halsingar,

--jkl

-----------------------------------------
The information contained in this transmission may contain privileged and confidential information and is intended only for the use of the person(s) named above. If you are not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender immediately by reply e-mail and destroy all copies of the original message. Please note that we do not accept account orders and/or instructions by e-mail, and therefore will not be responsible for carrying out such orders and/or instructions.
If you, as the intended recipient of this message, the purpose of which is to inform and update our clients, prospects and consultants of developments relating to our services and products, would not like to receive further e-mail correspondence from the sender, please "reply" to the sender indicating your wishes.  In the U.S.: 1345 Avenue of the Americas, New York, NY 10105.





More information about the FreeTDS mailing list