FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

-----Original Message-----
From: Hannu Niemi
Sent: Wednesday, September 15, 2004 5:12 AM

In short, the problem is our national characters (with umlauts) that 
are converted to question marks (in SELECT queries at least). When 
debugging for example 'a umlaut' is hex 84 (CP850?). The server 
collation is SQL_Scandinavian_CP850_CI_AS.
---End Original Message---

Hej Hannu! 

I assume you're using TDS 0.62 or later.  

Show us your freetds.conf and a short TDSDUMP log, just to make sure things 
are set up right.  It would be good if the log included a query with a failed 
conversion.  

What is the column's datatype?  Unicode datatypes (nchar, nvarchar) have seen 
more testing than single-byte charsets.  Are both broken for you?  

http://www.microsoft.com/globaldev/nlsweb/default.asp?submitted=40b
http://www.microsoft.com/globaldev/reference/oem/850.htm
http://www.microsoft.com/globaldev/reference/sbcs/1252.htm


As a Finnish speaker, is your server configured for Locale ID 0x40B?  In 
src/tds/iconv.c::lcid2charset(), that's interpreted as code page 1252.  That 
may not be right.  You said your 'a umlaut' is hex 84, which implies CP850; 
for CP1252, it's 0xE4.  0x84 is a DOUBLE LOW-9 QUOTATION MARK, which quite 
likely isn't in your client character set, causing FreeTDS to render it as 
'?'.  

IOW, if your server uses Locale ID 0x40B, I think FreeTDS may be mapping your 
server's char/varchar data according to the wrong charset.  It's using 
charset 1252, when it should use charset 850.  

What do these queries return?

1> sp_helpsort
2> select char(0xE4), char(0x84)
3> go
 Server default collation
 ------------------------------
 Latin1-General, binary sort

 - -
 ä ,

I don't see a conversion to any 850 charset, meaning FreeTDS won't tell 
iconv(3) that char/varchar data arriving from the server are encoded as 
CP850.  If that's the problem, your TDSDUMP log should provide enough 
information for us to generate a patch.  

By the way, as long as it's broken ;-) you shouldn't be able to insert 'a 
umlaut', either.  The INSERT statement should fail (not convert your data to 
'?').  

Halsingar,

--jkl

-----------------------------------------
The information contained in this transmission may contain privileged and 
confidential information and is intended only for the use of the person(s) 
named above. If you are not the intended recipient, or an employee or agent 
responsible for delivering this message to the intended recipient, any 
review, dissemination, distribution or duplication of this communication is 
strictly prohibited. If you are not the intended recipient, please contact 
the sender immediately by reply e-mail and destroy all copies of the original 
message. Please note that we do not accept account orders and/or instructions 
by e-mail, and therefore will not be responsible for carrying out such orders 
and/or instructions.
If you, as the intended recipient of this message, the purpose of which is to 
inform and update our clients, prospects and consultants of developments 
relating to our services and products, would not like to receive further 
e-mail correspondence from the sender, please "reply" to the sender 
indicating your wishes.  In the U.S.: 1345 Avenue of the Americas, New York, 
NY 10105.