FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

Lapo Luchini wrote:
> Daniel Fazekas <fdsubs <at> axelero.hu> writes:
> > >  however, when someone pastes a "smart-quoted" quotation mark from 
> > > word (hex92) into a field to put it into the database, it gets
> > > changed  into a question mark (hex63).
> > 
> > Try changing your client charset setting in freetds.conf to 
> > windows-1252.
> >
> > iso-8859-1 doesn't contain the "smart quotes."
> 
> I noticed that the server encoding can't be changed, and for char/text
> fields ISO-8859-1 is used.
> As noted ISO-8859-1 doesn't have quotes at 0x92.
> 
> But how changing the client encoding to CP1252 could solve the problem?
> The server CP1252 would be transformed from ISO-8859-1 (while it is
> not!) to CP1252, and a conversion error would be there anyway, as
> ISO-8859-1 doesn't have where it is the source of a conversion to UTF-8
> or to CP1252: it is the source of the conversion that it is wrong, not
> the destination.
> 
> Maybe I'm completely wrnog, but it seems to me that the "correct" thing
> to do is to use CP1252 as the *server* encoding, for single-byte fields:
> SQLServer may well state to use "iso_1" but, well, it is simply not
> true...

The patch isn't needed; it masks a configuration/application error.  

The original description of the OP's problem wasn't very clear, but I
think some basic principles were violated somehow.  If they are followed,
everything Just Works.  

Going back to the top of this message, 'someone pastes a "smart-quoted"
quotation mark from word (hex92) into a field to put it into the
database'.  How?  From a client, presumably, running an application or
tool or utility, presumably.  He didn't pry open the disk and shove bits
in.  If he's using FreeTDS, it's very important that the client encoding
reflect the data's encoding.  IOW, if the data are windows-1252, set up
the client to use that encoding.  

But the problem is even more fundamental: the column in the table
apparently is set to be ISO-8859-1, yet it is holding data encoded as
window-1252.  The database doesn't *know* how the data are encoded; it's
relying on you to tell it.  It will let you store Greek there, because
it's all ones and zeros.  It won't mind.  But when you retrieve it, it
will declare it's ISO-8859.  FreeTDS will pay attention to that
declaration, and attempt to convert the data to the client's preferred
encoding.  The results won't be pretty, not Greek, not English, not
intelligible.  

Moral: don't let the client put windows-1252 data in a column encoded as
ISO 8859-1.  Don't declare to the server that your windows-1252 data are
ISO 8859-1.  Be honest!  Say what you've got, and maybe use a unicode
datatype e.g. NVARCHAR to store it.  Then things will go much easier.  

Regards, 

--jkl