FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

All, 

Bill's message made me think about something we haven't fully considered:
How to deal with unconvertible characters?  

If iconv(3) returns an error, we skip over the input character and place a
'?' in the output buffer.  That choice is an outgrowth of our ASCII<->UCS2
assumptions: sending data to the server, all conversions were possible;
fetching from the server, most were, and what wasn't was handed to the
application as '?'.  

Basically, this is a bad idea.  I don't mind being in the business of
converting character data from one encoding to another, but I don't think
FreeTDS should munge the data.  If our putative UTF-8 client decides to
insert a row into a ISO-8859-1 server, I doubt he wants a '?' plopped in
wherever we couldn't encode his data.  

Sybase does this, which we could emulate:

<quote>
Error handling in character set conversion

When Adaptive Server encounters a conversion error in the data being
entered, it generates this message: 

Msg 2402, Severity 16 (EX_USER):
Error converting client characters into server's character set. Some
character(s) could not be converted.

A conversion error prevents query execution on insert and update statements.
If this occurs, review your data for problem characters and replace them.

When Adaptive Server encounters a conversion error while sending data to the
client, it replaces the bytes of the suspect characters with ASCII question
marks (?). However, the query batch continues to completion. When the
statement is complete, Adaptive Server sends the following message: 

Msg 2403, Severity 16 (EX_INFO):
WARNING! Some character(s) could not be converted into client's character
set. Unconverted bytes were changed to question marks ('?'). 
</quote>

Additionally, with some cleverness, we could provide a place for the caller
to install a callback function, similar to errhandler().  That function
could adjust the buffers and return a value telling us whether or not to
continue.  I don't know if there's any value in that.  

--jkl



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.