FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Frediano Ziglio [mailto:freddyz77 AT tin.it]
> Sent: April 30, 2003 2:01 AM
> Il mar, 2003-04-29 alle 23:56, Lowden, James K ha scritto:
> > > From: Frediano Ziglio [mailto:freddyz77 AT tin.it]
> > > Sent: April 29, 2003 2:37 PM
> > 
> > To do this right, we need a map from Microsoft's charset 
> > names to iconv names.  
> 
> Function is in CVS since a month... ms use a number to specify charset
> (lcid)

Ah, right, of course.  I even modified it!  

> > I set up the TDSICONVINFO* while adjusting the 
> > column_length, when I know
> > the server's and client's character set.  ATM is relies on 
> > the one-and-only
> > TDSICONVINFO structure, in the TDSSOCKET.  To make it 
> > general, we will need
> > utility functions to search the socket's array based on a 
> > pair of charset names.  
> 
> if I understand the pair mean client (constant for all connection) and
> server charset

Yes, I think you understand correctly.  It's true, the client charset is the
same for all TDSICONVINFO structures belonging to a TDSSOCKET.  

> > I think an array for TDSSOCKET::iconv_info is better than a 
> linked list,
> > because it will be reallocated infrequently.  Tell me if you think
> > different.  
> 
> All I need is some simple functions:
> - TDS_INT tds_conversion_init(TDSSOCKET*);
> - void tds_conversion_free(TDSSOCKET*);
> - TDSICONVINFO* tds_conversion_get(TDSSOCKET*,const TDS_ENCODING*)
> - TDSICONVINFO* tds_conversion_get_default(TDSSOCKET*)

Basically.  From where I sit, tds_conversion_free() will be just free(3),
because the array will be TDSICONVINFO*, not TDSICONVINFO**.  And I like the
idea that tds_conversion_get() will append to the array (with realloc) if
the conversion it specifies isn't already present.  

> (they sound a bit OOP, don't they?)

Yes. :-)

> Perhaps a "do not free this conversion" flag is useful or have some
> entry (0 and 1, for example) fixed ?

My thought is, any time a new conversion is required, add it to the socket's
TDSICONVINFO array.  Don't free the array until we free the connection.
That way, we're not constantly allocating and deallocating the same iconv
conversion descriptors.  After all, successive queries will tend to require
the same conversions.  

> You did a very good job with charsets. I don't know tds_get_char_data
> function...

Thanks.  I'm not done, but I think we're beginning to understand the issues.


> > we cannot silently insert a '?' whenever the 
> > conversion fails.   
> 
> It sounds ok. Perhaps a flag to select the behavior...

Maybe.  I don't know how tunable we want to make it.  It seems to me
Sybase's choices are reasonable: never write unconvertible characters, and
emit a message on reads.  That way, the application can deal with it.  The
only weakness I can see is that it's very difficult for the application to
know which characters couldn't be converted.  But I think this is fairly
minor: applications are not routinely in the business of displaying data
from an incompatible source.  Databases, in other words, are different from
email.  

> > I think it's OK to abort reading BIG5 data on failed conversions, too.  
> 
> Why treat big5 in a different way??

As a corollary to the above.  Applications normally will have a compatible
client character set.  Conversion errors should be rare, and will typically
require DBA intervention.  (In this sense, these errors should be thought of
as "unrecoverable").  With many encodings, substituting '?' and continuing
with the next input character isn't hard and is convenient for the client.
BIG5 and other stateful encodings make "skipping to the next input
character" hard.  I question the value of complex recovery logic for an
arcane unrecoverable error.  I say we punt intead.  

(In American football, when a team gets into a situation where they can't
carry the ball forward, they're forced to give it to their opponent.
Usually, they choose to kick the ball high and far, putting they other side
as far away from their goal as possible.  This kick is called a "punt".
It's a tactical choice, taken whenever the offense has failed.)

> query.c need some update... we use conversions here.

Noted.  It might help if you dropped in some "FIXME ICONV" comments, unless
you think it's obvious.  I haven't looked closely at query.c yet.  

> PS: I'm finishing some bad works in 0.61 (mainly RPC and dynamic in
> ODBC, this should improve performance in prepared query). Also I
> discovered that mssql7+ do ODBC escaping directly.

Excellent!  

Regards, 

--jkl


The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.