FreeTDS Development Group

[Frediano Ziglio <freddyz77 AT tin.it>]

Il mar, 2003-04-29 alle 23:56, Lowden, James K ha scritto:
> Hi Freddy, 
> 
> > From: Frediano Ziglio [mailto:freddyz77 AT tin.it]
> > Sent: April 29, 2003 2:37 PM
> > 
> > Current CVS do not work with TDS8 (mssql2k protocol).
> 
> I don't have access to a 2K server, so this will be a challenge.  :-(
> 
> > If you want to try FreeTDS snapshot use TDS7 even for mssql2k.
> > Problem is due to conversion stuff. During login TDS8 server send
> > default collation for database using env type 7. This collation is
> > readed and used to setup iconv according (using collation charset)
> 
> To do this right, we need a map from Microsoft's charset names to iconv
> names.  
> 

Function is in CVS since a month... ms use a number to specify charset
(lcid)

> > however all next conversation use still ucs2...
> > TDS7 do not have this problem cause return collation information in a
> > different way (not still supported by FreeTDS, env types 5 and 6).
> 
> UTF-8 clients really need (at least) two TDSICONVINFO structures: 
> 
> 1.  client <-> UCS-2 [nchar]
> 2.  client <-> server charset [char]
> 
> TDS 8.0 clients need more, potentially, one for each distinct client-server
> charset pair.  
> 

Sybase can have 3 conversion:
- default charset
- utf8 (large data, not still supported)
- utf16 (large data, not still supported)

(unichar are not supported in my test db so I can't try)

> My solution so far:
> 
> TDSSOCKET::iconv_info becomes an array (not done yet).  
> 
> TDSCOLINFO gets a new member, TDSICONVINFO *iconv_info, that points to one
> of the elements in TDSSOCKET::iconv_info.  (Done already, but only for UCS-2
> columns, and not committed).  If the server column doesn't need iconv
> intervention, its iconv_info is NULL.  
> 
> I set up the TDSICONVINFO* while adjusting the column_length, when I know
> the server's and client's character set.  ATM is relies on the one-and-only
> TDSICONVINFO structure, in the TDSSOCKET.  To make it general, we will need
> utility functions to search the socket's array based on a pair of charset
> names.  
> 

if I understand the pair mean client (constant for all connection) and
server charset

> I think an array for TDSSOCKET::iconv_info is better than a linked list,
> because it will be reallocated infrequently.  Tell me if you think
> different.  
> 

All I need is some simple functions:
- TDS_INT tds_conversion_init(TDSSOCKET*);
- void tds_conversion_free(TDSSOCKET*);
- TDSICONVINFO* tds_conversion_get(TDSSOCKET*,const TDS_ENCODING*)
- TDSICONVINFO* tds_conversion_get_default(TDSSOCKET*)
(they sound a bit OOP, don't they?)

> > TDS7 and TDS8 handle collation in slightly different ways:
> > 
> > TDS7
> > - every database have it's collation
>           ^^^^^^^^ every *server*
> ...

Mmm... I'll do some test...

> 
> > I think we should have some fixed iconv structures (like database one
> > and ucs2) and cache others conversions.
> 
> Right.  That's my array in TDSSOCKET.  :-)
> 

Perhaps a "do not free this conversion" flag is useful or have some
entry (0 and 1, for example) fixed ?

> > Also tds_get_string is not sufficient 
> 
> Cf. tds_get_char_data().  Takes a TDSCOLINFO structure.  Soon, that
> structure will have a pointer to a valid (cached) TDSICONVINFO structure.
> Then it will be easy to decide what kind of conversion is needed.  
> 

You did a very good job with charsets. I don't know tds_get_char_data
function...

> If tds_get_char_data() encounters a charset pair not already present in the
> TDSSOCKET::iconv_info array, it will have to ask for one to be added.  That
> function doesn't exist yet, either.  
> 

tds_conversion_get ? :)

> tds_get_string() is only useful if you *know* the inbound string is UCS-2
> and you don't have a TDSCOLINFO structure.  I.e., at login and when reading
> metadata.  
> 
> We had a long discussion about handling conversion errors.  Per my earlier
> message, we cannot silently insert a '?' whenever the conversion fails.  We
> must emit an error message.  For reads, we can proceed, but for writes we
> must refuse to process the data.  
> 

It sounds ok. Perhaps a flag to select the behavior...

> I think it's OK to abort reading BIG5 data on failed conversions, too.  
> 

Why threat big5 in a different way??

> Regards, 
> 

query.c need some update... we use conversions here.

bye
  freddy77

PS: I'm finishing some bad works in 0.61 (mainly RPC and dynamic in
ODBC, this should improve performance in prepared query). Also I
discovered that mssql7+ do ODBC escaping directly.