[freetds] charset, mssql
freddyz77 at tin.it
Wed Apr 30 02:01:16 EDT 2003
Il mar, 2003-04-29 alle 23:56, Lowden, James K ha scritto:
> Hi Freddy,
> > From: Frediano Ziglio [mailto:freddyz77 at tin.it]
> > Sent: April 29, 2003 2:37 PM
> > Current CVS do not work with TDS8 (mssql2k protocol).
> I don't have access to a 2K server, so this will be a challenge. :-(
> > If you want to try FreeTDS snapshot use TDS7 even for mssql2k.
> > Problem is due to conversion stuff. During login TDS8 server send
> > default collation for database using env type 7. This collation is
> > readed and used to setup iconv according (using collation charset)
> To do this right, we need a map from Microsoft's charset names to iconv
Function is in CVS since a month... ms use a number to specify charset
> > however all next conversation use still ucs2...
> > TDS7 do not have this problem cause return collation information in a
> > different way (not still supported by FreeTDS, env types 5 and 6).
> UTF-8 clients really need (at least) two TDSICONVINFO structures:
> 1. client <-> UCS-2 [nchar]
> 2. client <-> server charset [char]
> TDS 8.0 clients need more, potentially, one for each distinct client-server
> charset pair.
Sybase can have 3 conversion:
- default charset
- utf8 (large data, not still supported)
- utf16 (large data, not still supported)
(unichar are not supported in my test db so I can't try)
> My solution so far:
> TDSSOCKET::iconv_info becomes an array (not done yet).
> TDSCOLINFO gets a new member, TDSICONVINFO *iconv_info, that points to one
> of the elements in TDSSOCKET::iconv_info. (Done already, but only for UCS-2
> columns, and not committed). If the server column doesn't need iconv
> intervention, its iconv_info is NULL.
> I set up the TDSICONVINFO* while adjusting the column_length, when I know
> the server's and client's character set. ATM is relies on the one-and-only
> TDSICONVINFO structure, in the TDSSOCKET. To make it general, we will need
> utility functions to search the socket's array based on a pair of charset
if I understand the pair mean client (constant for all connection) and
> I think an array for TDSSOCKET::iconv_info is better than a linked list,
> because it will be reallocated infrequently. Tell me if you think
All I need is some simple functions:
- TDS_INT tds_conversion_init(TDSSOCKET*);
- void tds_conversion_free(TDSSOCKET*);
- TDSICONVINFO* tds_conversion_get(TDSSOCKET*,const TDS_ENCODING*)
- TDSICONVINFO* tds_conversion_get_default(TDSSOCKET*)
(they sound a bit OOP, don't they?)
> > TDS7 and TDS8 handle collation in slightly different ways:
> > TDS7
> > - every database have it's collation
> ^^^^^^^^ every *server*
Mmm... I'll do some test...
> > I think we should have some fixed iconv structures (like database one
> > and ucs2) and cache others conversions.
> Right. That's my array in TDSSOCKET. :-)
Perhaps a "do not free this conversion" flag is useful or have some
entry (0 and 1, for example) fixed ?
> > Also tds_get_string is not sufficient
> Cf. tds_get_char_data(). Takes a TDSCOLINFO structure. Soon, that
> structure will have a pointer to a valid (cached) TDSICONVINFO structure.
> Then it will be easy to decide what kind of conversion is needed.
You did a very good job with charsets. I don't know tds_get_char_data
> If tds_get_char_data() encounters a charset pair not already present in the
> TDSSOCKET::iconv_info array, it will have to ask for one to be added. That
> function doesn't exist yet, either.
tds_conversion_get ? :)
> tds_get_string() is only useful if you *know* the inbound string is UCS-2
> and you don't have a TDSCOLINFO structure. I.e., at login and when reading
> We had a long discussion about handling conversion errors. Per my earlier
> message, we cannot silently insert a '?' whenever the conversion fails. We
> must emit an error message. For reads, we can proceed, but for writes we
> must refuse to process the data.
It sounds ok. Perhaps a flag to select the behavior...
> I think it's OK to abort reading BIG5 data on failed conversions, too.
Why threat big5 in a different way??
query.c need some update... we use conversions here.
PS: I'm finishing some bad works in 0.61 (mainly RPC and dynamic in
ODBC, this should improve performance in prepared query). Also I
discovered that mssql7+ do ODBC escaping directly.
More information about the FreeTDS