Skip to Content.
Sympa Menu

freetds - RE: [freetds] charset, mssql

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: Frediano Ziglio <freddyz77 AT tin.it>
  • To: FreeTDS Development Group <freetds AT lists.ibiblio.org>
  • Subject: RE: [freetds] charset, mssql
  • Date: 30 Apr 2003 08:01:16 +0200

Il mar, 2003-04-29 alle 23:56, Lowden, James K ha scritto:
> Hi Freddy,
>
> > From: Frediano Ziglio [mailto:freddyz77 AT tin.it]
> > Sent: April 29, 2003 2:37 PM
> >
> > Current CVS do not work with TDS8 (mssql2k protocol).
>
> I don't have access to a 2K server, so this will be a challenge. :-(
>
> > If you want to try FreeTDS snapshot use TDS7 even for mssql2k.
> > Problem is due to conversion stuff. During login TDS8 server send
> > default collation for database using env type 7. This collation is
> > readed and used to setup iconv according (using collation charset)
>
> To do this right, we need a map from Microsoft's charset names to iconv
> names.
>

Function is in CVS since a month... ms use a number to specify charset
(lcid)

> > however all next conversation use still ucs2...
> > TDS7 do not have this problem cause return collation information in a
> > different way (not still supported by FreeTDS, env types 5 and 6).
>
> UTF-8 clients really need (at least) two TDSICONVINFO structures:
>
> 1. client <-> UCS-2 [nchar]
> 2. client <-> server charset [char]
>
> TDS 8.0 clients need more, potentially, one for each distinct client-server
> charset pair.
>

Sybase can have 3 conversion:
- default charset
- utf8 (large data, not still supported)
- utf16 (large data, not still supported)

(unichar are not supported in my test db so I can't try)

> My solution so far:
>
> TDSSOCKET::iconv_info becomes an array (not done yet).
>
> TDSCOLINFO gets a new member, TDSICONVINFO *iconv_info, that points to one
> of the elements in TDSSOCKET::iconv_info. (Done already, but only for UCS-2
> columns, and not committed). If the server column doesn't need iconv
> intervention, its iconv_info is NULL.
>
> I set up the TDSICONVINFO* while adjusting the column_length, when I know
> the server's and client's character set. ATM is relies on the one-and-only
> TDSICONVINFO structure, in the TDSSOCKET. To make it general, we will need
> utility functions to search the socket's array based on a pair of charset
> names.
>

if I understand the pair mean client (constant for all connection) and
server charset

> I think an array for TDSSOCKET::iconv_info is better than a linked list,
> because it will be reallocated infrequently. Tell me if you think
> different.
>

All I need is some simple functions:
- TDS_INT tds_conversion_init(TDSSOCKET*);
- void tds_conversion_free(TDSSOCKET*);
- TDSICONVINFO* tds_conversion_get(TDSSOCKET*,const TDS_ENCODING*)
- TDSICONVINFO* tds_conversion_get_default(TDSSOCKET*)
(they sound a bit OOP, don't they?)

> > TDS7 and TDS8 handle collation in slightly different ways:
> >
> > TDS7
> > - every database have it's collation
> ^^^^^^^^ every *server*
> ...

Mmm... I'll do some test...

>
> > I think we should have some fixed iconv structures (like database one
> > and ucs2) and cache others conversions.
>
> Right. That's my array in TDSSOCKET. :-)
>

Perhaps a "do not free this conversion" flag is useful or have some
entry (0 and 1, for example) fixed ?

> > Also tds_get_string is not sufficient
>
> Cf. tds_get_char_data(). Takes a TDSCOLINFO structure. Soon, that
> structure will have a pointer to a valid (cached) TDSICONVINFO structure.
> Then it will be easy to decide what kind of conversion is needed.
>

You did a very good job with charsets. I don't know tds_get_char_data
function...

> If tds_get_char_data() encounters a charset pair not already present in the
> TDSSOCKET::iconv_info array, it will have to ask for one to be added. That
> function doesn't exist yet, either.
>

tds_conversion_get ? :)

> tds_get_string() is only useful if you *know* the inbound string is UCS-2
> and you don't have a TDSCOLINFO structure. I.e., at login and when reading
> metadata.
>
> We had a long discussion about handling conversion errors. Per my earlier
> message, we cannot silently insert a '?' whenever the conversion fails. We
> must emit an error message. For reads, we can proceed, but for writes we
> must refuse to process the data.
>

It sounds ok. Perhaps a flag to select the behavior...

> I think it's OK to abort reading BIG5 data on failed conversions, too.
>

Why threat big5 in a different way??

> Regards,
>

query.c need some update... we use conversions here.

bye
freddy77

PS: I'm finishing some bad works in 0.61 (mainly RPC and dynamic in
ODBC, this should improve performance in prepared query). Also I
discovered that mssql7+ do ODBC escaping directly.





Archive powered by MHonArc 2.6.24.

Top of Page