FreeTDS Development Group

[ZIGLIO Frediano <Frediano.Ziglio AT vodafoneomnitel.it>]

> > 
> > This is a problem. Assuming server is latin1 and we convert 
> > to utf8; we have
> > a varchar(3) with all accent letter (like 'ééé'). Converting 
> > to utf8 result
> > 6 bytes... If client ask for data length we return 6 
> > characters, but if
> > client ask for column size? IMHO we should return 3, cause 
> > server size is
> > 3.. but if client use column size to compute size of buffer 
> it fail...
> 
> That is exactly my point in talking about "fixing up" the 
> metadata.  The
> purpose of TDSCOLINFO is to convey the server's description 
> of the data to
> the client.  If we change the data (with iconv), then we must 
> change the
> metadata to match.  Using your example, we must make it look 
> as though the
> server had sent UTF-8.  
> 
> (A good way to think about all this, but much harder to 
> implement, is as a
> server gateway.  Pretend you're a proxy TDS server, doing 
> protocol and/or
> charset conversions by adjusting the TDS  packets flowing through you.
> Your representation to your client has to be consistent, else 
> it's useless.)
> 
> Just to be clear: the column size is a byte count, not a 
> character count.  I
> don't think the APIs ever use a character count.  
> 
> What should we say, for UTF-8?  UTF-8 uses 1-4 byte/char, so 
> 4 would be
> safe.  But, how many bytes it uses is a function of the 
> Unicode code point
> it's representing, and we know the charset we're converting from.  For
> Latin1, we know the max is 2 byte/char.  Therefore, if your 
> example client
> asks for the column size, we'd better say 3 * 2 = 6.  Everytime. 
> 

Well, so for every conversion you should know how many bytes it take...
I think any ms charset -> utf8 take 3 bytes maximun.

> Pretending for a moment that the server charset is Latin1 and 
> the client is
> UCS-2, 
> 
> > 
> > In db-lib and ct-lib you set charset only before connection, 
> > so this is the
> > correct way. In ODBC I can't see any method to set charset... 
> > very strange..
> 
> Actually, it's an option.  I should have known. :/
> 
> In ODBC, I think you want SQL_COPT_SS_TRANSLATE.  The 
> documentation says it
> "causes the driver to translate characters between the client 
> and server
> code pages".  
> 

Well, all *SS* macro are sql server macro, not odbc macro.
They are all extension to ODBC... In windows client ODBC is a per-tread
configuration (not in the registry but in the TLS..).
In *nix we haven't still this option (perhaps using glibc 2.3... called
thread-local locale, if anyone have more informations...)

> In db-lib, it's very poorly described as db-lib options 
> DBANSItoOEM and
> DBOEMtoANSI.  The documentation of dbsetopt() implies this 
> option is sent to
> the server, but I don't believe it.  Elsewhere, the docs 
> speak of using this
> "DB-Library option", and there's no DBCC or SET option for this
> functionality.  Conversion is performed by the driver.  
> 
> I'm still running 7.0, not 2000.  In SQL Server 2000, 
> individual columns in
> a result set may each have their own character set (it's no longer a
> server-wide setting).  It's not clear to me how db-lib is 
> supposed to cope
> with that; if anyone has clear information or is willing to 
> dig through the
> docs, I'd be interested.  For the present, I'd say we do any and all
> conversions if DBANSItoOEM is set (the default), else none.  
> 
...
> 
> > Perhaps problem with ODBC is that client can read characters using
> > single-byte or UCS2... (MS use very seldom utf8...)
> 
> Sybase can, though.  We should study Sybase's metadata while 
> it uses UTF-8.
> 

Currently my Sybase server (only a test server) is 11.9.2. So it do not
support utf8. I think someone can test the behavior of Sybase using utf8
data.
- if server is utf8 and return data in iso8859-1 (iso_1) how it return
column sizes?
- if server is iso_1 (reinstallation required?) how it return column sizes?
- test unichar and univarchar columns
- test longvarbinary TDS type with UTF16 inside (does server use UCS2 or
real UTF16?)

> 
> > I discovered how to skip a character using iconv !!
> 
> Maybe I'm missing something.  It doesn't look hard to me.  
> 
> For single- or double-byte encoded inputs, we know how many 
> bytes to skip.  
> 
> For UTF-8, see http://czyborra.com/utf/#UTF-8 especially this table:
> 
>  bytes | bits | representation
>      1 |    7 | 0vvvvvvv
>      2 |   11 | 110vvvvv 10vvvvvv
>      3 |   16 | 1110vvvv 10vvvvvv 10vvvvvv
>      4 |   21 | 11110vvv 10vvvvvv 10vvvvvv 10vvvvvv
> 
> Any UTF-8 value >127 is part of a multibyte character.  If 
> (byte & 64) is
> true, you have a lead byte that will be followed by some 
> "10vvvvvv" bytes; a
> simple shift-loop will get you to the end.  Insert a '?' in 
> the output, and
> continue from there.  
> 
> UTF-8 had self-synchronization as part of its design; they 
> wanted to make
> this easy.  
> 

Well, skipping a fixed-size character (single-byte, ucs2/4) is simple. utf8
is a bit harder. But all these charset haven't state!! big5 or shift-jis
charset have state. This mean that conversion of a series of bytes using
these charset depend on internal state of iconv_t structure...

So for every charset we should know:
- is fixed byte? if so, how many bytes take a character?
- how many bytes can take a character?
- does this charset have a state?
And for every conversion we should know the ratio...

I written a small program to test if a charset is single-byte or not... is
still a test but can be useful...

freddy77

=================================
"STRICTLY PERSONAL AND CONFIDENTIAL

This message may contain confidential and proprietary material for the sole
use of the intended recipient. Any review or distribution by others is
strictly prohibited. If you are not the intended recipient please contact
the sender and delete all copies.
The contents of this message that do not relate to the official business of
our company shall be understood as neither given nor endorsed by it."

=================================