FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: ZIGLIO Frediano [mailto:Frediano.Ziglio AT vodafoneomnitel.it]
> Sent: April 2, 2003 3:41 AM
> 
> > For Latin1, we know the max is 2 byte/char.  Therefore, if your 
> > example client
> > asks for the column size, we'd better say 3 * 2 = 6.  Everytime. 
> 
> Well, so for every conversion you should know how many bytes 
> it take...
> I think any ms charset -> utf8 take 3 bytes maximun.

I haven't looked into which character sets Microsoft supports, but that's
bound to expand.  I agree with your point, though: we should establish the
max UTF-8 byte/char based on the character set we're using.  

I think UTF-8 is the only encoding we'll deal with that has a variable
byte/char ratio.  

> > In ODBC, I think you want SQL_COPT_SS_TRANSLATE.  The 
> > documentation says it
> > "causes the driver to translate characters between the client 
> > and server code pages".  
> 
> Well, all *SS* macro are sql server macro, not odbc macro.
> They are all extension to ODBC... 

Yes, they are extensions to the ODBC spec.  This one is implemented entirely
in the driver.  Other exetensions, such as bcp, would involve driver and
server.  

> Currently my Sybase server (only a test server) is 11.9.2. So 
> it do not
> support utf8. I think someone can test the behavior of Sybase 
> using utf8 data.
> - if server is utf8 and return data in iso8859-1 (iso_1) how it return
> column sizes?
> - if server is iso_1 (reinstallation required?) how it return 
> column sizes?
> - test unichar and univarchar columns
> - test longvarbinary TDS type with UTF16 inside (does server 
> use UCS2 or real UTF16?)

These would be useful and interesting tests.  I'm in the same boat you are
atm with 11.9.2.  Rather than upgrade my server, I'm going to do what makes
sense.  If someone provides countervailing information or later testing
shows we've introduced incompatible behavior, any repairs will be easier to
make because the functionality will be localized, primarily in token.c.  

> > UTF-8 had self-synchronization as part of its design; they 
> > wanted to make
> > this easy.  
> 
> Well, skipping a fixed-size character (single-byte, ucs2/4) 
> is simple. utf8
> is a bit harder. But all these charset haven't state!! big5 
> or shift-jis
> charset have state. This mean that conversion of a series of 
> bytes using
> these charset depend on internal state of iconv_t structure...

I hadn't considered stateful character sets e.g. shift-jis.  Are your
algorithms designed to cope with state?  

I follow the Sylpheed ML.  Sylpheed is an email client (and a good one!),
the brainchild of a Japanese fellow, Hiroyuki Yamamoto.  It supports UTF-8
and Japanese et. al. fonts, and uses iconv to perform conversions (because
UTF-8 is a valid email encoding but big5 and such are common local setups).
We could have a look at their approach and I could ask them how they handle
invalid sequences and nonconvertible characters in stateful encodings.  We
can't lift their code, though; it's GPL.  

> So for every charset we should know:
> - is fixed byte? if so, how many bytes take a character?
> - how many bytes can take a character?
> - does this charset have a state?
> And for every conversion we should know the ratio...

Absolutely.  That's beginning to sound like it should be a substructure of
TDSICONVINFO.  I think my byte/char lookup function should return all that
information.  

--jkl




The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.