FreeTDS Development Group

[ZIGLIO Frediano <Frediano.Ziglio AT vodafoneomnitel.it>]

> 
> Functions such as tds7_get_data_info() that set the column 
> width should
> fixup the width according to the client charset.  We need to keep
> server-side and client-side representations of column sizes; 
> most of libtds
> only needs to know about the client side.  The only time 
> server-side sizes
> are important is when calling functions that perform conversions.  
> 

This is a problem. Assuming server is latin1 and we convert to utf8; we have
a varchar(3) with all accent letter (like 'ééé'). Converting to utf8 result
6 bytes... If client ask for data length we return 6 characters, but if
client ask for column size? IMHO we should return 3, cause server size is
3.. but if client use column size to compute size of buffer it fail...

> Fixing up the metadata and adding charset information to 
> TDSICONVINFO will
> let us remove ad hoc hard-coded fixups sprinkled throughout 
> the code.  It
> won't be easy, but it will be possible.  
> 
> Freddy: tds7_get_data_info() discards 5 collation bytes 
> because we haven't
> installed our collation structure in TDSCOLINFO.  When we do, 
> we'll have
> server-reported per-column charset information.  We'll then 
> have to make a
> design decision about whether or not to convert all such 
> columns to a single
> client charset.  
> 

In db-lib and ct-lib you set charset only before connection, so this is the
correct way. In ODBC I can't see any method to set charset... very strange..
perhaps MS use windows to discover charset (you know MS products are very
unportable...) ??
Perhaps problem with ODBC is that client can read characters using
single-byte or UCS2... (MS use very seldom utf8...)

> Radical note.  Suppose we succeed in deasciifying FreeTDS, 
> and it becomes
> fully capable of handling UCS-2 and UTF-8 clients.  Then, for a UCS-2
> client, do we convert varchar data to UCS-2?  (They arrive in 
> the server's
> single-byte charset.)  If we mimic Microsoft's Win32 client 
> libraries, the
> answer is: No.  I propose that if in freetds.conf we find 
> "client charset =
> server charset", we do no conversion at all.  
> 

I think so... perhaps initializing iconv_t to -1 and cheching before
conversion.

I discovered how to skip a character using iconv !!
Not so easy, however...
When you encounter a unconvertible character you should:
1- output shift state to a temporary buffer
2- append some byte from stopped input string (all possible)
3- convert this temporary string to a single UCS4 character (you need
another iconv_t structure)
4- add a '?' to output string, converting from a ascii to charset
destination (another iconv_t structure)
5- syncronize input with temporary and continue...

Not so easy but run in all case... step 3 and 4 can be simplified using a
callback. So we can use a structure like

struct conversions {
        char from[64];
        char to[64];
        iconv_t conv;
        iconv_t skip_cache;
        iconv_t fill_cache;
        int (*skip)(struct conversions* conv, char** in, size_t * in_len,
char** out, size_t * out_len);
};

Default skip something like

int 
default_skip(struct conversions* conv, char** in, size_t * in_len, char**
out, size_t * out_len)
{
        char temp[16];
        char temp_out[4];
        initialize skip_cache (iconv_open("UCS4",conv->from)) if not
        initialize fill_cache (iconv_open(conv->to,"US_ASCII")) if not
        out shift state to temp
        append some bytes from in string
        convert a character from temp to temp_out using skip_cache
conversion
        skip a character from in using information from previous conversion
        temp[0] = '?';
        convert a character from temp (1 byte) to out using fill_cache
conversion
}

skip for single-byte to ascii-compatible (single-byte or utf8) is something
like

int 
byte_skip(struct conversions* conv, char** in, size_t * in_len, char** out,
size_t * out_len)
{
        if (*in_len) {
                ++*in;
                --*in_len;
        }
        if (*out_len) {
                **out = '?';
                ++*out;
                --*out_len;
        }
}

freddy77

=================================
"STRICTLY PERSONAL AND CONFIDENTIAL

This message may contain confidential and proprietary material for the sole
use of the intended recipient. Any review or distribution by others is
strictly prohibited. If you are not the intended recipient please contact
the sender and delete all copies.
The contents of this message that do not relate to the official business of
our company shall be understood as neither given nor endorsed by it."

=================================