FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

Frediano Ziglio wrote:
> Il giorno gio, 11/09/2008 alle 12.01 -0400, James K. Lowden ha scritto:
> > ZIGLIO, Frediano, VF-IT wrote:
> > > 
> > > Well... finally I decided to add a parameter in libTDS (still I
> > > don't know where, probably TDSSOCKET) to have a sort of "no convert
> > > please" for characters. This to support SQL_C_CHAR in ODBC. This
> > > flag will disable characters conversions for data only (rows and
> > > parameters, not column names and similar).
> > 
> > Let's keep the from-to model and not adopt a flag-based override.  
> > 
> 
> Mmm... well.. I would prefer ucs4 but this is not the problem.
> 
> First: ODBC have 2 client encoding, one single byte and one "wide" and
> can be mixed even on 2 columns (well.. even on a single column if we
> want!).

I think I understand, and I think you may be right about the
non-conversion flag.  I have a few general questions, if I may.  Then near
the end of this message I try to restate your case, just to make sure I
understand.  Let me know if I got it right.  

First: What do we mean by "wide"?  UCS-2?  UTF-16?  UCS-4?  UTF-8?  Is it
compile-time dependent?  Run-time?  

I've read the ML threads, but I'm still confused, as you can see.  

Second: Where is the documentation? (!)  I usually use
http://msdn.microsoft.com/en-us/library/ms714562(VS.85).aspx.  On
http://msdn.microsoft.com/en-us/library/ms713539(VS.85).aspx I see a
reference to a "ODBC 3.5 (or later) Unicode Driver".  Is that what we're
building?  

Third: Cf. http://msdn.microsoft.com/en-us/library/ms714102(VS.85).aspx

To support Unicode, do we provide new functions, or new values (e.g.
SQL_WCHAR) to existing functions?  (I suppose no new functions.)  

Fourth: Can you show me the API function call sequence that mixes
encodings for a single column?  I assume the server sends all values for a
column using a single encoding; I assume the client may somehow ask for
different encodings as the data are fetched.  

> Honestly I started coding SQL_C_WCHAR and *W stuff (I have a patch but
> mostly a proof of concept) using UTF-8 but I don't like too many
> encoding.

Maybe it would be easier to create a BRANCH_UNICODE branch in CVS where
you could work and we could refer to the source code.  

> If column_size is
> client size then column_data should be the client data! So instead of
> allocating and copy data I just put a curcol->column_data = src in the
> right place and it works! Simple, one single conversion from client to
> packet buffer without many copies... and with less allocations too! 

That sounds right.  

> (The
> extension is why don't let libTDS do the conversion, any conversion,
> using column_type and on_server.column_type... but don't consider this
> now).

That was always my design: keep track of both encodings, and have libtds
convert nearest the wire.  All internal storage uses the client's
encoding.  Server encoding is used only to communicate with the server.  

> The real problem raise from columns.... Columns
> are read in a buffer with a given (client) encoding. The row buffer is
> always encoded with a single encoding. This is correct for dblib and
> ctlib but ODBC have to mix 2 encodings. 

OK.  IIUC we have TDSCOLUMN::char_conv, but different rows may be
requested using different client encodings.  So, libtds fetches the data
from the wire, converts using char_conv, but then e.g. SQLGetData may use
a TargetType of SQL_C_CHAR for one row, and SQL_C_WCHAR for another.  

> So after all that thoughts I think that using server encodings on rows
> would help to avoid two conversions. 

So, your solution is to not convert in libtds.  Keep the server encoding,
whatever it is, for every column on every fetch.  Delay conversion until
SQLGetData or until writing the row buffer to the bound buffer.  

To avoid conversion in libtds, you propose a flag TDSSOCKET::no_data_conv.
 

I don't like negative flags.  What do you think of TDSSOCKET::use_iconv
instead?  
        if (tds->use_iconv) {
                tds_iconv(...);
        }

I suppose that is OK.  I don't have any better suggestion.  

Regards, 

--jkl