FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

Il giorno sab, 13/09/2008 alle 16.51 -0400, James K. Lowden ha scritto:
> Frediano Ziglio wrote:
> > Il giorno gio, 11/09/2008 alle 12.01 -0400, James K. Lowden ha scritto:
> > > ZIGLIO, Frediano, VF-IT wrote:
> > > > 
> > > > Well... finally I decided to add a parameter in libTDS (still I
> > > > don't know where, probably TDSSOCKET) to have a sort of "no convert
> > > > please" for characters. This to support SQL_C_CHAR in ODBC. This
> > > > flag will disable characters conversions for data only (rows and
> > > > parameters, not column names and similar).
> > > 
> > > Let's keep the from-to model and not adopt a flag-based override.  
> > > 
> > 
> > Mmm... well.. I would prefer ucs4 but this is not the problem.
> > 
> > First: ODBC have 2 client encoding, one single byte and one "wide" and
> > can be mixed even on 2 columns (well.. even on a single column if we
> > want!).
> 
> I think I understand, and I think you may be right about the
> non-conversion flag.  I have a few general questions, if I may.  Then near
> the end of this message I try to restate your case, just to make sure I
> understand.  Let me know if I got it right.  
> 
> First: What do we mean by "wide"?  UCS-2?  UTF-16?  UCS-4?  UTF-8?  Is it
> compile-time dependent?  Run-time?  
> 

It depends from DM... unixODBC use UCS-2 while iODBC use wchar_t
(usually UCS-4 on Unix.. or not?? I'm not sure, certainly Linux use
UCS-4). Using unixODBC there is a configuration flags to use wchar_t but
usually is not enabled

> I've read the ML threads, but I'm still confused, as you can see.  
> 
> Second: Where is the documentation? (!)  I usually use
> http://msdn.microsoft.com/en-us/library/ms714562(VS.85).aspx.  On
> http://msdn.microsoft.com/en-us/library/ms713539(VS.85).aspx I see a
> reference to a "ODBC 3.5 (or later) Unicode Driver".  Is that what we're
> building?  
> 

Yes, in a future (not very far) yes

> Third: Cf. http://msdn.microsoft.com/en-us/library/ms714102(VS.85).aspx
> 

Currently I'm trying to implement SQL_C_WCHAR, SQL_W(VAR/LONGVAR)CHAR
are quite ok.

> To support Unicode, do we provide new functions, or new values (e.g.
> SQL_WCHAR) to existing functions?  (I suppose no new functions.)  
> 

:) Unfortunately no. Unicode driver means that all ANSI version have a
corresponding unicode version (like SQLExecDirectW for SQLExecDirect).
The change is interpret characters as wide instead of ANSI.

> Fourth: Can you show me the API function call sequence that mixes
> encodings for a single column?  I assume the server sends all values for a
> column using a single encoding; I assume the client may somehow ask for
> different encodings as the data are fetched.  
> 

I never tried but I think reading TEXT/NTEXT with SQLGetData changing
SQL_C_CHAR/SQL_C_WCHAR for the same column.

> > Honestly I started coding SQL_C_WCHAR and *W stuff (I have a patch but
> > mostly a proof of concept) using UTF-8 but I don't like too many
> > encoding.
> 
> Maybe it would be easier to create a BRANCH_UNICODE branch in CVS where
> you could work and we could refer to the source code.  
> 

Mmm.. mainly branches are for multiple developing... perhaps I would
prefer a "from XXX date code is quite broken so if you want to use CVS
use cvs co -D XXX". Having two branches is like saying this branch is
"quite stable" so use 0.82 for stable or 0.83.dev.XXX. I see no point,
all feature of current 0.83 are in updated 0.82. I also see no big
developing for a branch. The only advantage I see is to decide to
provide a Unicode version and a no unicode version of FreeTDS.

To be sure we could branch and continue working on CVS HEAD, having a
BRANCH_NOUNICODE. Branch and merging is heavy on CVS, this way main
changes will go to the no-branch.

> > If column_size is
> > client size then column_data should be the client data! So instead of
> > allocating and copy data I just put a curcol->column_data = src in the
> > right place and it works! Simple, one single conversion from client to
> > packet buffer without many copies... and with less allocations too! 
> 
> That sounds right.  
> 
> > (The
> > extension is why don't let libTDS do the conversion, any conversion,
> > using column_type and on_server.column_type... but don't consider this
> > now).
> 
> That was always my design: keep track of both encodings, and have libtds
> convert nearest the wire.  All internal storage uses the client's
> encoding.  Server encoding is used only to communicate with the server.  
> 

here the extension is a bit "bigger". Something like having column_size
== SYBVARCHAR and on_server.column_size == SYBINT... that is having
libTDS do any conversions. And a possible extension to this is to use
client library column_type and have two callbacks to translate from/to
libTDS... but is better to concentrate in ODBC Unicode...

The problem with "convert nearest the wire" is that at the moment libTDS
fetch data and fill row buffer it doesn't know which encode the client
wants (SQL_C_CHAR or SQL_C_WCHAR). The only exception is if you use
SQLBindCol but there is no way ODBC can tell libTDS "I have a bind so
translate to this encoding".

All that stuff suggest me "direct binding"... well, mostly OT any ideas
on how to implement all new datatypes of mssql2005 and mssql2008 ?? One
problem with mssql2005 is the VARCHAR(MAX) which can be very large,
something like SYBLONGCHAR/SYBBLOB of Sybase (still not that supported).
I'm currently filling a table of types with flags (like ascii, unicode,
fixed and so on) and other informations (varint ms/sybase), the idea is
having a perl script that generate a table flags and possibly some
functions, just for change all these related stuff in a single place.

> > The real problem raise from columns.... Columns
> > are read in a buffer with a given (client) encoding. The row buffer is
> > always encoded with a single encoding. This is correct for dblib and
> > ctlib but ODBC have to mix 2 encodings. 
> 
> OK.  IIUC we have TDSCOLUMN::char_conv, but different rows may be
> requested using different client encodings.  So, libtds fetches the data
> from the wire, converts using char_conv, but then e.g. SQLGetData may use
> a TargetType of SQL_C_CHAR for one row, and SQL_C_WCHAR for another.  
> 
> > So after all that thoughts I think that using server encodings on rows
> > would help to avoid two conversions. 
> 
> So, your solution is to not convert in libtds.  Keep the server encoding,
> whatever it is, for every column on every fetch.  Delay conversion until
> SQLGetData or until writing the row buffer to the bound buffer.  
> 
> To avoid conversion in libtds, you propose a flag TDSSOCKET::no_data_conv.
>  
> 
> I don't like negative flags.  What do you think of TDSSOCKET::use_iconv
> instead?  
>       if (tds->use_iconv) {
>               tds_iconv(...);
>       }
> 
> I suppose that is OK.  I don't have any better suggestion.  
> 

Renamed

freddy77