FreeTDS Development Group

[Frediano Ziglio <freddy77 AT gmail.com>]

Il giorno gio, 11/09/2008 alle 12.01 -0400, James K. Lowden ha scritto:
> ZIGLIO, Frediano, VF-IT wrote:
> > 
> > Well... finally I decided to add a parameter in libTDS (still I don't
> > know where, probably TDSSOCKET) to have a sort of "no convert please"
> > for characters. This to support SQL_C_CHAR in ODBC. This flag will
> > disable characters conversions for data only (rows and parameters, not
> > column names and similar).
> 
> I don't think a flag is the right answer.  
> 
> We have a a simple auto-conversion model.  The client encoding is
> converted to the server encoding automatically as the data are written to
> the server.  
> 
> We could handle the whole thing by requiring the API library to use one
> consistent encoding.  If *everything* is UTF-16, then autoconvert becomes
> a no-op.  
> 
> The other way is to handle parameters the way we do columns: each has its
> own encoding.  
> 
> Let's keep the from-to model and not adopt a flag-based override.  
> 

Mmm... well.. I would prefer ucs4 but this is not the problem.

First: ODBC have 2 client encoding, one single byte and one "wide" and
can be mixed even on 2 columns (well.. even on a single column if we
want!).

Honestly I started coding SQL_C_WCHAR and *W stuff (I have a patch but
mostly a proof of concept) using UTF-8 but I don't like too many
encoding.

I was fixing Sebastien's issues till I wrote last utf8 test enhanced and
I thought that TDSCOLUMN->column_size is client size and not server
column size while TDSCOLUMN->on_server.column_size is the column_size
(*2 is unicode). I fixed this issue (now src/tds/query.c use
on_server.column_size) but got an enlightenment! If column_size is
client size then column_data should be the client data! So instead of
allocating and copy data I just put a curcol->column_data = src in the
right place and it works! Simple, one single conversion from client to
packet buffer without many copies... and with less allocations too! (The
extension is why don't let libTDS do the conversion, any conversion,
using column_type and on_server.column_type... but don't consider this
now).

The real problem raise from columns, not from parameters... yes even
parameters have and support char_conv (perhaps not filled for
single/multi-byte but supported) but the problem are columns. Columns
are read in a buffer with a given (client) encoding. The row buffer is
always encoded with a single encoding. This is correct for dblib and
ctlib but ODBC have to mix 2 encodings. Yes, one solution is to use a
wide encoding and convert back on ODBC layer... mainly two conversions
for most uses. So... a solution which does zero or one conversions for
input parameters and one or two (but mostly two) conversions for normal
rows. Well.. works but 0.82 would be more efficient and less space
bloated.

So after all that thoughts I think that using server encodings on rows
would help to avoid two conversions. 

Oh... well obviously the situation on ODBC is not so easy you have to
consider that you can pass a parameters in chunks (SQLPutData) and read
in chunks (SQLGetData).

You can think I'm moving conversions from libTDS to ODBC but as
explained ODBC layer would have to handle conversions too so instead of
adding two encoding support to libTDS I would choose to move conversions
to upper layer. ODBC is the only library that use two conversions at a
time.

Well... as usual nothing is set on stone :)

Frediano