Skip to Content.
Sympa Menu

freetds - Re: [freetds] no conversion changes...

freetds AT lists.ibiblio.org

Subject: FreeTDS Development Group

List archive

Chronological Thread  
  • From: "ZIGLIO, Frediano, VF-IT" <Frediano.Ziglio AT vodafone.com>
  • To: "FreeTDS Development Group" <freetds AT lists.ibiblio.org>
  • Subject: Re: [freetds] no conversion changes...
  • Date: Fri, 19 Sep 2008 14:53:09 +0200

> Frediano Ziglio wrote:
> > Il giorno sab, 13/09/2008 alle 16.51 -0400, James K. Lowden
> ha scritto:
> > > First: What do we mean by "wide"? UCS-2? UTF-16?
> UCS-4? UTF-8? Is
> > > it compile-time dependent? Run-time?
> >
> > It depends from DM... unixODBC use UCS-2 while iODBC use wchar_t
> > (usually UCS-4 on Unix.. or not?? I'm not sure, certainly Linux use
> > UCS-4). Using unixODBC there is a configuration flags to
> use wchar_t but
> > usually is not enabled
>
> OK, so it's a compile-time setting.
>
> > > I see a
> > > reference to a "ODBC 3.5 (or later) Unicode Driver". Is that what
> > > we're building?
> >
> > Yes, in a future (not very far) yes
>
> OK.
>
> > Unicode driver means that all ANSI version have a
> > corresponding unicode version (like SQLExecDirectW for
> SQLExecDirect).
> > The change is interpret characters as wide instead of ANSI.
>
> I see. So a Unicode appliction calls SQLExecDirectW, or does a
> preprocessor macro convert it, something like:
>
> #if UNICODE
> #define SQLExecDirect SQLExecDirectW
> #endif
>
> ??
>

Yes

> > > Fourth: Can you show me the API function call sequence that mixes
> > > encodings for a single column?
> >
> > I never tried but I think reading TEXT/NTEXT with
> SQLGetData changing
> > SQL_C_CHAR/SQL_C_WCHAR for the same column.
>
> The app can call SQLBindCol in between calls to SQLFetch, too:
>
> "A column can be bound, unbound, or rebound at any
> time, even after data
> has been fetched from the result set. The new binding takes effect the
> next time that a function that uses bindings is called."
> (http://msdn.microsoft.com/en-us/library/ms711010(VS.85).aspx)
>

Yes, it's correct... however using SQLGetData you don't have a bind active.

> > > Maybe it would be easier to create a BRANCH_UNICODE branch in CVS
> >
> > I would
> > prefer a "from XXX date code is quite broken so if you want
> to use CVS
> > use cvs co -D XXX".
>
> OK. Maybe just a CVS tag UNICODE_BASE or something like
> that, so we don't
> have to remember the date.
>
> As long as db-lib and ct-lib remain useable in CVS HEAD, I'm happy. I
> intend to change tds_connect() for example, so broken code
> would be bad.
>

I did the branch however currently changes to code are very limited and still
"#if
ENABLE_DEVELOPMENT" marked.

> > To be sure we could branch and continue working on CVS
> HEAD, having a
> > BRANCH_NOUNICODE. Branch and merging is heavy on CVS, this way main
> > changes will go to the no-branch.
>
> We can do that later if HEAD becomes too unstable to use.
>
> > The problem with "convert nearest the wire" is that at the
> moment libTDS
> > fetch data and fill row buffer it doesn't know which encode
> the client
> > wants (SQL_C_CHAR or SQL_C_WCHAR).
>
> Understood.
>
> > All that stuff suggest me "direct binding"...
>
> Right. Convert at the ODBC<->libtds boundary.
>

Yes, from TODOs I can see:
- Upper layer should be able to read data from network and handle rows
- Possibility to stop sending/receiving data for SQLPutData and similar
- limit copy of data (now wire -> tds -> temp -> client)
use a method like zero-copy
In token.c getting row should write data even to client
(callback, new user types, see ctlib conversion functions)
In query.c writing data do the same (use additional column fields?)

I think I'll start implementing some of them... I think that we should have
an option
where libTDS could stop before columns/parameters and return data as streams.
Something
like tds_read_data(tds, buffer, size). Perhaps it would be better to define
even a
structure like TDSSTREAM with a read "method". So even tdsconvert could use
this
structure. Well.. this would give to libTDS the possibility to return data,
even
chunked/blob without allocating any data. You should ask me why it helps with
ODBC...
well... it helps with SQLGetData :). I discovered that SQLGetData can return
*StrLen_or_IndPtr as SQL_NO_TOTAL. This help me in situations where you have
to convert
for instance big5->ucs2 or similar. There are issues as some client call
SQLGetData with
BufferLength == 0 to determine the size to allocate but this is not my
problem :). Another
note for blob1 test... I changed it to align behavior to MS one but I think
that MS
behavior is not correct :(. From documentation: "If the length of character
data
(including the null-termination character) exceeds BufferLength, SQLGetData
truncates the
data to BufferLength less the length of a null-termination character. It then
null-terminates the data.", actually MS can truncate data using less bytes
due to
conversion binary -> char, for instance passing 0x010203 and a BufferLength
of 4
(SQL_C_CHAR) documentation state it should return "012" (terminated) but MS
actually
returns "01" (terminated). So test was correct... I'll rollback change if
nobody disagree.

> > how to implement all new datatypes of mssql2005 and mssql2008 ?? One
> > problem with mssql2005 is the VARCHAR(MAX) which can be very large,
> > something like SYBLONGCHAR/SYBBLOB of Sybase (still not
> that supported).
>
> I haven't thought about this very much. I know that for
> zero-copy, the
> client library must provide the target buffer into which
> libtds places the
> data.
>

IMO there are two way to implement it:
- libTDS store in buffer provided by client
- client read data from libTDS directly (like using streams above)

Actually streams are not the easy cause client must be able to
- read size if available. Think BLOBs, mainly they have a size that is passed
before all
data, but chunked versions (like VARCHAR(MAX) which I would like to implement
or SYBBLOB
Sybase type) can have no initial size
- discard pending data (for truncation and similar)

> > I'm currently filling a table of types with flags (like
> ascii, unicode,
> > fixed and so on) and other informations (varint ms/sybase)
>
> Excellent. You might want "cardinal" type, too.
>

Yes, I think this is the missing information. The table is now at
misc/types.txt. There is
a script at misc/types.sh that convert it to CSV and a script at
src/tds/types.pl that
convert this table to src/tds/types.h which replace some old functions.

I didn't actually a "cardinal type" cause is actually very libTDS bounded.
Mainly it's
just a fix of conversions to single-bytes and limiting types.

Well... but what's SYBNVARCHAR type?? Where does this type came?? Is not
documented in
Sybase so I thik is MS but why having a SYBNVARCHAR if XSYBNVARCHAR is
available?? From my
memory they introduced nvarchar and types with extended (>255) size at the
same time...

bye
freddy77

Attachment: smime.p7s
Description: S/MIME cryptographic signature




Archive powered by MHonArc 2.6.24.

Top of Page