FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: Jonathan Monroe [mailto:monroej_freetds AT actualtechnologies.com]
> Sent: February 23, 2004 11:47 AM
> > > On  Sun, 22 Feb 2004 at 10:03:02 +0100, Frediano Ziglio wrote:
> > > >>> On Feb 18, 2004, at 2:51 PM, Jonathan Monroe wrote:
> > > >>>
> > > >>>> I think I have found an issue with the way the SQL_C_BINARY
> > > >>> data type
> > > >>>> is handled in the ODBC code.
> >
> > Mm.... here the behavior is different. We convert 
> characters as we get
> > from wire so the behavior it's exactly the same of 
> varchar/char/text.
> 
> IMHO the purpose of doing a SQL_C_BINARY bind is to get access to the 
> raw bytes of the field - otherwise you would just bind using 
> SQL_C_CHAR.  I realize that much thought has preceded this 
> conversation 
> and MS compatibility isn't always a desired goal.  But to me, 
> SQL_C_BINARY implies raw bits, while SQL_C_CHAR implies (possibly 
> transformed) text. In this case, it seems that a SQL_C_BINARY bind 
> should return both bytes of each character in an nvarchar field.


Johnathan, 

First of all, welcome to the project and thanks for your efforts.  I'm glad 
to see Actual Technologies joining the fray.  You've raised a thorny issue 
here regarding a binary bind of UCS-2 data; let's see what can be done.

Our approach has been to convert all UCS-2 data (nchar, nvarchar, ntext, and 
metadata strings) into the client's encoding upon arrival, and to adjust the 
column buffer sizes accordingly.  The UCS-2 data are converted 
instantaneously, before they're available for fetching by the client library. 
 As far as the application can see, they're locally encoded (UTF-8, ISO 
8859-1, whatever), and that's that.  We don't support UCS-2 clients yet, so 
there's no possibility of seeing UCS-2 data.  

Now comes the requirement to *not* convert, to faithfully convey the bits as 
bits and let the, uh, chips fall where they may.   Problem being, of course, 
that we've long discarded the pre-converted data.  What to do?  

1.  We could keep the server's data around.  We don't need to discard them; 
it's just (usually) more efficient. 

2.  We could unconvert them.  We have the server's metadata.  It's not 
terribly efficient, but it's not terribly slow, either.  

3.  We could have a no-convert option, per server.  We need this for BCP.  
That would impose a significant burden on the client, however: it would have 
do *all* data encoding conversions.  Potentially, we could add 
convert-to-client-encoding as a binding option, and to the tds_convert() and 
its brethren.  

I think #1 is too wasteful of client buffer memory.  I can hear Frediano from 
here choking on the idea of keeping two encodings of a 500 MB text column 
around, just in case someone wants the pre-converted version.  Stand well 
clear of that one.  

Option #2 is probably the most feasible in the short run.  Additional binding 
options and the choice of defeating built-in conversions (#3) seem like the 
best architectural answers.  

> > Keep in mind that endianism change the order...
> 
> Yeah.  Again, it seems like I should be able to get the raw bytes as 
> they are stored in the database, but I'll concede that the correct 
> behavior is probably undefined.

Well, you really don't know how the bytes are stored in the database; at any 
rate afaik it's undocumented.  You really only know how they're represented 
on the wire.  Our endian policy, of necessity, is "library makes right".  
That is, FreeTDS presents data to the client in the client's native form.  If 
the server happens to deliver little endian data to a big endian client, we 
fix it.  Simple like that.  

> I'm more than happy to help with the work here. I've spent 
> considerable time crawling around in the code 

The ODBC driver is a big piece of work, and we always have room for capable 
hands.  It seems to me expanding tds_convert() to deal with convert-to-client 
and convert-to-server encodings would be an excellent place to start.  And, 
yes, unit tests are a good way to ensure that a given functionality 
starts/continues to work as expected.  :-)  

BTW, the best reference point for any patch is CVS HEAD.  It's sometimes 
feasible to apply them to the release, and sometimes we do, but our focus is 
always on the next release.  

--jkl

-----------------------------------------
The information contained in this transmission may contain privileged and 
confidential information and is intended only for the use of the person(s) 
named above. If you are not the intended recipient, or an employee or agent 
responsible for delivering this message to the intended recipient, any 
review, dissemination, distribution or duplication of this communication is 
strictly prohibited. If you are not the intended recipient, please contact 
the sender immediately by reply e-mail and destroy all copies of the original 
message. Please note that we do not accept account orders and/or instructions 
by e-mail, and therefore will not be responsible for carrying out such orders 
and/or instructions.
If you, as the intended recipient of this message, the purpose of which is to 
inform and update our clients, prospects and consultants of developments 
relating to our services and products, would not like to receive further 
e-mail correspondence from the sender, please "reply" to the sender 
indicating your wishes.  In the U.S.: 1345 Avenue of the Americas, New York, 
NY 10105.