FreeTDS Development Group

["Lowden, James K" <LowdenJK AT bernstein.com>]

> From: ZIGLIO Frediano [mailto:Frediano.Ziglio AT vodafoneomnitel.it]
> Sent: April 24, 2003 4:51 AM
> > 
> > 1.  Use a chain of dynamically allocated pointers, similar to
> > _bcp_get_term_data().  
> > 2.  Make tds_iconv available through tds_convert, and use 
> > that interface. 
> > 
> > 3.  Write the iconv outputs to a temporary file until we 
> > reach the end. 
> > Then we'll know the length, and can read the file back in for 
> > shipment to
> > the server.  
> 
> Mmm... you work with only client charset coding, so 
> tds_convert shouldn't handle charset conversions. 

Right.  

> Be optimistic: string is very short, allocate a small 
> fixed-size buffer (1024 bytes). No problem at all.

Slightly less optimistic but works in all cases except [n]text: one
8000-byte buffer.  

> The solution is to compute converted size, output it and then 
> start converting. This require only to be able to seek in 
> source bcp file (to compute converted size you need to 
> convert one time) after size computing. This require also two 
> conversion but few memory.

I hadn't thought of that.  You suggest using iconv once, discarding the
results, to obtain the output size.  Put that size to the stream, and call
iconv again, putting each output chunk to the stream.  Good idea!  

Better still, I think it will be unnecessary.  Your message caused me to
rethink the design.  I had thought that client->server conversion should
happen on the fly at the last minute, in transit to the wire, but I now see
the error of my ways.  

_bcp_read_hostfile() builds a row image by reading each field into memory,
no matter how large (well, I just noticed that it will fail if the data file
is undelimited and the field's data is > 1 MB).  Each field is passed
through tds_convert() to put the data in binary format in preparation for
writing to the wire.  It even calls tds_convert() to "convert"
char->varchar, which is a bit silly.  

_bcp_read_hostfile() is the place to call iconv().  There's no need to have
an in-memory image of character data in the client's encoding; it might as
well be converted to server encoding when it's read from the file.  

> I find myself wishing for a flavor of
> > fprintf(3) that would take an iconv conversion descriptor and 
> > write the converted data to a stream.  

> I don't follow you....

I mean that iconv(3) is analogous to sprintf(3): it processes one buffer and
writes to another.  What would be nice would be "ficonv()", whose output
went to a stream instead.  But it wouldn't solve our problem, because our
stream wants us to begin with the count of following bytes.  

Regards, 

--jkl



The information contained in this transmission may contain privileged and
confidential information and is intended only for the use of the person(s)
named above.  If you are not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient, any
review, dissemination, distribution or duplication of this communication is
strictly prohibited. If you are not the intended recipient, please contact
the sender immediately by reply e-mail and destroy all copies of the
original message. Please note that we do not accept account orders and/or
instructions by e-mail, and therefore will not be responsible for carrying
out such orders and/or instructions.