FreeTDS Development Group

["James K. Lowden" <jklowden AT schemamania.org>]

I worked out tonight what's wrong with my bcp uploads.  The symptom was:

Msg 4810, Level 17, State 1
Server 'CPRO200', Line 1
        Expected the TEXT token in data stream for bulk copy of text or
image data.

The problem was: I was lying about the size of the data.  My TDS_ROW token
(0xD1) was promising to send as many bytes as were read from the data
file, not the number output by iconv.  In case you don't seize immediately
on the details: when sending a row of data, each column is prefixed by the
data's (not the column's) size.  

Heretofore, bcp.c simply read the file and sent it, no conversion, no
nothing.  Unless the file happened to have UCS-2 data, it was not possible
afaik to load nchar columns.  Well, not usefully.  This simple processing
made it fairly easy to prefix the data with its size: just keep track of
the number of bytes read or, sometimes, output by tds_convert.  

Now, the datafile will be encoded according to the client's character set.
 bcp.c will read the datafile, convert it to the column's datatype and
then (in the case of character data) to the server's encoding.  

So far, tds_iconv has worked with relatively small chunks of data (256
bytes), to avoid allocating memory dynamically.  We haven't had a
situation where the caller needed to have the whole buffer converted at
once; it has always been sufficient to convert some, write some, and
repeat.  Now we have a situation where we can't do that.  

For most columns, this is no big deal: even nvarchar columns have a
maximum of 8000 bytes.  Text columns OTOH can be very big, and might
easily exceed available memory.  

What to do?  There are a few options:

1.  Use a chain of dynamically allocated pointers, similar to
_bcp_get_term_data().  
2.  Make tds_iconv available through tds_convert, and use that interface. 

3.  Write the iconv outputs to a temporary file until we reach the end. 
Then we'll know the length, and can read the file back in for shipment to
the server.  

I think #3 is the most robust, but slowest.  #2 would be convenient to use
but would be limited to 50% of available memory (because tds_convert()
wouldn't know how big a chunk to return until iconv was done, and iconv
would have to park its output somewhere).  

I guess I'm leaning toward #3.  I find myself wishing for a flavor of
fprintf(3) that would take an iconv conversion descriptor and write the
converted data to a stream.  

--jkl