FreeTDS Development Group

[Frediano Ziglio <freddyz77 AT tin.it>]

Il mer, 2004-02-04 alle 22:05, Lowden, James K ha scritto:
> > From: Thompson, Bill D (London) [mailto:bill_d_thompson AT ml.com]
> > Sent: February 4, 2004 11:11 AM
> > 
> > > Do you still want me to add src/tds/bcp.c?  
> > 
> > yes - at the least it could hold the wire handling code that 
> > currently is in
> > dblib/bcp.c - I'd prefer to call it src/tds/bulk.c though...
> 
> OK.  
> 
> > We already know that we may have problems with partial packets being
> > submitted to the wire, because we may find an error as we process each
> > column.
> > So, it would make sense for us to construct the "row" buffer 
> > beforehand (as
> > we do with TDS 5.0).
> 
> Short answer: I reread _bcp_read_hostfile().  It constructs a row buffer; 
> _bcp_exec_in() sends it.  
> 
> Long answer:
> 
> Whenever I think about this, I think about Freddy.  As a design, building a 
> row image and shipping it is attractive; I've said before that the file 
> uploading functions could be implemented terms of bcp_bind(), et. al.  But 
> Freddy comes to mind because he's always concerned with text and binary 
> datatypes, and copies thereof, and what happens when the varbinary column 
> reads in a 4 GB file?  You obviously can't build a row image in such a 
> case; you have to spoon-feed the data packet-by-packet to the server, until 
> EOF (in effect).  
> 
> I implemented a partial solution to this with tds_iconv_fread(), but that 
> function is called from _bcp_read_hostfile(), which allocates for it an 
> output buffer large enough to hold the whole column after passing the data 
> through iconv().  Obviously, that doesn't avoid the row-image-in-memory 
> issue.  
> 
> The best alternative I've come up with is to support bcp_moretext().  That 
> would require, ISTM, some kind of "bcp_bind_tmpfile()", something that 
> associates a chunk of a file with a column.  The row image need not be 
> precomposed in memory; rather, a "row image description" -- a list of 
> pointers/handles and lengths of post-iconv, post-dbconvert, server-ready 
> data -- would be enough.  
> 

I didn't understand this paragraphs that much... What do you mean by
"something that associates a chunk of a file with a column" ?

> I would also like to remove my recently added restriction on bcp inputs: 
> fseek().  We seek on the input data file to determine the column size.  
> That precludes using freebcp, say, as a filter on stdin.  Binding extra 
> large datatypes to a workfile would let us not seek on the input.  
> 
> bcp_exec() [in] can then be implemented as:
> 
>       for each column {
>               bcp_bind() or bcp_bind_tmpfile()
>       }
>       for each row {
>               for each column {
>                       for each chunk {
>                               fread() /* to iconv input */
>                               if (char type)
>                                       iconv() /* to temp buffer */
>                               if (varbinary etc.) {
>                                       fwrite() /* no dbconvert */

fwrite where ? to a temporary file ?

>                               }
>                       }
>                       dbconvert() /* to bound buffer */
>               }
>               bcp_sendrow()
>               for each such column {
>                       bcp_moretext()
>               }
>               if (! nrows % batchsize == 0)
>                       bcp_batch()
>       }
>       bcp_done()
> 
> The subtle part being that bcp_sendrow() and bcp_moretext() are really 
> front-ends for a _bcp_send_columns() that interleaves the columns correctly 
> and ships the packet(s).  
> 

The harder thing it's length. You need first to send length and then
data. To know length you need to scan all column if terminated or to
convert to proper charset if we need to convert it (text). If we know
length before we can just copy from file to wire. I don't think it's
good to use temporary files just to avoid fseek... perhaps it would be
better to test if we can fseek and then use temporary file.

> One nice outcome of this arrangement is that "for each chunk" becomes the 
> only part sensitive to the existence of a prefix or terminator.  We save a 
> few hundred lines of code right there.  
> 
> Tell me what you think of all this.  I'd like implement some of it.  
> 

freddy77