FreeTDS Development Group

["James K. Lowden" <jklowden AT schemamania.org>]

Looking forward to moving bcp into libtds, and grappling through the code
again tonight, it occurs to me our next generation bcp code might benefit
from some redesign, and some new abstractions.  

Fundamentally, bcp is about moving data from one stream to another with
the least wasted motion.  The native format hacking we've been up to
lately has made that especially clear: I usually think about bcp as a
row-by-row conversion operation, but that's because I rely exclusively on
delimited files (that are always in character form).  A native bcp
transfer involves hardly more than reading and writing buffers.  

That leads me to this picture:

data ->  stream
        ---------
        transform
        ---------
         stream   -> data

It really shouldn't matter which stream is attached to a file, and which
to a server.  I mean, a handle's a handle, right?  Apart from the gritty
details of socket handling, the bcp library should be content copying
file->socket, socket->file, file->file, or socket->socket.  Shouldn't
really matter, should it?  

The "transform" has many components: character/binary format, dbconvert()
conversions, column order mapping with format files, not to mention
iconv().  I can think of many uses of a freebcp-like utility that could:

1.  decode a native file, writing a delimited one, or
2.  reorder the columns of a file, or
3.  convert the character encoding of a file

Substitute "server" for "file" anywhere in the above list.  Mix and match.
 

I think all that's needed is a really thin layer masking the difference
between sockets and disk files.  Does that sound feasible or infeasible to
you?  

--jkl