FreeTDS Development Group

["James K. Lowden" <jklowden AT freetds.org>]

Robert Klemme wrote:
> I know that it's open source and I could look at the sources, but why
> is there a need for seek/tell at all?  I mean, this should be a
> straight forward read from start to beginning task.  It's also usually
> better performance wise....

Well, I'm glad you know it's open source....  ;-)

The reason isn't a very good one, and it's my fault.  I haven't been
terribly happy with the answer, but I haven't bothered to fix it because
there are bigger fish to fry.  Specifically, I'd like freebcp to convert
files from one format to another (input and output both files or even
pipes).  I'd also like to address character set issues: the client's
character set does not necessarily reflect the encoding used in the bcp
data file.  When I tackle them, I'll fix the seeking.  

The reason isn't totally stupid, though.  The program looks ahead to the
next delimiter,then backs up and converts the text to the server's
encoding, feeding the iconv(3) output straight into the stream.  The first
byte sent to the stream, as determined by the protocol, is the length of
the ensuing field.   By reading ahead and seeking back, we avoid
allocating memory to hold everything.  

Previous versions had a fixed limit.  If your field was wider than that,
freebcp just didn't work.  Allocating memory is also a really bad idea,
because the field can be a text field, potentially measured in gigabytes. 


My next solution is to park each byte as read in a temporary file, then
rewind on that.  It's nominally inefficient, but for most fields there
won't be any I/O on the temporary file, because standard I/O uses 8 KB
buffers.   

--jkl