FreeTDS Development Group

[Robert Klemme <shortcutter AT googlemail.com>]

James,

thank you for taking the time to explain this!  I really appreciate it.

2005/12/7, James K. Lowden <jklowden AT freetds.org>:
> Robert Klemme wrote:
> > I know that it's open source and I could look at the sources, but why
> > is there a need for seek/tell at all?  I mean, this should be a
> > straight forward read from start to beginning task.  It's also usually
> > better performance wise....
>
> Well, I'm glad you know it's open source....  ;-)

:-)

> The reason isn't a very good one, and it's my fault.  I haven't been
> terribly happy with the answer, but I haven't bothered to fix it because
> there are bigger fish to fry.  Specifically, I'd like freebcp to convert
> files from one format to another (input and output both files or even
> pipes).

I.e. you want to make it operate in streaming mode?

>  I'd also like to address character set issues: the client's
> character set does not necessarily reflect the encoding used in the bcp
> data file.  When I tackle them, I'll fix the seeking.

Yeah, sounds like a natural consequence of making it streaming compatible.

> The reason isn't totally stupid, though.  The program looks ahead to the
> next delimiter,then backs up and converts the text to the server's
> encoding, feeding the iconv(3) output straight into the stream.  The first
> byte sent to the stream, as determined by the protocol, is the length of
> the ensuing field.   By reading ahead and seeking back, we avoid
> allocating memory to hold everything.

Ah, I see.

> Previous versions had a fixed limit.  If your field was wider than that,
> freebcp just didn't work.  Allocating memory is also a really bad idea,
> because the field can be a text field, potentially measured in gigabytes.
>
>
> My next solution is to park each byte as read in a temporary file, then
> rewind on that.  It's nominally inefficient, but for most fields there
> won't be any I/O on the temporary file, because standard I/O uses 8 KB
> buffers.

Well, you might not experience as many physical writes as logical
writes but IMHO there will be write operations nevertheless.  I'd
suggest a more sophisticated approach: put a stream on top of the OS
stream (aka file handle) that does internal buffering and resorts to
using a tmp file if too much space is needed.  That way freebcp will
get away without any additional OS IO operations most of the time.

I've looked at the sources but didn't find an easy way to integrate
something like that.  That's probably because my C is a bit rusty and
also I couldn't spent much time yet.  I can offer to provide an
implementation of such a buffered stream (or try to find one and
integrate it) if you provide me with the necessary interfaces and
probably a bit more info.  If you think it's feasible / worth the
effort just email me off list and we can discuss this a bit further.

Kind regards

robert