FreeTDS Development Group

["Pickett, David" <David.Pickett AT phlx.com>]

Memory, flat file, or temp file, and no good way to choose!  Well, you could
handle it two ways: either if you like silently correct behavior, make it a
command line arg., or simply check initially to see if input is flat.

If flat, you might even mmap it into your (maybe 64 bit mmap64 if you really
fear gigabyte fields) memory space to run faster and use less code with no
seeks, reads or buffer swap space, albeit with more paging activity as less
active pages are flushed from RAM.

If it is not a flat file, seeks and mmap are both out, so you are going to
need space somewhere in a file system: swap, /tmp, /var/tmp or the like.  It
seems silly to have to check available space in each arena to pick one and
have code for all three or more.  You could specify workspace by an argument
or environmental variable, like sort.  However, virtual memory is rarely
scarce any more, so if not a flat file, realloc buffer seems like a good
choice that will only get better.

You only get messed up if they have both a pipe and huge fields, and then
there is nowhere to hide on the client side!

Best regards,

David

-----Original Message-----
From: James K. Lowden [mailto:jklowden AT freetds.org]
Sent: Tuesday, December 06, 2005 8:38 PM
To: FreeTDS Development Group
Subject: Re: [freetds] freebcp file limit


Robert Klemme wrote:
> I know that it's open source and I could look at the sources, but why
> is there a need for seek/tell at all?  I mean, this should be a
> straight forward read from start to beginning task.  It's also usually
> better performance wise....

Well, I'm glad you know it's open source....  ;-)

The reason isn't a very good one, and it's my fault.  I haven't been
terribly happy with the answer, but I haven't bothered to fix it because
there are bigger fish to fry.  Specifically, I'd like freebcp to convert
files from one format to another (input and output both files or even
pipes).  I'd also like to address character set issues: the client's
character set does not necessarily reflect the encoding used in the bcp
data file.  When I tackle them, I'll fix the seeking.  

The reason isn't totally stupid, though.  The program looks ahead to the
next delimiter,then backs up and converts the text to the server's
encoding, feeding the iconv(3) output straight into the stream.  The first
byte sent to the stream, as determined by the protocol, is the length of
the ensuing field.   By reading ahead and seeking back, we avoid
allocating memory to hold everything.  

Previous versions had a fixed limit.  If your field was wider than that,
freebcp just didn't work.  Allocating memory is also a really bad idea,
because the field can be a text field, potentially measured in gigabytes. 


My next solution is to park each byte as read in a temporary file, then
rewind on that.  It's nominally inefficient, but for most fields there
won't be any I/O on the temporary file, because standard I/O uses 8 KB
buffers.   

--jkl
_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/freetds