[freetds] Problems with large image transfers.

Lowden, James K LowdenJK at bernstein.com
Wed Sep 22 10:12:33 EDT 2004


-----Original Message-----
From: Thompson, Bill D (London)
Sent: Wednesday, September 22, 2004 8:18 AM

freebcp is kind of my baby, so I'll also take a look at the issues.

There's a variety of issues with the program when it comes to BLOB data
- James has found some, and I can see several more.
-----End Original Message-----

Hiya Bill, 

It occurred to me last night, thinking this over, that we need
tds_iconv_fwrite(), analogous to tds_iconv_fread(), something that reads
from the TDS stream and writes chunks of converted data directly to the
file.  The issue of course being that we can't allocate an iconv(3)
output buffer for an entire 2GB image column.  

But even that's insufficient.  tds_iconv_fread() requires the caller to
allocate a buffer large enough to hold the converted data.  In the case
of reading an image column from the datafile, we still need enough
memory to hold the entire (converted) image.  

These potentially giant datatypes will force us to abandon our "one
column, one buffer" design.  For sending, we'll have to assemble the
row/column image on disk, then send it in chunks.  For receiving, we can
read a chunk, iconv it, dbconvert it, and write it.  As I sit here, it
seems to me that it's easiest to write the iconv output to a temporary
file, then back up and run dbconvert() over chunks of it.  What do you
think?  

Jeezopetes, what a pain!  Whoever decided putting PDFs in a database was
good idea in the first place, anyway?  

--jkl

-----------------------------------------
The information contained in this transmission may contain privileged and confidential information and is intended only for the use of the person(s) named above. If you are not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender immediately by reply e-mail and destroy all copies of the original message. Please note that we do not accept account orders and/or instructions by e-mail, and therefore will not be responsible for carrying out such orders and/or instructions.
If you, as the intended recipient of this message, the purpose of which is to inform and update our clients, prospects and consultants of developments relating to our services and products, would not like to receive further e-mail correspondence from the sender, please "reply" to the sender indicating your wishes.  In the U.S.: 1345 Avenue of the Americas, New York, NY 10105.





More information about the FreeTDS mailing list