FreeTDS Development Group

["Thompson, Bill D (London)" <bill_d_thompson AT ml.com>]

Hey James,

A stupid question I know - I've never really got to grips with the iconv
issues...
but why do we need to invoke iconv for BLOB data ?

I don't know if this goes with your idea or not, but I was also
thinking...

Really, BLOB data and conventional BCP host files don't mix all that
well.

If you want to import large binary data into a database column, I can't
see how you are ever going to get that data into a host file that
(free)bcp can interpret. 

In order for freebcp to understand the total length of the binary data,
you are going to have to either:

1) specify a length prefix or
2) specify a "terminating" sequence - "\t" or whatever

You've already pointed out the problems with the latter approach, but in
either case I can't see how you are going to create a file that
conforms. 
You will probably have a single file that contains your image or PDF or
whatever. How can you prepend a length prefix or a append a terminator
onto that data in the file ? 

Exporting would have the same problems, wouldn't it ? would the
resulting file be in any way useful if it had some sort of data either
prepended or appended to the binary data enclosed ? I find it difficult
to see how.

Me and google hung around the schoolyard for a while, and we discovered
that MS have a utility called BII (bulk image Import).
This differs from bcp in that it will recognise a *filename* as the data
in the hostfile, and will then read the data from the named file. This
seems to be an idea that has some power. I'd prefer perhaps to say that
freebcp only supports the import of BLOB data along these lines.  

I guess we then have to decide how we "send the data in chunks". 
Hmmm. The last upgrades to freebcp did away with this idea, for reasons
that we discussed previously.
I'd hate to lose what we gained in stability from that idea, just so
that we can cater for this (pretty specialised) requirement.

just some thoughts, I'll carry on thinking...

Bill

 

-----Original Message-----
From: freetds-bounces AT lists.ibiblio.org
[mailto:freetds-bounces AT lists.ibiblio.org] On Behalf Of Lowden, James K
Sent: 22 September 2004 15:13
To: FreeTDS Development Group
Subject: RE: [freetds] Problems with large image transfers.


-----Original Message-----
From: Thompson, Bill D (London)
Sent: Wednesday, September 22, 2004 8:18 AM

freebcp is kind of my baby, so I'll also take a look at the issues.

There's a variety of issues with the program when it comes to BLOB data
- James has found some, and I can see several more.
-----End Original Message-----

Hiya Bill, 

It occurred to me last night, thinking this over, that we need
tds_iconv_fwrite(), analogous to tds_iconv_fread(), something that reads
from the TDS stream and writes chunks of converted data directly to the
file.  The issue of course being that we can't allocate an iconv(3)
output buffer for an entire 2GB image column.  

But even that's insufficient.  tds_iconv_fread() requires the caller to
allocate a buffer large enough to hold the converted data.  In the case
of reading an image column from the datafile, we still need enough
memory to hold the entire (converted) image.  

These potentially giant datatypes will force us to abandon our "one
column, one buffer" design.  For sending, we'll have to assemble the
row/column image on disk, then send it in chunks.  For receiving, we can
read a chunk, iconv it, dbconvert it, and write it.  As I sit here, it
seems to me that it's easiest to write the iconv output to a temporary
file, then back up and run dbconvert() over chunks of it.  What do you
think?  

Jeezopetes, what a pain!  Whoever decided putting PDFs in a database was
good idea in the first place, anyway?  

--jkl

-----------------------------------------
The information contained in this transmission may contain privileged
and confidential information and is intended only for the use of the
person(s) named above. If you are not the intended recipient, or an
employee or agent responsible for delivering this message to the
intended recipient, any review, dissemination, distribution or
duplication of this communication is strictly prohibited. If you are not
the intended recipient, please contact the sender immediately by reply
e-mail and destroy all copies of the original message. Please note that
we do not accept account orders and/or instructions by e-mail, and
therefore will not be responsible for carrying out such orders and/or
instructions.
If you, as the intended recipient of this message, the purpose of which
is to inform and update our clients, prospects and consultants of
developments relating to our services and products, would not like to
receive further e-mail correspondence from the sender, please "reply" to
the sender indicating your wishes.  In the U.S.: 1345 Avenue of the
Americas, New York, NY 10105.


_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/freetds 
--------------------------------------------------------
 
If you are not an intended recipient of this e-mail, please notify the 
sender, delete it and do not read, act upon, print, disclose, copy, retain or 
redistribute it. Click here for important additional terms relating to this 
e-mail.     http://www.ml.com/email_terms/ 
--------------------------------------------------------