FreeTDS Development Group

["Li, Maggie (IT)" <Maggie.Li AT morganstanley.com>]

> Also, what does "ident $(command -v freebcp)" tell you?  

$Id: tds.h,v 1.167 2003/12/31 11:33:10 freddy77 Exp $
     $Id: tdsver.h.in,v 1.4 2002/11/21 16:53:43 freddy77 Exp $
     $Id: tds_sysdep_public.h.in,v 1.7 2003/12/30 10:27:42 freddy77 Exp
$
     $Id: tds_sysdep_private.h,v 1.11 2003/12/29 16:08:03 freddy77 Exp $
     $Id: sybdb.h,v 1.54 2003/11/01 23:02:08 jklowden Exp $
     $Id: sybfront.h,v 1.2 2002/11/21 16:53:43 freddy77 Exp $
     $Id: freebcp.h,v 1.7 2003/03/19 17:05:13 jklowden Exp $
     $Id: freebcp.c,v 1.28 2003/12/09 10:19:16 freddy77 Exp $

I also tried to split the file into 5 million lines each and freebcp
them, and there is no segmentation fault.  But there is another problem.
It seems to be missing one row for the last splitted file.  I tried
splitted a much smaller file and freebcp them and nothing is missing,
seem to only happen with this large file.  

Thanks,
Maggie

-----Original Message-----
From: freetds-bounces AT lists.ibiblio.org
[mailto:freetds-bounces AT lists.ibiblio.org] On Behalf Of James K. Lowden
Sent: Monday, December 05, 2005 8:38 PM
To: FreeTDS Development Group
Subject: Re: [freetds] freebcp file limit

Li, Maggie (IT) wrote:
> I need to freebcp a large file (about 15 million lines, 4 gb in size),

> but after about 5 million lines, it gives segmentation fault.

Wow.  Thanks for the report.  

It would take quite a bit of effort for me to reproduce that.  My
testing facilities are across the Internet.  Even if they have that much
space, it would take hours just to transmit the data.  

I do have Sybase installed around here somewhere, on a machine that's
normally booted in something else.  If you could send me your DDL, I'll
work on it.  It would be very helpful, too, if you could send a
backtrace.
 

Also, what does "ident $(command -v freebcp)" tell you?  

>  Is there a size limit for freebcp?

No, there's not supposed to be.  The file is read in chunks and
transmitted to the server piecemeal.  You apparently have tickled a bug,
some bit of memory that isn't being freed.  

The good news is there may be a workaround.  As long as your data file
is in plain text (not a "native" bcp file), you can split it up.  You
could use sed(1) to do the job, or run something this, which takes
advantage of the first/last options of freebcp:

#! /bin/sh
BCP='freebcp db.own.table in file.dat -S server -c -U -P'
for FIRST in $(jot 16 0)
do
        LAST=$(( ${FIRST} + 1 ))
        printf  "${BCP} -b 100000 -F %8d -L %8d \n" \
                ${FIRST}000001 ${LAST}000000
done

Pipe the output to sh, as in 

        $ ./foo | sh -e -x

and go to lunch.  

HTH.

--jkl


--jkl
_______________________________________________
FreeTDS mailing list
FreeTDS AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/freetds
--------------------------------------------------------

NOTICE: If received in error, please destroy and notify sender.  Sender does 
not waive confidentiality or privilege, and use is prohibited.