Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] XOM and Large Files

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Mike Colbert <mbcolbert AT yahoo.com>
  • To: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] XOM and Large Files
  • Date: Wed, 26 Jul 2006 21:07:26 -0700 (PDT)

--- Tatu Saloranta <cowtowncoder AT yahoo.com> wrote:

> --- John Cowan <cowan AT ccil.org> wrote:
>
> > Tatu Saloranta scripsit:
> >
> > > Reason is that most (all?) parsers I
> > > know of buffer their input, so adding
> > > BufferedInputStream just adds one unnecessary
> > memory
> > > copy operation to the chain.
> >
> > Well, apparently not. Whichever parser Mike is
> > using obviously is not
> > buffering the InputStream.
>
> Maybe. Or maybe not. ;-)

I'm afraid I poisoned this entire thread early on by misspeaking on a couple
of
salient points.

To summarize and clarify:

Initially, I was using an InputStreamReader on a FileInputStream to read the
contents of the file into a StringBuffer. This was so I could write in a root
element start and end tag, since my files do not have root elements. The file
content was then passed as a String to a StringReader, then to the Builder.
In
retrospect, a sloppy and grossly inefficient approach.

I then tried wrapping the Readers in BufferedReaders. This alone gave me huge
gains in speed. I'm sure it was the InputStreamReader on the FileInputStream
being buffered that made all of the difference.

I then scrapped the whole idea of reading the file content into a String in
favor of passing a SequencedInputStream on a FileInputStream (with root
element
start and end tags sequenced in) directly to the Builder. Again, huge
performance gains. And no explicit buffering.

And that's where I am now. I can process a 530Mb file in about 2 mins with
memory utilization maxed out at 3.6Gb (per process).

So, the bottom line is that passing an InputStream directly to the Builder is
the way to go. All the buffering seems to be taken care of that way.

Next, I'm going to try Wolfgang's suggestion of using a streaming NodeFactory
to enhance it further to be able to process even larger files and hopefully
get even more throughput.

Mike









Archive powered by MHonArc 2.6.24.

Top of Page