Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] XOM and Large Files

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <wolfgang.hoschek AT mac.com>
  • To: Mike Colbert <mbcolbert AT yahoo.com>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] XOM and Large Files
  • Date: Wed, 26 Jul 2006 10:32:32 -0700

Large heaps cost time because VM memory subsystems don't scale well with the size of the heap (linear or worse). Eventually, performance will be dominated purely by allocation and gc. Further, modern systems have deep memory hierarchies, with L1 and L2 caches being much faster than main memory.

If the problem at hand is amenable to streaming without too much complexity trouble, try using it :-)

Wolfgang.

On Jul 26, 2006, at 8:32 AM, Mike Colbert wrote:

A streaming NodeFactory enables efficient filtering and can avoid the
need to build a main memory tree, which is particularly useful for
large documents. The XOM Tutorial has a section on such streaming:
http://www.xom.nu/tutorial.xhtml#d0e1424

That's an option, but I wonder if this will really speed things up in my case.

It seems the bottlenecks are 1.) reading the entire file contents into a String
using a standard InputStream read. 2.) Parsing the String. Once the XOM
document is built, the traversal is very fast.

Using a NodeFactory will eliminate the traversal step, obviously, because I can
combine my processing with the parsing, but how will it actually speed up the
parsing itself? Or is it a matter of it being time-consuming to create the
in-memory tree itself, once it's parsed?

If it matters, all I am doing is taking a set of identifiers, finding
corresponding records in these large files, and reading them out to disk.


A NodeFactory is a user defined high level callback fed by the
Builder, conceptually similar to a SAX ContentHandler.

Yes, great feature of the XOM API. I've written a NodeFactory before and use
it all the time as part of a Visitor framework I wrote for XOM. Good stuff.

Mike








Archive powered by MHonArc 2.6.24.

Top of Page