Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] indexOf O(1) patch?

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Elliotte Harold <elharo AT metalab.unc.edu>
  • To: Wolfgang Hoschek <whoschek AT yahoo.com>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] indexOf O(1) patch?
  • Date: Tue, 01 Feb 2005 15:43:50 -0500

Wolfgang Hoschek wrote:


6% difference in memory is minimal. XOM is already
quite good at memory consumption: 162883k / 25 MB =
6.5 memory bytes per file byte.

Ouch. That's worse than I thought. I was thinking XOM was in the 3-4 bytes per file byte range, maybe 4.5 at the outside.


Having said that, there are plenty of unexplored
opportunities in XOM to reduce memory *without*
compromisising performance.
ArrayLists could be replaced with arrays, saving some
4 + 4 bytes per Element and per Attribute list.

That's on the TODO list.

Additional namespace declarations consume bizarre
amounts of memory which is a problem for artificially
generated XML.

I don't know that these come up so often, though. Are there really cases where every element is going to have these? Typically, it's just the elements at the root that pick these up.

> Qnames with prefixes are not interned
hence the substrings consume lots of memory,

Good catch. I hadn't thought of that, and it's easy to fix.

> the
baseURI of each Node is almost always the same so
those 4 bytes could be reduced,

This might need more elaboration. What would change? I don't see how we can support different base URIs on different elements without carrying around this field.

the type of an
attribute could be reduced from 4 bytes to 1 byte,

I'm not sure about this one. We could make it a one-byte type, sure, but would the VM actually store that value in a byte, or would it use 4 bytes anyway?

and
there are probably more potential improvements with
unknown quantified effects that could be done, but
havent't been done.

If you find any more, holler.

But more importantly, always keep in mind that working
with very large XML document trees is always
expensive, produces huge *tenured* heaps which puts
huge pressure on the VM allocator and collector, and
most importantly it is inherently fragile: Application
data size and disk storage capacity grow at a *much*
higher rate (more than Moore's law) than main memory
size, and infinitely faster than any minor XOM code
memory optimization tweaking. Thus, if today an
application's files can still just fit into memory,
they most likely won't fit anymore 6 months down the
road, and the app will break. There are hard limits
with main memory trees, and no amount of tweaking will
make them go away in any significant manner.

It seems to me that we can move the bar of where the main memory size becomes an issue though. In particular, anything that's added to each and every node really does make a noticeable difference in memory size. The more fat I can trim from XOM, the more documents it will be able to process.

--
Elliotte Rusty Harold elharo AT metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim




Archive powered by MHonArc 2.6.24.

Top of Page