[XOM-interest] Writing large XML documents

Peter Corbett ptc24 at cam.ac.uk
Tue Dec 18 04:25:19 EST 2007


Eric Wang wrote:
> On Dec 17, 2007 11:22 PM, Tatu Saloranta <cowtowncoder at yahoo.com> wrote:
> 
> 
>>That is bit misleading, in a way -- it's more accurate
>>to say that you can build tree representations for
>>sub-trees of xml content (and potentially also discard
>>earlier parts, trim while you go, if you want to get
>>fancy). Although underlying parsers are all streaming
>>(XOM is not a parser but uses underlying parsers like
>>Xerces), result is still a tree that has to completely
>>fit in memory.
> 
> 
> True, but I don't build a tree at all when processing large XML files.
> I use a custom NodeFactory subclass and do all my processing in
> finishMakingElement().
> 
I've had a lot of joy using XOM to read subtrees from a large XML file, 
basically the XML file was a long list (tens or hundreds of thousands) 
of moderately-sized elements (about a hundred to a thousand subnodes or 
so each, I'd guess), and you could read one into a XOM tree, process the 
tree in the normal XOMmy ways including running XPaths on it, discard it 
and move onto the next element, thus making the whole process fit nicely 
in memory when slurping in the entire tree would just have failed.

My instinct would be to subclass the Serializer; it contains lots of 
protected methods that look like they could be hooked up to the 
NodeFactory methods with a bit of coding. You'd need to do something 
slightly clever to get around the fact that they are protected, but that 
shouldn't be too tricky.

Peter Corbett



More information about the XOM-interest mailing list