Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] Procesing very large text node

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Christophe Marchand <christophe.marchand AT axyus.com>
  • To: XOM API for Processing XML with Java <xom-interest AT lists.ibiblio.org>
  • Subject: Re: [XOM-interest] Procesing very large text node
  • Date: Sun, 07 Feb 2010 19:46:23 +0100

As far as I know, for such huge files, only a SAX parser may solve your problem. I calculate MD5SUM on included gzip/base64 files of about 500Mb without any problem. Well, calculating a digital signature is another problem...

Christophe

Michael Ludwig wrote:
Marcin Waldowski schrieb am 07.02.2010 um 12:20:35 (+0100):

<Object Encoding="http://www.w3.org/2000/09/xmldsig#base64"; Id="1"
MimeType="text/plain">
U29tZSBmaWxlIGNvbnRlbnQ=
</Object>

The text node of this tag contains base64 representation of some file.
It can be very big file (2GB is a good example). I need to calculate
SHA1 of that file.

Is it possible to process that tag with XOM in some clever way without
using lots of memory and without issuing OutOfMemoryError?

I think you would want to take control of the document building process,
which in XOM you can do by subclassing nu.xom.NodeFactory.

However, as a usability improvement over SAX, the handler used for
dealing with text nodes, NodeFactory#makeText, always gets fed the whole
text contents in one string argument; so in order to save memory you
must prevent that handler from ever being called, otherwise the JVM will
have consumed that much memory - or issued an OutOfMemoryError.

I can't see any way to prevent makeText from being called from within
XOM. If I'm not mistaken, you need a pull parser in order to control the
parsing process so you can skip events, while the SAX parser which XOM
uses underneath is a push parser, where you're only at the receiving end
of the event queue.

I think you have to descend to the SAX level in order to get the degree
of control you want - org.xml.sax.ContentHandler#characters is called
for each chunk of characters, and that's were you could hook in your
digester.






Archive powered by MHonArc 2.6.24.

Top of Page