Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] Parsing large element content in streaming mode

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Arshad Noor <arshad.noor AT strongauth.com>
  • To: XOM API for Processing XML with Java <xom-interest AT lists.ibiblio.org>
  • Cc: Elliotte Rusty Harold <elharo AT ibiblio.org>
  • Subject: Re: [XOM-interest] Parsing large element content in streaming mode
  • Date: Mon, 22 Nov 2010 14:52:14 -0800

I appreciate the prompt response, Elliotte.

While I should think about this in a little more detail, off the top of
my head, I think what will help is a subclass of Node called ByteStream
(or TextStream) with the following properties:

1) The class will be used to read/write content that are too large to
fit into main memory;

2) The class will set/get values using byte-arrays. (Since encryption
only deals with byte-arrays, text must always be converted to bytes.
However, I can see the benefit of dealing with String-streams too
for non-crypto software);

3) The class will allow the programmer to set parameters like the size
of buffer to return when getting the Value-content, the estimated
amount of data remaining within the Node's value-content (based on
the original file-size and what has been read so far), etc.

Example: int getValue(byte[] buffer, int offset, int length);
int getNextValue(byte[] buffer, int offset, int length);
byte[] getValue(int length);
byte[] getNextValue(int length);
int getRemainder();
void setBufferSize(int length);

Programmer provides the "byte[] buf" in the first two calls; XOM
creates and returns it in the next two; offset is where the parser
starts storing (length) data in the buffer.

4) In the NodeFactory, a makeByteStream() - or makeTextStream() method
(or both) will allow software to use that method repeatedly to
process the voluminous data. Or, maybe a single makeStream() call
must, in turn, make repeated getValue() (calls as shown in #3) until
the element's content is completely processed.

I'm not sure this fits into your design requirements; if it does, I'm
more than happy to help test the new code to validate that it works (our test-case is a 5GB file). If this is too much to do in the short
term, here's what will help at a minimum:

- The ability to read the underlying stream, at the start of the
element's content, in a repeated loop till the end-tag. The only
setting I would like to specify in such a method is the number of
bytes to return in each iteration. with that, I should be able to
do what I need.

Thank you.

Arshad Noor
StrongAuth, Inc.

On 11/22/2010 02:18 AM, Elliotte Rusty Harold wrote:
Doesn't sound like you're missing anything. I suspect your use case is
just extreme enough that you maybe hitting some genuine limitations.

XML encryption is something I've thought about exploring for XOM. Can
you describe ina little more detail what methods you'd need to make
this work? I.e. if there's one method or two extra methods on Node or
Element or NodeFactory or some such that would make your job easier
then maybe I can add them. Though if you really need the possibility
to put gigabytes of text in one text node, that's going to require
somewhat more major surgery. Perhaps not impossible but not trivial.





Archive powered by MHonArc 2.6.24.

Top of Page