[XOM-interest] Parsing large element content in streaming mode

Arshad Noor arshad.noor at strongauth.com
Mon Nov 22 17:52:14 EST 2010


I appreciate the prompt response, Elliotte.

While I should think about this in a little more detail, off the top of
my head, I think what will help is a subclass of Node called ByteStream
(or TextStream) with the following properties:

1) The class will be used to read/write content that are too large to
    fit into main memory;

2) The class will set/get values using byte-arrays.  (Since encryption
    only deals with byte-arrays, text must always be converted to bytes.
    However, I can see the benefit of dealing with String-streams too
    for non-crypto software);

3) The class will allow the programmer to set parameters like the size
    of buffer to return when getting the Value-content, the estimated
    amount of data remaining within the Node's value-content (based on
    the original file-size and what has been read so far), etc.

    Example: int getValue(byte[] buffer, int offset, int length);
	    int getNextValue(byte[] buffer, int offset, int length);
	    byte[] getValue(int length);
	    byte[] getNextValue(int length);
	    int getRemainder();
	    void setBufferSize(int length);

     Programmer provides the "byte[] buf" in the first two calls; XOM
     creates and returns it in the next two; offset is where the parser
     starts storing (length) data in the buffer.
	     	
4) In the NodeFactory, a makeByteStream() - or makeTextStream() method
    (or both) will allow software to use that method repeatedly to
    process the voluminous data.  Or, maybe a single makeStream() call
    must, in turn, make repeated getValue() (calls as shown in #3) until
    the element's content is completely processed.

I'm not sure this fits into your design requirements; if it does, I'm
more than happy to help test the new code to validate that it works (our 
test-case is a 5GB file).  If this is too much to do in the short
term, here's what will help at a minimum:

- The ability to read the underlying stream, at the start of the
   element's content, in a repeated loop till the end-tag.  The only
   setting I would like to specify in such a method is the number of
   bytes to return in each iteration.  with that, I should be able to
   do what I need.

Thank you.

Arshad Noor
StrongAuth, Inc.

On 11/22/2010 02:18 AM, Elliotte Rusty Harold wrote:
> Doesn't sound like you're missing anything. I suspect your use case is
> just extreme enough that you maybe hitting some genuine limitations.
>
> XML encryption is something I've thought about exploring for XOM. Can
> you describe ina little more detail what methods you'd need to make
> this work? I.e. if there's one method or two extra methods on Node or
> Element or NodeFactory or some such that would make your job easier
> then maybe I can add them. Though if you really need the possibility
> to put gigabytes of text in one text node, that's going to require
> somewhat more major surgery. Perhaps not impossible but not trivial.
>



More information about the XOM-interest mailing list