[XOM-interest] Parsing large element content in streaming mode
arshad.noor at strongauth.com
Mon Nov 22 17:52:14 EST 2010
I appreciate the prompt response, Elliotte.
While I should think about this in a little more detail, off the top of
my head, I think what will help is a subclass of Node called ByteStream
(or TextStream) with the following properties:
1) The class will be used to read/write content that are too large to
fit into main memory;
2) The class will set/get values using byte-arrays. (Since encryption
only deals with byte-arrays, text must always be converted to bytes.
However, I can see the benefit of dealing with String-streams too
for non-crypto software);
3) The class will allow the programmer to set parameters like the size
of buffer to return when getting the Value-content, the estimated
amount of data remaining within the Node's value-content (based on
the original file-size and what has been read so far), etc.
Example: int getValue(byte buffer, int offset, int length);
int getNextValue(byte buffer, int offset, int length);
byte getValue(int length);
byte getNextValue(int length);
void setBufferSize(int length);
Programmer provides the "byte buf" in the first two calls; XOM
creates and returns it in the next two; offset is where the parser
starts storing (length) data in the buffer.
4) In the NodeFactory, a makeByteStream() - or makeTextStream() method
(or both) will allow software to use that method repeatedly to
process the voluminous data. Or, maybe a single makeStream() call
must, in turn, make repeated getValue() (calls as shown in #3) until
the element's content is completely processed.
I'm not sure this fits into your design requirements; if it does, I'm
more than happy to help test the new code to validate that it works (our
test-case is a 5GB file). If this is too much to do in the short
term, here's what will help at a minimum:
- The ability to read the underlying stream, at the start of the
element's content, in a repeated loop till the end-tag. The only
setting I would like to specify in such a method is the number of
bytes to return in each iteration. with that, I should be able to
do what I need.
On 11/22/2010 02:18 AM, Elliotte Rusty Harold wrote:
> Doesn't sound like you're missing anything. I suspect your use case is
> just extreme enough that you maybe hitting some genuine limitations.
> XML encryption is something I've thought about exploring for XOM. Can
> you describe ina little more detail what methods you'd need to make
> this work? I.e. if there's one method or two extra methods on Node or
> Element or NodeFactory or some such that would make your job easier
> then maybe I can add them. Though if you really need the possibility
> to put gigabytes of text in one text node, that's going to require
> somewhat more major surgery. Perhaps not impossible but not trivial.
More information about the XOM-interest