XOM API for Processing XML with Java

["Galip Aydin" <gaydin AT grids.ucs.indiana.edu>]

Hi Wolfgang,
My test results, java files and XML docs are here:
http://gf8.ucs.indiana.edu:7474/bnux/

The fi-bnux-comparison.doc has the graphs to summarize the tests and the tar 
ball has the source, jars and xml files. The xml files are named according 
to the number of features they contain. A feature is basically an xml 
element to describe some geographic information.

I would appreciate any feedback on bnux usage,

Thanks
Galip.


----- Original Message ----- 
From: "Wolfgang Hoschek" <wolfgang.hoschek AT mac.com>
To: "Galip Aydin" <gaydin AT grids.ucs.indiana.edu>
Cc: <xom-interest AT lists.ibiblio.org>
Sent: Thursday, July 20, 2006 7:04 PM
Subject: Re: [XOM-interest] BNUX Question


> Hi Galip,
>
> That's a bit odd indeed. Here are some ideas:
>
> Are you sure you are using woodstox-3.0 (rather than some other STAX 
> impl)? Did you also try the SAX input route, using the same XML  parser 
> that's used for FI?
>
> Perhaps wrapping the input stream into a reasonably sized 
> BufferedInputStream helps (e.g. 8 KB)? This isn't necessary for bnux 
> because it does its own internal buffering, and it's not necessary  for 
> xerces/SAX either, and I thought it isn't necessary for woodstox  either, 
> but it can't hurt to try experimenting with that.
>
> How are you measuring the SAX <--> FI route?
>
> Does the driver run a single JVM instance for more than 30 secs  (ensuring 
> warmup to compiler and VM steady state)? Does the file  system cache play 
> a role? Try removing actual disk and network I/O to  focus on the 
> interesting portions.
>
> What kind of input data is being used? Does it use DTDs? Could you  share 
> the docs with us so we could play with it?
>
> Finally, what's the use case you are working on? For best efficiency, 
> perhaps you could try to eliminate reading and writing standard  textual 
> XML entirelly, using a binary route throughout. In other  words, if the 
> the sender directly generates binary and the receiver  directly consumes 
> binary the expensive intermediate file conversion  step from and to 
> textual XML can be avoided altogether.
>
> Regards,
> Wolfgang.
>
> On Jul 20, 2006, at 3:28 PM, Galip Aydin wrote:
>
> > Wolfgang,
> > I've been doing some tests to compare BNUX and Fast Infoset  performance 
> > for various document sizes, my findings are;
> > - Encoding: FI (with SAX) is almost twice as fast as BNUX (with  stax) ,
> > - Decoding: BNUX is 2 to 3 times faster than FI.
> >
> > So I have no problem with decoding but somehow I can't get good  results 
> > with encoding, and was wondering if I am doing anything  wrong. The 
> > fastest way I could measure for xml2bnux among your  previous suggestions 
> > is this:
> >
> > serializer = codec.createStreamingSerializer(out, 0);
> > redirector = XOMUtil.getRedirectingNodeFactory(serializer);
> > StaxUtil.createBuilder(null, redirector).build(in);
> >
> >
> > The sizes of the XML documents are between 500KB to 5MB.
> > Any ideas what might be happening?
> > Thanks,
> > Galip.
> >
> >
> > ----- Original Message ----- From: "Wolfgang Hoschek" 
> > <wolfgang.hoschek AT mac.com>
> > To: "Galip Aydin" <gaydin AT grids.ucs.indiana.edu>
> > Cc: <xom-interest AT lists.ibiblio.org>
> > Sent: Tuesday, July 18, 2006 6:57 PM
> > Subject: Re: [XOM-interest] BNUX Question
> >
> >
> > > On Jul 18, 2006, at 3:42 PM, Wolfgang Hoschek wrote:
> > >
> > > > On Jul 18, 2006, at 3:12 PM, Galip Aydin wrote:
> > > >
> > > > > Hi All;
> > > > > What is the fastest way to convert an XML document to bnux and  vice-
> > > > > versa,
> > > > > and does anybody have any examples?
> > > > > I see that the last version of bnux is faster than Fast Infoset but
> > > > > I want
> > > > > to make sure that I am using the right methods.
> > > > >
> > > > > Here is what I use:
> > > > >      Document doc = new Builder().build(new File(XMLpath));
> > > > >      BinaryXMLCodec codec = new BinaryXMLCodec();
> > > > >      byte[] bnuxDoc = codec.serialize(doc, 0);
> > > >
> > > > The given code materializes the document as a XOM main memory tree,
> > > > requiring memory proportional to the size of the input document. It
> > > > is convenient but can be expensive or even prohibitive for large
> > > > documents. A more efficient approach is using streaming to convert
> > > > arbitrarily large XML documents to and from bnux, requiring only
> > > > memory proportional to the maximum depth of the XML tree, i.e.
> > > > constant memory for all practical purposes. This is outlined by the
> > > > example in section "Streaming conversion of standard textual XML to
> > > > and from binary format"
> > > > http://dsd.lbl.gov/nux/#Streaming%20conversion%20of%20standard%
> > > > 20textual%20XML%20to%20and%20from%20binary%20format
> > > >
> > > >    // streaming conversion of standard textual XML to bnux binary  XML:
> > > >    InputStream  in  = new FileInputStream("samples/data/ weblog.xml");
> > > >    OutputStream out = new FileOutputStream("/tmp/weblog.xml.bnux");
> > > >    StreamingSerializerFactory factory = new  StreamingSerializerFactory
> > > > ();
> > > >    StreamingSerializer serializer =  factory.createBinaryXMLSerializer
> > > > (out, 0);
> > > >    NodeFactory redirector = XOMUtil.getRedirectingNodeFactory
> > > > (serializer);
> > > >
> > > >    new Builder(redirector).build(in); // performs streaming  conversion
> > > >
> > > >    in.close();
> > > >    out.close();
> > > >
> > > >
> > > >
> > > >    // streaming conversion of bnux binary XML to standard textual  XML:
> > > >    InputStream  in  = new FileInputStream("/tmp/weblog.xml.bnux");
> > > >    OutputStream out = new FileOutputStream("/tmp/weblog.xml");
> > > >    StreamingSerializerFactory factory = new  StreamingSerializerFactory
> > > > ();
> > > >    StreamingSerializer serializer = factory.createXMLSerializer(out,
> > > > "UTF-8");
> > > >    NodeFactory redirector = XOMUtil.getRedirectingNodeFactory
> > > > (serializer);
> > > >
> > > >    new BinaryXMLCodec().deserialize(in, redirector); // performs
> > > > streaming conversion
> > > >
> > > >    in.close();
> > > >    out.close();
> > > >
> > > > There's also a small command line test tool that's essentially a
> > > > glorified version of said streaming example, allowing to convert
> > > > arbitrarily large XML documents to and from bnux. Example:
> > > >
> > > > # bin/fire-bnux < samples/data/teams.xml > /tmp/teams.xml.bnux
> > > > # bin/fire-bnux < /tmp/teams.xml.bnux > /tmp/teams.xml
> > > >
> > > > See class BinaryXMLConverter for the source code. Previously
> > > > mentioned approach uses SAX. Alternatively, you can also use StAX by
> > > > replacing the line
> > > >
> > > > new Builder(redirector).build(in) // performs streaming conversion
> > > >
> > > > with
> > > >
> > > > StaxUtil.createBuilder(null, redirector).build(in) // performs
> > > > streaming conversion
> > > >
> > > > Using StAX with woodstox-3.0 is more efficient than SAX with
> > > > xerces-2.8.0, in particular for very small documents. For detailed
> > > > numbers, see http://dsd.lbl.gov/~hoschek/tmp/plots-stax2.pdf (test
> > > > documents are sorted by file size, left to right).
> > > >
> > > > For reading or writing large numbers of very small documents (e.g.
> > > > 200 bytes), it is more efficient to reuse the Builder and
> > > > BinaryXMLCodec objects, minimizing setup time.
> > > >
> > > > Wolfgang.
> > >
> > > I should add that when writing standard XML it is more efficient  to 
> > > use StAX as well (see the plots). In the example above this  can be 
> > > done by replacing the line
> > >
> > > // use XOM serializer:
> > > StreamingSerializer serializer = factory.createXMLSerializer(out,
> > > "UTF-8");
> > >
> > > with
> > >
> > > // use StAX serializer (woodstox-3.0 recommended):
> > > XMLStreamWriter writer = XMLOutputFactory.newInstance 
> > > ().createXMLStreamWriter(out, "UTF-8");
> > > StreamingSerializer serializer = factory.createStaxSerializer (writer);
> > >
> > > Again, when using many small documents, reusing the StAX 
> > > XMLOutputFactory instance increases efficiency significantly.
> > >
> > > Similarly, such object reuse on the XML --> bnux route can be   achieved 
> > > by replacing
> > >
> > > StreamingSerializer serializer = factory.createBinaryXMLSerializer 
> > > (out, 0);
> > >
> > > with
> > >
> > > StreamingSerializer serializer = codec.createStreamingSerializer  (out, 
> > > 0);
> > >
> > > where "codec" is a BinaryXMLCodec instance that's beeing reused N 
> > > times.
> > >
> > > Wolfgang.