xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Re: [XOM-interest] BNUX Question

From: Wolfgang Hoschek <wolfgang.hoschek AT mac.com>
To: Galip Aydin <gaydin AT grids.ucs.indiana.edu>
Cc: xom-interest AT lists.ibiblio.org
Subject: Re: [XOM-interest] BNUX Question
Date: Wed, 26 Jul 2006 15:24:25 -0700

OK. I guess I forgot to include the corresponding new Util.java file, sorry!
It should compile if you add the following new method to Util.java:

/** Reuses XOM's way of representing attribute types */
public static byte getAttributeTypeCode(Attribute.Type type) {
return (byte) type.hashCode();
}

Wolfgang.

On Jul 26, 2006, at 2:29 PM, Galip Aydin wrote:

Wolfgang,

I tried to re-compile the snapshot code but got this error message;
"BinaryXMLCodec.java": getAttributeTypeCode(nu.xom.Attribute) in nux.xom.binary.Util cannot be applied to (nu.xom.Attribute.Type) at line 1432, column 37

The line that causes this message:
nodeTokens.add(Util.getAttributeTypeCode(attrType));

I previously removed this line to get it to compile (bad idea!) which may be the reason I was getting the error message I mentioned in my previous message.

Do you know if anything is wrong with this line?

Thanks
Galip.

----- Original Message ----- From: "Wolfgang Hoschek" <wolfgang.hoschek AT mac.com>
To: "Galip Aydin" <gaydin AT grids.ucs.indiana.edu>
Cc: <xom-interest AT lists.ibiblio.org>
Sent: Tuesday, July 25, 2006 8:34 PM
Subject: Re: [XOM-interest] BNUX Question

Can't reproduce this here.

Are you sure you're using woodstox? Other stax impls are known to be buggy, including the RI. Running

XMLInputFactory inputFactory = XMLInputFactory.newInstance();
System.out.println(inputFactory.getClass().getName());

should yield "com.ctc.wstx.stax.WstxInputFactory".

To be sure, I'd recommend getting rid of *all* jars related to STAX on the classpath, except wstx-asl-3.0.jar and stax-1.0.1.jar, both of which ship with woodstox. In particular, remove jsr173_1.0_ri.jar and jsr173_api.jar.

With a buggy XML parser, there's potential for malformed input to be handed to a serializer such as BinaryXMLStreamWriter or StreamingBinaryXMLSerializer.

StreamingBinaryXMLSerializer is a high level API, performing full wellformedness checks. Serialization would throw an exception if there were a problem. In constrast, BinaryXMLStreamWriter is a low level API, performing only basic wellformedness checks, and those don't include checking for malformed NCNames. BinaryXMLStreamWriter would let such malformed data pass through into the output file, where the problem would later be detected when attempting to reparse, throwing a BinaryParsingException.

[As an aside, XMLInputFactory.newInstance() is expensive, so reusing the instance N times is often a good idea].

Wolfgang.

On Jul 25, 2006, at 12:10 PM, Galip Aydin wrote:

Wolfgang,
I was able to compile and run the prototype code, however when I try to convert the generated bnux documents back to XML I am getting following error, any ideas?

nux.xom.binary.BinaryParsingException: NCNames cannot start with the character a
at nux.xom.binary.BinaryXMLCodec.deserialize(BinaryXMLCodec.java: 627)
at nux.xom.binary.BinaryXMLCodec.deserialize(BinaryXMLCodec.java: 535)
at nux.xom.binary.TestNux.bnux2xml(TestNux.java:46)
at nux.xom.binary.TestNux.main(TestNux.java:60

Here is methods I use:

public void xml2bnux(InputStream in, OutputStream out) throws Exception {
//XMLStreamReader reader = StaxUtil.createXMLStreamReader (in, "");
XMLStreamWriter writer = new BinaryXMLCodec ().createXMLStreamWriter(out,0);
XMLInputFactory inputFactory = XMLInputFactory.newInstance();
XMLStreamReader reader = inputFactory.createXMLStreamReader (in);
new XMLStreamCopier(reader, writer).copy(false); // performs streaming conversion
reader.close();
writer.close();
}

public void bnux2xml (byte[] bytes)throws Exception{
BinaryXMLCodec bnuxCodec = new BinaryXMLCodec();
Document d = bnuxCodec.deserialize(bytes);
System.out.println(d.toXML());
}

Thanks,
Galip.

----- Original Message ----- From: "Wolfgang Hoschek" <wolfgang.hoschek AT mac.com>
To: "Galip Aydin" <gaydin AT grids.ucs.indiana.edu>
Cc: <xom-interest AT lists.ibiblio.org>
Sent: Monday, July 24, 2006 12:51 AM
Subject: Re: [XOM-interest] BNUX Question

Galip,

The prototype source code is at

http://dsd.lbl.gov/nux-download/snapshots/bnuxwriter-0.1.tar.gz

The main class is BinaryXMLStreamWriter, instantiated via

XMLStreamWriter writer = BinaryXMLCodec.createXMLStreamWriter(...)

To compile the files into a nux.jar file, copy the unpacked files into nux/src/nux/xom/binary/

and run

cd nux; ant jar

(Quite a few Stax features aren't implemented, but those aren't required for copying via XMLStreamCopier from an XMLStreamReader to an XMLStreamWriter. Of course, turning this class into production quality code would take a bit more effort.)

Enjoy,
Wolfgang.

On Jul 23, 2006, at 12:59 PM, Galip Aydin wrote:

Wolfgang,
First, thank you for taking your time this weekend to do the profiling.

[Let me know if you'd like to experiment with the prototype.]

Yes I would like that, please let me know how.

[As an aside, some of the larger documents appear to consist of N
concatenations of a base document. If that's indeed the case, if
would be better to use real data throughout, if available. The reason
is that binary tokenization schemes can take advantage of such
artificial redundancy in manners that are not representative of real
world usage.]

That is indeed the case, I just wanted to have certain number of features per doc, so simply did copy/paste. But I will generate a new test set from real data.

Galip.

Re: [XOM-interest] BNUX Question , (continued)