Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] NUX, StreamingPathFilter and OutOfMemoryError

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <wolfgang.hoschek AT mac.com>
  • To: Cedric Coulon <coulon AT comp.nus.edu.sg>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] NUX, StreamingPathFilter and OutOfMemoryError
  • Date: Mon, 19 Nov 2007 21:47:58 -0800

Thanks. Interestingly this OOM can be reproduced with sun java-1.6 but not with java-1.5. I'm not sure why yet.

Wolfgang.

On Nov 19, 2007, at 9:01 PM, Cedric Coulon wrote:

Hi,

First, thanks you for your help.

I tried with this two options (setProperty entityExpansionLimit and java -Xmx10m or 512m) but the result is the same... I use Eclipse, jdk 1.6.0_03-b05 and nux 1.6 (with xom-1.1 and saxon9 but I also tried with jars included in nux) under Windows XP.

Here is the detailed exception:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append (XMLStringBuffer.java:205)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.refresh (XMLDocumentScannerImpl.java:1493)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.invokeListene rs(XMLEntityScanner.java:2070)
at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanQName (XMLEntityScanner.java:820)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl .scanStartElement(XMLDocumentFragmentScannerImpl.java:1277)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl $FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2740)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next (XMLDocumentScannerImpl.java:645)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl .scanDocument(XMLDocumentFragmentScannerImpl.java:508)
at com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse (DTDConfiguration.java:496)
at com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse (DTDConfiguration.java:552)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse (XMLParser.java:107)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse (AbstractSAXParser.java:1205)
at nu.xom.Builder.build(Builder.java:1127)
at nu.xom.Builder.build(Builder.java:614)
at nu.xom.Builder.build(Builder.java:990)
at Parser.main(Parser.java:35)

35 builder.build(new File("dblp.xml")); // http:// dblp.uni-trier.de/xml/



Wolfgang Hoschek wrote:
This example works fine here with a 10 MB heap (java -Xmx10m) and the nux-1.6 release. I just downloaded the DBLP data (400 MB) again and ran your snippet with sun java-1.5. BTW, since the DBLP data includes a lot of external entities you need to bump up the entity expansion limit according to http://java.sun.com/ webservices/docs/1.5/jaxp/ReleaseNotes.html via

System.setProperty("entityExpansionLimit", String.valueOf (Integer.MAX_VALUE));

More detailed config and version info for your environment might help to pinpoint the problem.

Wolfgang.

On Nov 19, 2007, at 2:39 AM, Cedric Coulon wrote:

Hi,

I want to parse a large xml file and extract the element book. The xml
source file is dblp database (dblp.xml - 411M -
http://dblp.uni-trier.de/xml/).
So I use NUX and the StreamingPathFilter class. However I have an
OutOfMemoryError... I don't know what I'm doing wrong.

Thanks you for your help.

Bye,
Cedric.

Here is the source of my java program:

public static void main(String[] args) {
try {
StreamingTransform myTransform = new StreamingTransform() {
public Nodes transform(Element e) {
System.out.println(XOMUtil.toPrettyXML(e));
return new Nodes();
}
};

Builder builder = new Builder(new
StreamingPathFilter("/dblp/book", null).createNodeFactory(null,
myTransform));
builder.build(new File("dblp.xml")); //
http://dblp.uni-trier.de/xml/
} catch (Exception e) {
e.printStackTrace();
}
}

_______________________________________________
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest







Archive powered by MHonArc 2.6.24.

Top of Page