Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] NUX, StreamingPathFilter and OutOfMemoryError

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <wolfgang.hoschek AT mac.com>
  • To: Cedric Coulon <coulon AT comp.nus.edu.sg>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] NUX, StreamingPathFilter and OutOfMemoryError
  • Date: Mon, 19 Nov 2007 22:09:18 -0800

The same OOM happens with java-1.6 if XOM and Nux are excluded from the picture completely, by running the XML file through an empty SAX handler, even with 100 MB heap. Hm, beyond the SAX parser class name, something in the jaxp XMLReader impl must have changed from java-1.5 to 1.6.


Using SAX parser class: com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

public static void main(String[] args) throws Exception {
System.setProperty("entityExpansionLimit", String.valueOf (Integer.MAX_VALUE));
XMLReader reader = createXMLReader();
reader.parse(new InputSource(new FileInputStream(args[0])));
}

private static XMLReader createXMLReader() throws SAXException, ParserConfigurationException {
SAXParserFactory factory = SAXParserFactory.newInstance();
factory.setNamespaceAware(true);
factory.setValidating(false);
XMLReader reader = factory.newSAXParser().getXMLReader();
System.out.println("Using SAX parser class: " + reader.getClass().getName());
reader.setFeature("http://xml.org/sax/features/namespaces";, true);
reader.setFeature("http://xml.org/sax/features/namespace- prefixes", false);
return reader;
}

Wolfgang.

On Nov 19, 2007, at 9:47 PM, Wolfgang Hoschek wrote:

Thanks. Interestingly this OOM can be reproduced with sun java-1.6
but not with java-1.5. I'm not sure why yet.

Wolfgang.

On Nov 19, 2007, at 9:01 PM, Cedric Coulon wrote:

Hi,

First, thanks you for your help.

I tried with this two options (setProperty entityExpansionLimit and
java -Xmx10m or 512m) but the result is the same... I use Eclipse,
jdk 1.6.0_03-b05 and nux 1.6 (with xom-1.1 and saxon9 but I also
tried with jars included in nux) under Windows XP.

Here is the detailed exception:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append
(XMLStringBuffer.java:205)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.refres h
(XMLDocumentScannerImpl.java:1493)
at
com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.invokeListen e
rs(XMLEntityScanner.java:2070)
at
com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanQName
(XMLEntityScanner.java:820)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp l
.scanStartElement(XMLDocumentFragmentScannerImpl.java:1277)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp l
$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2740)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next
(XMLDocumentScannerImpl.java:645)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImp l
.scanDocument(XMLDocumentFragmentScannerImpl.java:508)
at
com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
(DTDConfiguration.java:496)
at
com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
(DTDConfiguration.java:552)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse
(XMLParser.java:107)
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse
(AbstractSAXParser.java:1205)
at nu.xom.Builder.build(Builder.java:1127)
at nu.xom.Builder.build(Builder.java:614)
at nu.xom.Builder.build(Builder.java:990)
at Parser.main(Parser.java:35)

35 builder.build(new File("dblp.xml")); // http://
dblp.uni-trier.de/xml/



Wolfgang Hoschek wrote:
This example works fine here with a 10 MB heap (java -Xmx10m) and
the nux-1.6 release. I just downloaded the DBLP data (400 MB)
again and ran your snippet with sun java-1.5. BTW, since the DBLP
data includes a lot of external entities you need to bump up the
entity expansion limit according to http://java.sun.com/
webservices/docs/1.5/jaxp/ReleaseNotes.html via

System.setProperty("entityExpansionLimit", String.valueOf
(Integer.MAX_VALUE));

More detailed config and version info for your environment might
help to pinpoint the problem.

Wolfgang.

On Nov 19, 2007, at 2:39 AM, Cedric Coulon wrote:

Hi,

I want to parse a large xml file and extract the element book.
The xml
source file is dblp database (dblp.xml - 411M -
http://dblp.uni-trier.de/xml/).
So I use NUX and the StreamingPathFilter class. However I have an
OutOfMemoryError... I don't know what I'm doing wrong.

Thanks you for your help.

Bye,
Cedric.

Here is the source of my java program:

public static void main(String[] args) {
try {
StreamingTransform myTransform = new
StreamingTransform() {
public Nodes transform(Element e) {
System.out.println(XOMUtil.toPrettyXML(e));
return new Nodes();
}
};

Builder builder = new Builder(new
StreamingPathFilter("/dblp/book", null).createNodeFactory(null,
myTransform));
builder.build(new File("dblp.xml")); //
http://dblp.uni-trier.de/xml/
} catch (Exception e) {
e.printStackTrace();
}
}

_______________________________________________
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest



_______________________________________________
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest





Archive powered by MHonArc 2.6.24.

Top of Page