[XOM-interest] NUX, StreamingPathFilter and OutOfMemoryError
Wolfgang Hoschek
wolfgang.hoschek at mac.com
Tue Nov 20 01:22:49 EST 2007
Looks like a bug in the java-1.6 SAX XML parser. A heap dump on OOM
shows that there is a (single!) instance of
com.sun.org.apache.xerces.internal.util.XMLStringBuffer that consumes
40 MB of memory in a single char[] array.
The bug disappears when using an official Apache Xerces release
instead of sun's weird xerces variant, for example via
java -Xmx10m -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/
Users/hoschek/unix/java/share/apache/xerces-2.9.0/
I suggest using this route as a workaround.
Wolfgang.
On Nov 19, 2007, at 10:09 PM, Wolfgang Hoschek wrote:
> The same OOM happens with java-1.6 if XOM and Nux are excluded from
> the picture completely, by running the XML file through an empty SAX
> handler, even with 100 MB heap. Hm, beyond the SAX parser class name,
> something in the jaxp XMLReader impl must have changed from java-1.5
> to 1.6.
>
>
> Using SAX parser class:
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> public static void main(String[] args) throws Exception {
> System.setProperty("entityExpansionLimit", String.valueOf
> (Integer.MAX_VALUE));
> XMLReader reader = createXMLReader();
> reader.parse(new InputSource(new FileInputStream(args[0])));
> }
>
> private static XMLReader createXMLReader() throws SAXException,
> ParserConfigurationException {
> SAXParserFactory factory = SAXParserFactory.newInstance();
> factory.setNamespaceAware(true);
> factory.setValidating(false);
> XMLReader reader = factory.newSAXParser().getXMLReader();
> System.out.println("Using SAX parser class: " +
> reader.getClass().getName());
> reader.setFeature("http://xml.org/sax/features/namespaces",
> true);
> reader.setFeature("http://xml.org/sax/features/namespace-
> prefixes", false);
> return reader;
> }
>
> Wolfgang.
>
> On Nov 19, 2007, at 9:47 PM, Wolfgang Hoschek wrote:
>
>> Thanks. Interestingly this OOM can be reproduced with sun java-1.6
>> but not with java-1.5. I'm not sure why yet.
>>
>> Wolfgang.
>>
>> On Nov 19, 2007, at 9:01 PM, Cedric Coulon wrote:
>>
>>> Hi,
>>>
>>> First, thanks you for your help.
>>>
>>> I tried with this two options (setProperty entityExpansionLimit and
>>> java -Xmx10m or 512m) but the result is the same... I use Eclipse,
>>> jdk 1.6.0_03-b05 and nux 1.6 (with xom-1.1 and saxon9 but I also
>>> tried with jars included in nux) under Windows XP.
>>>
>>> Here is the detailed exception:
>>>
>>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap
>>> space
>>> at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append
>>> (XMLStringBuffer.java:205)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.refre
>>> s
>>> h
>>> (XMLDocumentScannerImpl.java:1493)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.invokeListe
>>> n
>>> e
>>> rs(XMLEntityScanner.java:2070)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanQName
>>> (XMLEntityScanner.java:820)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm
>>> p
>>> l
>>> .scanStartElement(XMLDocumentFragmentScannerImpl.java:1277)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm
>>> p
>>> l
>>> $FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:
>>> 2740)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next
>>> (XMLDocumentScannerImpl.java:645)
>>> at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm
>>> p
>>> l
>>> .scanDocument(XMLDocumentFragmentScannerImpl.java:508)
>>> at
>>> com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
>>> (DTDConfiguration.java:496)
>>> at
>>> com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
>>> (DTDConfiguration.java:552)
>>> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse
>>> (XMLParser.java:107)
>>> at
>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse
>>> (AbstractSAXParser.java:1205)
>>> at nu.xom.Builder.build(Builder.java:1127)
>>> at nu.xom.Builder.build(Builder.java:614)
>>> at nu.xom.Builder.build(Builder.java:990)
>>> at Parser.main(Parser.java:35)
>>>
>>> 35 builder.build(new File("dblp.xml")); // http://
>>> dblp.uni-trier.de/xml/
>>>
>>>
>>>
>>> Wolfgang Hoschek wrote:
>>>> This example works fine here with a 10 MB heap (java -Xmx10m) and
>>>> the nux-1.6 release. I just downloaded the DBLP data (400 MB)
>>>> again and ran your snippet with sun java-1.5. BTW, since the DBLP
>>>> data includes a lot of external entities you need to bump up the
>>>> entity expansion limit according to http://java.sun.com/
>>>> webservices/docs/1.5/jaxp/ReleaseNotes.html via
>>>>
>>>> System.setProperty("entityExpansionLimit", String.valueOf
>>>> (Integer.MAX_VALUE));
>>>>
>>>> More detailed config and version info for your environment might
>>>> help to pinpoint the problem.
>>>>
>>>> Wolfgang.
>>>>
>>>> On Nov 19, 2007, at 2:39 AM, Cedric Coulon wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want to parse a large xml file and extract the element book.
>>>>> The xml
>>>>> source file is dblp database (dblp.xml - 411M -
>>>>> http://dblp.uni-trier.de/xml/).
>>>>> So I use NUX and the StreamingPathFilter class. However I have an
>>>>> OutOfMemoryError... I don't know what I'm doing wrong.
>>>>>
>>>>> Thanks you for your help.
>>>>>
>>>>> Bye,
>>>>> Cedric.
>>>>>
>>>>> Here is the source of my java program:
>>>>>
>>>>> public static void main(String[] args) {
>>>>> try {
>>>>> StreamingTransform myTransform = new
>>>>> StreamingTransform() {
>>>>> public Nodes transform(Element e) {
>>>>> System.out.println(XOMUtil.toPrettyXML(e));
>>>>> return new Nodes();
>>>>> }
>>>>> };
>>>>>
>>>>> Builder builder = new Builder(new
>>>>> StreamingPathFilter("/dblp/book", null).createNodeFactory(null,
>>>>> myTransform));
>>>>> builder.build(new File("dblp.xml")); //
>>>>> http://dblp.uni-trier.de/xml/
>>>>> } catch (Exception e) {
>>>>> e.printStackTrace();
>>>>> }
>>>>> }
>>>>>
>>>>> _______________________________________________
>>>>> XOM-interest mailing list
>>>>> XOM-interest at lists.ibiblio.org
>>>>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
>>>>
>>>
>>
>> _______________________________________________
>> XOM-interest mailing list
>> XOM-interest at lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
>
> _______________________________________________
> XOM-interest mailing list
> XOM-interest at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/xom-interest
More information about the XOM-interest
mailing list