[XOM-interest] NUX, StreamingPathFilter and OutOfMemoryError

Wolfgang Hoschek wolfgang.hoschek at mac.com
Tue Nov 20 01:22:49 EST 2007


Looks like a bug in the java-1.6 SAX XML parser. A heap dump on OOM  
shows that there is a (single!) instance of  
com.sun.org.apache.xerces.internal.util.XMLStringBuffer that consumes  
40 MB of memory in a single char[] array.

The bug disappears when using an official Apache Xerces release  
instead of sun's weird xerces variant, for example via

java -Xmx10m -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/ 
Users/hoschek/unix/java/share/apache/xerces-2.9.0/

I suggest using this route as a workaround.

Wolfgang.

On Nov 19, 2007, at 10:09 PM, Wolfgang Hoschek wrote:

> The same OOM happens with java-1.6 if XOM and Nux are excluded from
> the picture completely, by running the XML file through an empty SAX
> handler, even with 100 MB heap. Hm, beyond the SAX parser class name,
> something in the jaxp XMLReader impl must have changed from java-1.5
> to 1.6.
>
>
> Using SAX parser class:
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
>      public static void main(String[] args) throws Exception {
>          System.setProperty("entityExpansionLimit", String.valueOf
> (Integer.MAX_VALUE));
>          XMLReader reader = createXMLReader();
>          reader.parse(new InputSource(new FileInputStream(args[0])));
>      }
> 	
>      private static XMLReader createXMLReader() throws SAXException,
> ParserConfigurationException {
>          SAXParserFactory factory = SAXParserFactory.newInstance();
>          factory.setNamespaceAware(true);
>          factory.setValidating(false);
>          XMLReader reader = factory.newSAXParser().getXMLReader();
>          System.out.println("Using SAX parser class: " +
> reader.getClass().getName());
>          reader.setFeature("http://xml.org/sax/features/namespaces",
> true);
>          reader.setFeature("http://xml.org/sax/features/namespace-
> prefixes", false);
>          return reader;
>      }
>
> Wolfgang.
>
> On Nov 19, 2007, at 9:47 PM, Wolfgang Hoschek wrote:
>
>> Thanks. Interestingly this OOM can be reproduced with sun java-1.6
>> but not with java-1.5. I'm not sure why yet.
>>
>> Wolfgang.
>>
>> On Nov 19, 2007, at 9:01 PM, Cedric Coulon wrote:
>>
>>> Hi,
>>>
>>> First, thanks you for your help.
>>>
>>> I tried with this two options (setProperty entityExpansionLimit and
>>> java -Xmx10m or 512m) but the result is the same... I use Eclipse,
>>> jdk 1.6.0_03-b05 and nux 1.6 (with xom-1.1 and saxon9 but I also
>>> tried with jars included in nux) under Windows XP.
>>>
>>> Here is the detailed exception:
>>>
>>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap
>>> space
>>>    at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append
>>> (XMLStringBuffer.java:205)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.refre 
>>> s
>>> h
>>> (XMLDocumentScannerImpl.java:1493)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.invokeListe 
>>> n
>>> e
>>> rs(XMLEntityScanner.java:2070)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanQName
>>> (XMLEntityScanner.java:820)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm 
>>> p
>>> l
>>> .scanStartElement(XMLDocumentFragmentScannerImpl.java:1277)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm 
>>> p
>>> l
>>> $FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java: 
>>> 2740)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next
>>> (XMLDocumentScannerImpl.java:645)
>>>    at
>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm 
>>> p
>>> l
>>> .scanDocument(XMLDocumentFragmentScannerImpl.java:508)
>>>    at
>>> com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
>>> (DTDConfiguration.java:496)
>>>    at
>>> com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
>>> (DTDConfiguration.java:552)
>>>    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse
>>> (XMLParser.java:107)
>>>    at
>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse
>>> (AbstractSAXParser.java:1205)
>>>    at nu.xom.Builder.build(Builder.java:1127)
>>>    at nu.xom.Builder.build(Builder.java:614)
>>>    at nu.xom.Builder.build(Builder.java:990)
>>>    at Parser.main(Parser.java:35)
>>>
>>> 35             builder.build(new File("dblp.xml"));    // http://
>>> dblp.uni-trier.de/xml/
>>>
>>>
>>>
>>> Wolfgang Hoschek wrote:
>>>> This example works fine here with a 10 MB heap (java -Xmx10m) and
>>>> the nux-1.6 release. I just downloaded the DBLP data (400 MB)
>>>> again and ran your snippet with sun java-1.5. BTW, since the DBLP
>>>> data includes a lot of external entities you need to bump up the
>>>> entity expansion limit according to http://java.sun.com/
>>>> webservices/docs/1.5/jaxp/ReleaseNotes.html via
>>>>
>>>> System.setProperty("entityExpansionLimit", String.valueOf
>>>> (Integer.MAX_VALUE));
>>>>
>>>> More detailed config and version info for your environment might
>>>> help to pinpoint the problem.
>>>>
>>>> Wolfgang.
>>>>
>>>> On Nov 19, 2007, at 2:39 AM, Cedric Coulon wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want to parse a large xml file and extract the element book.
>>>>> The xml
>>>>> source file is dblp database (dblp.xml - 411M -
>>>>> http://dblp.uni-trier.de/xml/).
>>>>> So I use NUX and the StreamingPathFilter class. However I have an
>>>>> OutOfMemoryError... I don't know what I'm doing wrong.
>>>>>
>>>>> Thanks you for your help.
>>>>>
>>>>> Bye,
>>>>> Cedric.
>>>>>
>>>>> Here is the source of my java program:
>>>>>
>>>>>     public static void main(String[] args) {
>>>>>         try {
>>>>>             StreamingTransform myTransform = new
>>>>> StreamingTransform() {
>>>>>                 public Nodes transform(Element e) {
>>>>>                     System.out.println(XOMUtil.toPrettyXML(e));
>>>>>                     return new Nodes();
>>>>>                 }
>>>>>             };
>>>>>
>>>>>             Builder builder = new Builder(new
>>>>> StreamingPathFilter("/dblp/book", null).createNodeFactory(null,
>>>>> myTransform));
>>>>>             builder.build(new File("dblp.xml"));    //
>>>>> http://dblp.uni-trier.de/xml/
>>>>>         } catch (Exception e) {
>>>>>             e.printStackTrace();
>>>>>         }
>>>>>     }
>>>>>
>>>>> _______________________________________________
>>>>> XOM-interest mailing list
>>>>> XOM-interest at lists.ibiblio.org
>>>>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
>>>>
>>>
>>
>> _______________________________________________
>> XOM-interest mailing list
>> XOM-interest at lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
>
> _______________________________________________
> XOM-interest mailing list
> XOM-interest at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/xom-interest



More information about the XOM-interest mailing list