XOM API for Processing XML with Java

[Wolfgang Hoschek <wolfgang.hoschek AT mac.com>]

Looks like a bug in the java-1.6 SAX XML parser. A heap dump on OOM  
shows that there is a (single!) instance of  
com.sun.org.apache.xerces.internal.util.XMLStringBuffer that consumes  
40 MB of memory in a single char[] array.

The bug disappears when using an official Apache Xerces release  
instead of sun's weird xerces variant, for example via

java -Xmx10m -XX:+HeapDumpOnOutOfMemoryError -Djava.endorsed.dirs=/ 
Users/hoschek/unix/java/share/apache/xerces-2.9.0/

I suggest using this route as a workaround.

Wolfgang.

On Nov 19, 2007, at 10:09 PM, Wolfgang Hoschek wrote:

> The same OOM happens with java-1.6 if XOM and Nux are excluded from
> the picture completely, by running the XML file through an empty SAX
> handler, even with 100 MB heap. Hm, beyond the SAX parser class name,
> something in the jaxp XMLReader impl must have changed from java-1.5
> to 1.6.
>
>
> Using SAX parser class:
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
>      public static void main(String[] args) throws Exception {
>          System.setProperty("entityExpansionLimit", String.valueOf
> (Integer.MAX_VALUE));
>          XMLReader reader = createXMLReader();
>          reader.parse(new InputSource(new FileInputStream(args[0])));
>      }
>         
>      private static XMLReader createXMLReader() throws SAXException,
> ParserConfigurationException {
>          SAXParserFactory factory = SAXParserFactory.newInstance();
>          factory.setNamespaceAware(true);
>          factory.setValidating(false);
>          XMLReader reader = factory.newSAXParser().getXMLReader();
>          System.out.println("Using SAX parser class: " +
> reader.getClass().getName());
>          reader.setFeature("http://xml.org/sax/features/namespaces";,
> true);
>          reader.setFeature("http://xml.org/sax/features/namespace-
> prefixes", false);
>          return reader;
>      }
>
> Wolfgang.
>
> On Nov 19, 2007, at 9:47 PM, Wolfgang Hoschek wrote:
>
> > Thanks. Interestingly this OOM can be reproduced with sun java-1.6
> > but not with java-1.5. I'm not sure why yet.
> >
> > Wolfgang.
> >
> > On Nov 19, 2007, at 9:01 PM, Cedric Coulon wrote:
> >
> > > Hi,
> > >
> > > First, thanks you for your help.
> > >
> > > I tried with this two options (setProperty entityExpansionLimit and
> > > java -Xmx10m or 512m) but the result is the same... I use Eclipse,
> > > jdk 1.6.0_03-b05 and nux 1.6 (with xom-1.1 and saxon9 but I also
> > > tried with jars included in nux) under Windows XP.
> > >
> > > Here is the detailed exception:
> > >
> > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap
> > > space
> > >    at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append
> > > (XMLStringBuffer.java:205)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.refre 
> > > s
> > > h
> > > (XMLDocumentScannerImpl.java:1493)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.invokeListe 
> > > n
> > > e
> > > rs(XMLEntityScanner.java:2070)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanQName
> > > (XMLEntityScanner.java:820)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm 
> > > p
> > > l
> > > .scanStartElement(XMLDocumentFragmentScannerImpl.java:1277)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm 
> > > p
> > > l
> > > $FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java: 
> > > 2740)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next
> > > (XMLDocumentScannerImpl.java:645)
> > >    at
> > > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerIm 
> > > p
> > > l
> > > .scanDocument(XMLDocumentFragmentScannerImpl.java:508)
> > >    at
> > > com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
> > > (DTDConfiguration.java:496)
> > >    at
> > > com.sun.org.apache.xerces.internal.parsers.DTDConfiguration.parse
> > > (DTDConfiguration.java:552)
> > >    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse
> > > (XMLParser.java:107)
> > >    at
> > > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse
> > > (AbstractSAXParser.java:1205)
> > >    at nu.xom.Builder.build(Builder.java:1127)
> > >    at nu.xom.Builder.build(Builder.java:614)
> > >    at nu.xom.Builder.build(Builder.java:990)
> > >    at Parser.main(Parser.java:35)
> > >
> > > 35             builder.build(new File("dblp.xml"));    // http://
> > > dblp.uni-trier.de/xml/
> > >
> > >
> > >
> > > Wolfgang Hoschek wrote:
> > > > This example works fine here with a 10 MB heap (java -Xmx10m) and
> > > > the nux-1.6 release. I just downloaded the DBLP data (400 MB)
> > > > again and ran your snippet with sun java-1.5. BTW, since the DBLP
> > > > data includes a lot of external entities you need to bump up the
> > > > entity expansion limit according to http://java.sun.com/
> > > > webservices/docs/1.5/jaxp/ReleaseNotes.html via
> > > >
> > > > System.setProperty("entityExpansionLimit", String.valueOf
> > > > (Integer.MAX_VALUE));
> > > >
> > > > More detailed config and version info for your environment might
> > > > help to pinpoint the problem.
> > > >
> > > > Wolfgang.
> > > >
> > > > On Nov 19, 2007, at 2:39 AM, Cedric Coulon wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I want to parse a large xml file and extract the element book.
> > > > > The xml
> > > > > source file is dblp database (dblp.xml - 411M -
> > > > > http://dblp.uni-trier.de/xml/).
> > > > > So I use NUX and the StreamingPathFilter class. However I have an
> > > > > OutOfMemoryError... I don't know what I'm doing wrong.
> > > > >
> > > > > Thanks you for your help.
> > > > >
> > > > > Bye,
> > > > > Cedric.
> > > > >
> > > > > Here is the source of my java program:
> > > > >
> > > > >     public static void main(String[] args) {
> > > > >         try {
> > > > >             StreamingTransform myTransform = new
> > > > > StreamingTransform() {
> > > > >                 public Nodes transform(Element e) {
> > > > >                     System.out.println(XOMUtil.toPrettyXML(e));
> > > > >                     return new Nodes();
> > > > >                 }
> > > > >             };
> > > > >
> > > > >             Builder builder = new Builder(new
> > > > > StreamingPathFilter("/dblp/book", null).createNodeFactory(null,
> > > > > myTransform));
> > > > >             builder.build(new File("dblp.xml"));    //
> > > > > http://dblp.uni-trier.de/xml/
> > > > >         } catch (Exception e) {
> > > > >             e.printStackTrace();
> > > > >         }
> > > > >     }
> > > > >
> > > > > _______________________________________________
> > > > > XOM-interest mailing list
> > > > > XOM-interest AT lists.ibiblio.org
> > > > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >
> > _______________________________________________
> > XOM-interest mailing list
> > XOM-interest AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/xom-interest
>
> _______________________________________________
> XOM-interest mailing list
> XOM-interest AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/xom-interest