XOM API for Processing XML with Java

[Christophe Marchand <christophe.marchand AT axyus.com>]

Sometime, writing your own SAX handler to code your controls is quicker 
than everything else ; and if your code is clean, it will work on very 
very very huge files.

Regards,
Christophe

Le 02/05/2012 15:21, Geet Gangwar a écrit :
> Hi Michael,
>
> As suggested by you I have used /xbrli:xbrl/xbrli:context to get all
> context from the file. But apart from context there are other child nodes
> of /xbrli:xbrl nodes which are around 50 millions in number and I want a
> list of all those elements based on attribute. I am using Xpath as
> /xbrli:xbrl/[@contextRef] but getting heap space errors. The problem is
> there is not specific node name available for nodes all names are different.
>
>
> Regards
>
> Geet
>
> On Mon, Apr 30, 2012 at 7:12 PM, Michael Kay<mike AT saxonica.com>  wrote:
>
> > In that case I would suggest starting with an initial path of
> > /xbrli:xbrl/xbrli:context rather than /xbrli:xbrl.
> >
> > Michael Kay
> > Saxonica
> >
> > On 30/04/2012 14:16, Geet Gangwar wrote:
> > > There multiple occurrences of contexts element say upto 10 million.
> > >
> > > Validation logic for each object is same. I need to pick one context
> > > perform the validation logic, pick another perform the same logic.
> > >
> > > Regards
> > >
> > > Geet
> > >
> > > On Mon, Apr 30, 2012 at 6:33 PM, Michael Kay<mike AT saxonica.com>   wrote:
> > >
> > > > Follow-up questions then:
> > > >
> > > > (1) how big is an xbrli:context object?
> > > >
> > > > (2) is the validation of each xbrli:context object independent of the
> > > > others?
> > > >
> > > > (3) what is the logic of the validation that needs to be performed?
> > > >
> > > > Michael Kay
> > > > Saxonica
> > > >
> > > > On 30/04/2012 13:37, Geet Gangwar wrote:
> > > > > Hi Michael,
> > > > >
> > > > > Basically I need to perform some validation on the xml files. So my
> > > > > requirement is to pick particular block from xml and validate that
> > block.
> > > > > For example say I want to pull all context i.e
> > > > >
> > > > > XPATH expression like :  /xbrli:xbrl/xbrli:context
> > > > >
> > > > > from the document and validate them.
> > > > >
> > > > > Regards
> > > > >
> > > > > Geet
> > > > >
> > > > > On Mon, Apr 30, 2012 at 5:41 PM, Michael Kay<mike AT saxonica.com>
> > wrote:
> > > > > > NUX implements what one might call "chunked streaming": it selects a
> > > > > > sequence of subtrees of your document and applies XQuery processing to
> > > > > > each one in turn. (This is effectively the same as saxon:stream() in
> > > > > > Saxon.) You need to have enough memory to hold the largest of these
> > > > > > subtrees. Because your path /xbrli:xbrl effectively selects the whole
> > > > > > document, your largest subtree is the same as the document size, so
> > you
> > > > > > aren't really streaming at all.
> > > > > >
> > > > > > If you can describe the query/transformation you want to carry out,
> > then
> > > > > > I might be able to suggest a way of doing it in a streamed manner
> > either
> > > > > > using NUX, or using native facilities of Saxon (whose streaming
> > > > > > capabilities have advanced considerably since NUX was last released).
> > > > > >
> > > > > > Michael Kay
> > > > > > Saxonica
> > > > > >
> > > > > >
> > > > > > 2012/4/30 Geet Gangwar<geetgangwar AT gmail.com>
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have a requirement of processing large XML files of about 4GB. I
> > am
> > > > > > using
> > > > > > > > the NUX distribution to process the files.
> > > > > > > > But when I fire a Xquery to this read this file, I get JAVA Heap
> > Space
> > > > > > > > error.
> > > > > > > >
> > > > > > > > I am using 64bit JDK1.6 and also I have increased the JVM size to 5
> > GB
> > > > > > by
> > > > > > > > -Xmx5G.
> > > > > > > >
> > > > > > > > Please help me how to solve this problem.
> > > > > > > >
> > > > > > > > I am pasting my sample code also.
> > > > > > > >
> > > > > > > > String path = "/xbrli:xbrl/";
> > > > > > > >                 Map prefixes = new HashMap();
> > > > > > > >                 prefixes.put("xbrli", "
> > > > http://www.xbrl.org/2003/instance
> > > > > > ");
> > > > > > > >                 prefixes.put("link", "
> > http://www.xbrl.org/2003/linkbase
> > > > ");
> > > > > > > >                 prefixes.put("xlink", "http://www.w3.org/1999/xlink
> > ");
> > > > > > > >                 prefixes.put("xsd", "
> > http://www.w3.org/2001/XMLSchema
> > > > ");
> > > > > > > >                 StreamingTransform myTransform = new
> > > > StreamingTransform() {
> > > > > > > >                     private Nodes NONE = new Nodes();
> > > > > > > >
> > > > > > > >                     // execute XQuery against each element matching
> > > > > > location
> > > > > > > > path
> > > > > > > >                     public Nodes transform(Element subtree) {
> > > > > > > >                         Nodes results = XQueryUtil.xquery(subtree,
> > > > > > > > "xbrli:context[id = 'c-01']");
> > > > > > > >
> > > > > > > >                         for (int i=0; i<     results.size(); i++) {
> > > > > > > >                             // do something useful with query
> > results;
> > > > > > here we
> > > > > > > > just print them
> > > > > > > >
> > > > > > > > System.out.println(XOMUtil.toPrettyXML(results.get(i)));
> > > > > > > >                         }
> > > > > > > >                         return NONE; // current subtree becomes
> > subject
> > > > to
> > > > > > > > garbage collection
> > > > > > > >                         // returning empty node list removes current
> > > > > > subtree
> > > > > > > > from document being build.
> > > > > > > >                         // returning new Nodes(subtree) retains the
> > > > current
> > > > > > > > subtree.
> > > > > > > >                         // returning new Nodes(some other nodes)
> > > > replaces
> > > > > > the
> > > > > > > > current subtree with
> > > > > > > >                         // some other nodes.
> > > > > > > >                         // if you want (SAX) parsing to terminate at
> > > > this
> > > > > > > > point, simply throw an exception
> > > > > > > >                     }
> > > > > > > >                 };
> > > > > > > >
> > > > > > > >                 // parse document with a filtering Builder
> > > > > > > >                 StreamingPathFilter filter = new
> > > > StreamingPathFilter(path,
> > > > > > > > prefixes);
> > > > > > > >                 Builder builder = new
> > > > > > Builder(filter.createNodeFactory(null,
> > > > > > > > myTransform));
> > > > > > > >                 Document doc = builder.build(new
> > > > > > File("F:/sample/cntxt.xml"));
> > > > > > > >                 System.out.println(doc.getRootElement().getValue());
> > > > > > > >                 System.out.println("doc.size()=" +
> > > > > > > > doc.getRootElement().getChildCount());
> > > > > > > >                 System.out.println(XOMUtil.toPrettyXML(doc));
> > > > > > > >
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > Geet
> > > > > > > > _______________________________________________
> > > > > > > > XOM-interest mailing list
> > > > > > > > XOM-interest AT lists.ibiblio.org
> > > > > > > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> > > > > > > _______________________________________________
> > > > > > > XOM-interest mailing list
> > > > > > > XOM-interest AT lists.ibiblio.org
> > > > > > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> > > > > > _______________________________________________
> > > > > > XOM-interest mailing list
> > > > > > XOM-interest AT lists.ibiblio.org
> > > > > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> > > > > _______________________________________________
> > > > > XOM-interest mailing list
> > > > > XOM-interest AT lists.ibiblio.org
> > > > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> > > > _______________________________________________
> > > > XOM-interest mailing list
> > > > XOM-interest AT lists.ibiblio.org
> > > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> > > _______________________________________________
> > > XOM-interest mailing list
> > > XOM-interest AT lists.ibiblio.org
> > > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> > _______________________________________________
> > XOM-interest mailing list
> > XOM-interest AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> _______________________________________________
> XOM-interest mailing list
> XOM-interest AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/xom-interest