XOM API for Processing XML with Java

[Geet Gangwar <geetgangwar AT gmail.com>]

Hi Michael,

As suggested by you I have used /xbrli:xbrl/xbrli:context to get all
context from the file. But apart from context there are other child nodes
of /xbrli:xbrl nodes which are around 50 millions in number and I want a
list of all those elements based on attribute. I am using Xpath as
/xbrli:xbrl/[@contextRef] but getting heap space errors. The problem is
there is not specific node name available for nodes all names are different.


Regards

Geet

On Mon, Apr 30, 2012 at 7:12 PM, Michael Kay <mike AT saxonica.com> wrote:

> In that case I would suggest starting with an initial path of
> /xbrli:xbrl/xbrli:context rather than /xbrli:xbrl.
>
> Michael Kay
> Saxonica
>
> On 30/04/2012 14:16, Geet Gangwar wrote:
> > There multiple occurrences of contexts element say upto 10 million.
> >
> > Validation logic for each object is same. I need to pick one context
> > perform the validation logic, pick another perform the same logic.
> >
> > Regards
> >
> > Geet
> >
> > On Mon, Apr 30, 2012 at 6:33 PM, Michael Kay<mike AT saxonica.com>  wrote:
> >
> >> Follow-up questions then:
> >>
> >> (1) how big is an xbrli:context object?
> >>
> >> (2) is the validation of each xbrli:context object independent of the
> >> others?
> >>
> >> (3) what is the logic of the validation that needs to be performed?
> >>
> >> Michael Kay
> >> Saxonica
> >>
> >> On 30/04/2012 13:37, Geet Gangwar wrote:
> >>> Hi Michael,
> >>>
> >>> Basically I need to perform some validation on the xml files. So my
> >>> requirement is to pick particular block from xml and validate that
> block.
> >>> For example say I want to pull all context i.e
> >>>
> >>> XPATH expression like :  /xbrli:xbrl/xbrli:context
> >>>
> >>> from the document and validate them.
> >>>
> >>> Regards
> >>>
> >>> Geet
> >>>
> >>> On Mon, Apr 30, 2012 at 5:41 PM, Michael Kay<mike AT saxonica.com>
> wrote:
> >>>
> >>>> NUX implements what one might call "chunked streaming": it selects a
> >>>> sequence of subtrees of your document and applies XQuery processing to
> >>>> each one in turn. (This is effectively the same as saxon:stream() in
> >>>> Saxon.) You need to have enough memory to hold the largest of these
> >>>> subtrees. Because your path /xbrli:xbrl effectively selects the whole
> >>>> document, your largest subtree is the same as the document size, so
> you
> >>>> aren't really streaming at all.
> >>>>
> >>>> If you can describe the query/transformation you want to carry out,
> then
> >>>> I might be able to suggest a way of doing it in a streamed manner
> either
> >>>> using NUX, or using native facilities of Saxon (whose streaming
> >>>> capabilities have advanced considerably since NUX was last released).
> >>>>
> >>>> Michael Kay
> >>>> Saxonica
> >>>>
> >>>>
> >>>> 2012/4/30 Geet Gangwar<geetgangwar AT gmail.com>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I have a requirement of processing large XML files of about 4GB. I
> am
> >>>> using
> >>>>>> the NUX distribution to process the files.
> >>>>>> But when I fire a Xquery to this read this file, I get JAVA Heap
> Space
> >>>>>> error.
> >>>>>>
> >>>>>> I am using 64bit JDK1.6 and also I have increased the JVM size to 5
> GB
> >>>> by
> >>>>>> -Xmx5G.
> >>>>>>
> >>>>>> Please help me how to solve this problem.
> >>>>>>
> >>>>>> I am pasting my sample code also.
> >>>>>>
> >>>>>> String path = "/xbrli:xbrl/";
> >>>>>>                Map prefixes = new HashMap();
> >>>>>>                prefixes.put("xbrli", "
> >> http://www.xbrl.org/2003/instance
> >>>> ");
> >>>>>>                prefixes.put("link", "
> http://www.xbrl.org/2003/linkbase
> >> ");
> >>>>>>                prefixes.put("xlink", "http://www.w3.org/1999/xlink
> ");
> >>>>>>                prefixes.put("xsd", "
> http://www.w3.org/2001/XMLSchema
> >> ");
> >>>>>>                StreamingTransform myTransform = new
> >> StreamingTransform() {
> >>>>>>                    private Nodes NONE = new Nodes();
> >>>>>>
> >>>>>>                    // execute XQuery against each element matching
> >>>> location
> >>>>>> path
> >>>>>>                    public Nodes transform(Element subtree) {
> >>>>>>                        Nodes results = XQueryUtil.xquery(subtree,
> >>>>>> "xbrli:context[id = 'c-01']");
> >>>>>>
> >>>>>>                        for (int i=0; i<    results.size(); i++) {
> >>>>>>                            // do something useful with query
> results;
> >>>> here we
> >>>>>> just print them
> >>>>>>
> >>>>>> System.out.println(XOMUtil.toPrettyXML(results.get(i)));
> >>>>>>                        }
> >>>>>>                        return NONE; // current subtree becomes
> subject
> >> to
> >>>>>> garbage collection
> >>>>>>                        // returning empty node list removes current
> >>>> subtree
> >>>>>> from document being build.
> >>>>>>                        // returning new Nodes(subtree) retains the
> >> current
> >>>>>> subtree.
> >>>>>>                        // returning new Nodes(some other nodes)
> >> replaces
> >>>> the
> >>>>>> current subtree with
> >>>>>>                        // some other nodes.
> >>>>>>                        // if you want (SAX) parsing to terminate at
> >> this
> >>>>>> point, simply throw an exception
> >>>>>>                    }
> >>>>>>                };
> >>>>>>
> >>>>>>                // parse document with a filtering Builder
> >>>>>>                StreamingPathFilter filter = new
> >> StreamingPathFilter(path,
> >>>>>> prefixes);
> >>>>>>                Builder builder = new
> >>>> Builder(filter.createNodeFactory(null,
> >>>>>> myTransform));
> >>>>>>                Document doc = builder.build(new
> >>>> File("F:/sample/cntxt.xml"));
> >>>>>>                System.out.println(doc.getRootElement().getValue());
> >>>>>>                System.out.println("doc.size()=" +
> >>>>>> doc.getRootElement().getChildCount());
> >>>>>>                System.out.println(XOMUtil.toPrettyXML(doc));
> >>>>>>
> >>>>>> Regards
> >>>>>>
> >>>>>> Geet
> >>>>>> _______________________________________________
> >>>>>> XOM-interest mailing list
> >>>>>> XOM-interest AT lists.ibiblio.org
> >>>>>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >>>>>>
> >>>>> _______________________________________________
> >>>>> XOM-interest mailing list
> >>>>> XOM-interest AT lists.ibiblio.org
> >>>>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >>>>>
> >>>> _______________________________________________
> >>>> XOM-interest mailing list
> >>>> XOM-interest AT lists.ibiblio.org
> >>>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >>>>
> >>> _______________________________________________
> >>> XOM-interest mailing list
> >>> XOM-interest AT lists.ibiblio.org
> >>> http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >>>
> >> _______________________________________________
> >> XOM-interest mailing list
> >> XOM-interest AT lists.ibiblio.org
> >> http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >>
> > _______________________________________________
> > XOM-interest mailing list
> > XOM-interest AT lists.ibiblio.org
> > http://lists.ibiblio.org/mailman/listinfo/xom-interest
> >
> _______________________________________________
> XOM-interest mailing list
> XOM-interest AT lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/xom-interest
>