xom-interest AT lists.ibiblio.org
Subject: XOM API for Processing XML with Java
List archive
- From: Christophe Marchand <christophe.marchand AT axyus.com>
- To: XOM API for Processing XML with Java <xom-interest AT lists.ibiblio.org>
- Subject: Re: [XOM-interest] Processing XML file of 4GB
- Date: Wed, 02 May 2012 16:11:11 +0200
Sometime, writing your own SAX handler to code your controls is quicker than everything else ; and if your code is clean, it will work on very very very huge files.
Regards,
Christophe
Le 02/05/2012 15:21, Geet Gangwar a écrit :
Hi Michael,
As suggested by you I have used /xbrli:xbrl/xbrli:context to get all
context from the file. But apart from context there are other child nodes
of /xbrli:xbrl nodes which are around 50 millions in number and I want a
list of all those elements based on attribute. I am using Xpath as
/xbrli:xbrl/[@contextRef] but getting heap space errors. The problem is
there is not specific node name available for nodes all names are different.
Regards
Geet
On Mon, Apr 30, 2012 at 7:12 PM, Michael Kay<mike AT saxonica.com> wrote:
In that case I would suggest starting with an initial path of_______________________________________________
/xbrli:xbrl/xbrli:context rather than /xbrli:xbrl.
Michael Kay
Saxonica
On 30/04/2012 14:16, Geet Gangwar wrote:
There multiple occurrences of contexts element say upto 10 million.block.
Validation logic for each object is same. I need to pick one context
perform the validation logic, pick another perform the same logic.
Regards
Geet
On Mon, Apr 30, 2012 at 6:33 PM, Michael Kay<mike AT saxonica.com> wrote:
Follow-up questions then:
(1) how big is an xbrli:context object?
(2) is the validation of each xbrli:context object independent of the
others?
(3) what is the logic of the validation that needs to be performed?
Michael Kay
Saxonica
On 30/04/2012 13:37, Geet Gangwar wrote:
Hi Michael,
Basically I need to perform some validation on the xml files. So my
requirement is to pick particular block from xml and validate that
wrote:For example say I want to pull all context i.e
XPATH expression like : /xbrli:xbrl/xbrli:context
from the document and validate them.
Regards
Geet
On Mon, Apr 30, 2012 at 5:41 PM, Michael Kay<mike AT saxonica.com>
youNUX implements what one might call "chunked streaming": it selects a
sequence of subtrees of your document and applies XQuery processing to
each one in turn. (This is effectively the same as saxon:stream() in
Saxon.) You need to have enough memory to hold the largest of these
subtrees. Because your path /xbrli:xbrl effectively selects the whole
document, your largest subtree is the same as the document size, so
thenaren't really streaming at all.
If you can describe the query/transformation you want to carry out,
eitherI might be able to suggest a way of doing it in a streamed manner
amusing NUX, or using native facilities of Saxon (whose streaming
capabilities have advanced considerably since NUX was last released).
Michael Kay
Saxonica
2012/4/30 Geet Gangwar<geetgangwar AT gmail.com>
Hi,
I have a requirement of processing large XML files of about 4GB. I
Spaceusing
the NUX distribution to process the files.
But when I fire a Xquery to this read this file, I get JAVA Heap
GBerror.
I am using 64bit JDK1.6 and also I have increased the JVM size to 5
http://www.xbrl.org/2003/linkbasehttp://www.xbrl.org/2003/instanceby
-Xmx5G.
Please help me how to solve this problem.
I am pasting my sample code also.
String path = "/xbrli:xbrl/";
Map prefixes = new HashMap();
prefixes.put("xbrli", "
");
prefixes.put("link", "
");");
prefixes.put("xlink", "http://www.w3.org/1999/xlink
http://www.w3.org/2001/XMLSchemaprefixes.put("xsd", "
results;");
StreamingTransform() {StreamingTransform myTransform = new
locationprivate Nodes NONE = new Nodes();
// execute XQuery against each element matching
path
public Nodes transform(Element subtree) {
Nodes results = XQueryUtil.xquery(subtree,
"xbrli:context[id = 'c-01']");
for (int i=0; i< results.size(); i++) {
// do something useful with query
subjecthere we
just print them
System.out.println(XOMUtil.toPrettyXML(results.get(i)));
}
return NONE; // current subtree becomes
_______________________________________________to_______________________________________________
currentsubtreegarbage collection
// returning empty node list removes current
from document being build.
// returning new Nodes(subtree) retains the
replacessubtree.
// returning new Nodes(some other nodes)
thisthe
current subtree with
// some other nodes.
// if you want (SAX) parsing to terminate at
StreamingPathFilter(path,point, simply throw an exception
}
};
// parse document with a filtering Builder
StreamingPathFilter filter = new
______________________________________________________________________________________________Builder(filter.createNodeFactory(null,prefixes);
Builder builder = new
File("F:/sample/cntxt.xml"));myTransform));
Document doc = builder.build(new
_______________________________________________System.out.println(doc.getRootElement().getValue());_______________________________________________
System.out.println("doc.size()=" +
doc.getRootElement().getChildCount());
System.out.println(XOMUtil.toPrettyXML(doc));
Regards
Geet
_______________________________________________
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest
-
Re: [XOM-interest] Processing XML file of 4GB,
Geet Gangwar, 05/02/2012
- Re: [XOM-interest] Processing XML file of 4GB, Christophe Marchand, 05/02/2012
- Re: [XOM-interest] Processing XML file of 4GB, Michael Kay, 05/02/2012
Archive powered by MHonArc 2.6.24.