XOM API for Processing XML with Java

[Nils_Kilden-Pedersen AT Countrywide.Com]

xom-interest-bounces AT lists.ibiblio.org wrote on 02/02/2005 04:29:47 AM:

> The question is perhaps what counts as a large document. I'd say it's a 
> document that can't fit in available memory. The less memory XOM uses, 
> the fewer documents count as large. They're some interesting ideas here, 

> and I think they're worth exploring.

But how will you know in advance if the document will fit in memory? I 
think Wolfgang's got a good point here. You don't want to find out by an 
OutOfMemoryException, so if you're going to process "large" documents, you 
need a different strategy anyway.
The scenarios I've seen are either static "small" documents, or dynamic 
(growing) "large" documents.

Nils
>From elharo AT metalab.unc.edu  Wed Feb  2 12:25:33 2005
Return-Path: <elharo AT metalab.unc.edu>
X-Original-To: XOM-interest AT lists.ibiblio.org
Delivered-To: XOM-interest AT lists.ibiblio.org
Received: from mail22.sea5.speakeasy.net (mail22.sea5.speakeasy.net
        [69.17.117.24])
        by lists.ibiblio.org (Postfix) with ESMTP id C95E24C008
        for <XOM-interest AT lists.ibiblio.org>;
        Wed,  2 Feb 2005 12:25:32 -0500 (EST)
Received: (qmail 5390 invoked from network); 2 Feb 2005 17:25:32 -0000
Received: from dsl254-067-087.nyc1.dsl.speakeasy.net (HELO [192.168.254.100])
        (elharo@[216.254.67.87]) (envelope-sender <elharo AT metalab.unc.edu>)
        by mail22.sea5.speakeasy.net (qmail-ldap-1.03) with AES256-SHA
        encrypted SMTP
        for <Nils_Kilden-Pedersen AT Countrywide.Com>; 2 Feb 2005 17:25:31 -0000
Message-ID: <42010D0A.9040008 AT metalab.unc.edu>
Date: Wed, 02 Feb 2005 12:25:30 -0500
From: Elliotte Harold <elharo AT metalab.unc.edu>
User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US;
        rv:1.7.5) Gecko/20041217
X-Accept-Language: en-us, en, fr
MIME-Version: 1.0
To: Nils_Kilden-Pedersen AT Countrywide.Com
Subject: Re: [XOM-interest] indexOf O(1) patch?
References: 
<OF675D11FF.BD5D0330-ON88256F9C.005B1328-88256F9C.005B6B22 AT countrywide.com>
In-Reply-To: 
<OF675D11FF.BD5D0330-ON88256F9C.005B1328-88256F9C.005B6B22 AT countrywide.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: XOM-interest AT lists.ibiblio.org
X-BeenThere: xom-interest AT lists.ibiblio.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: xom-interest.lists.ibiblio.org
List-Unsubscribe: <http://lists.ibiblio.org/mailman/listinfo/xom-interest>,
        <mailto:xom-interest-request AT lists.ibiblio.org?subject=unsubscribe>
List-Archive: <https://lists.ibiblio.org/sympa/arc/xom-interest>
List-Post: <mailto:xom-interest AT lists.ibiblio.org>
List-Help: <mailto:sympa AT lists.ibiblio.org?subject=HELP>
List-Subscribe: <http://lists.ibiblio.org/mailman/listinfo/xom-interest>,
        <mailto:xom-interest-request AT lists.ibiblio.org?subject=subscribe>
X-List-Received-Date: Wed, 02 Feb 2005 17:25:33 -0000

Nils_Kilden-Pedersen AT Countrywide.Com wrote:


> But how will you know in advance if the document will fit in memory? 

Experiment. The fact is I constantly see people having trouble 
processing XML due to the size of their documents. I see this far more 
often than I see people having problems with speed.

Less than two minutes ago I was reading a thread about this on the saxon 
mailing list, where someone needed to process an 80MB document on a 
512MB machine. What's big is all relative to the installed memory and 
similar considerations. However, I think it's a big win for XOM to be 
able to process documents as large as it possibly can. The fact that 
there will always be larger documents than can comfortably be handled in 
memory does not convince me that XOM shouldn't do the best job it can.

This is yet another example of the common binary fallacy. Here the 
argument takes the form that XOM can't be perfect so it shouldn't try to 
be better. The fact is, XOM can be better; and it's worth making it 
better. Perhaps we'll eventually reach a point where the cost of making 
XOM documents smaller outweighs the gains in so doing, but I don't think 
XOM has reached that point yet. That Wolfgang can use an alternative 
builder and come up with documents that are a third smaller than what 
the standard builder produces tells me there's still a lot of room for 
improvement.

-- 
Elliotte Rusty Harold  elharo AT metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim