Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] XOM-interest Digest, Vol 29, Issue 17

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: "Warwick Bailey" <warwick AT icodeon.com>
  • To: <xom-interest AT lists.ibiblio.org>
  • Subject: Re: [XOM-interest] XOM-interest Digest, Vol 29, Issue 17
  • Date: Thu, 30 Jun 2005 07:30:51 +0100





Warwick Bailey
------------------------------------------------------
Simple. Sequencing. Now.

Icodeon Ltd e: sales AT icodeon.com
Studio 471 t: +44 (0)870 950 6582
48 Regent Street w: www.icodeon.com
Cambridge CB2 1FD
England

Registered Company in England and Wales: 5068195
VAT Registered: 851 6418 24

-----Original Message-----
From: xom-interest-bounces AT lists.ibiblio.org
[mailto:xom-interest-bounces AT lists.ibiblio.org] On Behalf Of
xom-interest-request AT lists.ibiblio.org
Sent: 30 June 2005 00:33
To: xom-interest AT lists.ibiblio.org
Subject: XOM-interest Digest, Vol 29, Issue 17

Send XOM-interest mailing list submissions to
xom-interest AT lists.ibiblio.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.ibiblio.org/mailman/listinfo/xom-interest
or, via email, send a message with subject or body 'help' to
xom-interest-request AT lists.ibiblio.org

You can reach the person managing the list at
xom-interest-owner AT lists.ibiblio.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of XOM-interest digest..."


Today's Topics:

1. Re: Serializer performance patches (Elliotte Harold)
2. Re: Serializer performance patches (Elliotte Harold)
3. Re: Serializer performance patches (Wolfgang Hoschek)
4. Re: Serializer performance patches (Wolfgang Hoschek)
5. Re: Serializer performance patches (Elliotte Harold)
6. Re: Serializer performance patches (Wolfgang Hoschek)
7. Re: Serializer performance patches (Robert Koberg)
8. Re: Serializer performance patches (Wolfgang Hoschek)
9. Re: Serializer performance patches (Michael Kay)
10. Re: Serializer performance patches (Wolfgang Hoschek)


----------------------------------------------------------------------

Message: 1
Date: Wed, 29 Jun 2005 16:06:14 -0400
From: Elliotte Harold <elharo AT metalab.unc.edu>
Subject: Re: [XOM-interest] Serializer performance patches
To: Wolfgang Hoschek <whoschek AT lbl.gov>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <42C2FF36.9030000 AT metalab.unc.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

My initial tests show some definite improvement here. I see almost a
factor of two speed-up.

However, this breaks 11 separate unit tests. There's quite a bit of
overhead involved in tracking indentation and line breaks and such, and
this patch may be getting its speed up by avoiding some of that. I need
to investigate further.


--
Elliotte Rusty Harold elharo AT metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim


------------------------------

Message: 2
Date: Wed, 29 Jun 2005 16:13:27 -0400
From: Elliotte Harold <elharo AT metalab.unc.edu>
Subject: Re: [XOM-interest] Serializer performance patches
To: Wolfgang Hoschek <whoschek AT lbl.gov>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <42C300E7.3070107 AT metalab.unc.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Testing the various patches individually the biggest win seems to be in
the change to writeMarkup, the second biggest (surprisingly) in
writeAttributeValue. writePCDATA doesn't seem to make a big difference.
That probably reflects the document I'm testing with at the moment,
though, which has more than an average amount of data in attributes. I
need to try a few others.

--
Elliotte Rusty Harold elharo AT metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim


------------------------------

Message: 3
Date: Wed, 29 Jun 2005 13:21:26 -0700
From: Wolfgang Hoschek <whoschek AT lbl.gov>
Subject: Re: [XOM-interest] Serializer performance patches
To: Elliotte Harold <elharo AT metalab.unc.edu>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <E124D52A-B88A-49C2-B2CE-377168A363FE AT lbl.gov>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

I'm measuring various doc flavours, in particular the ones in the
samples/data/ dir of the nux download. On some docs, writePCDATA is
also substantial.

It's possible that there's a bug in the indentation fix. The
string.substring() memory overlay principle is sound, though.
Actually, even on JDK 1.4 the old version made use allocations and
copies for deeply nested docs. Consider that as soon as you do
StringBuffer.add(), the shared backing array is lost, and allocs and
mem copies are done, even in JDK 1.4...

Wolfgang.

On Jun 29, 2005, at 1:13 PM, Elliotte Harold wrote:

> Testing the various patches individually the biggest win seems to
> be in the change to writeMarkup, the second biggest (surprisingly)
> in writeAttributeValue. writePCDATA doesn't seem to make a big
> difference. That probably reflects the document I'm testing with at
> the moment, though, which has more than an average amount of data
> in attributes. I need to try a few others.
>
> --
> Elliotte Rusty Harold elharo AT metalab.unc.edu
> XML in a Nutshell 3rd Edition Just Published!
> http://www.cafeconleche.org/books/xian3/
> http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/
> ref=nosim
>



------------------------------

Message: 4
Date: Wed, 29 Jun 2005 13:28:02 -0700
From: Wolfgang Hoschek <whoschek AT lbl.gov>
Subject: Re: [XOM-interest] Serializer performance patches
To: Elliotte Harold <elharo AT metalab.unc.edu>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <C7BFB3A6-30F7-42F4-BD79-931C7FBC4562 AT lbl.gov>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

On Jun 29, 2005, at 1:13 PM, Elliotte Harold wrote:

> Testing the various patches individually the biggest win seems to
> be in the change to writeMarkup, the second biggest (surprisingly)
> in writeAttributeValue. writePCDATA doesn't seem to make a big
> difference.

I measured them individually and finally in combination, in the
following order:

0. write(Element) flush issue
1. writeMarkup
2. writeAttributeValue
3. writePCData
4. indentation for indent=0 and indent=4

It might help you to repeat that order, if you like...



------------------------------

Message: 5
Date: Wed, 29 Jun 2005 16:30:51 -0400
From: Elliotte Harold <elharo AT metalab.unc.edu>
Subject: Re: [XOM-interest] Serializer performance patches
To: Wolfgang Hoschek <whoschek AT lbl.gov>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <42C304FB.7090603 AT metalab.unc.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

After investigating this further, I'm fairly convinced that the
performance gain Wolfgang found is real, but I don't think my initial
hypothesis as to where that came from is correct. The "slow path" is
doing more than simply looking for characters that need to be escaped.
It's also counting characters for purposes of line breaks and indents
and so forth; and I think most of the speed gain in this proposal comes
from avoiding that. The failing unit tests post-patch all seem to
indicate places where the the fast path failed to update fields the slow
path would have caught.

I've had a FastSerializer class in my pantry (experimental code I don't
necessarily make public) for a while now, and it shows numbers on my
system that are close to what this patch from Wolfgang provides; not
quite as fast but within 20%, and definitely faster than the usual
Serializer. FastSerializer is not a subclass of Serializer. It's a
complete reimplementation that only does UTF-8 and provides no line
breaking or indenting or other pretty-printing features. Possibly
something like this needs to be published, but don't think it's a
replacement for the standard Serializer class.

There is still about a 20% improvement from Wolfgang's work beyond what
my own FastSerializer does. I'm not sure where that's coming from. I
need to explore the differences between the two classes a little more to
see what he may have done that I didn't.

However, whichever way we go, I don't think this patch can work with the
existing Serializer. It breaks the pretty printing too badly. What it
makes clear though is just how much pretty printing (even the
possibility of it, whether you do it or not) costs in serialization
time. It might be worth releasing a much simpler, non-configurable
serializer.

--
Elliotte Rusty Harold elharo AT metalab.unc.edu
XML in a Nutshell 3rd Edition Just Published!
http://www.cafeconleche.org/books/xian3/
http://www.amazon.com/exec/obidos/ISBN=0596007647/cafeaulaitA/ref=nosim


------------------------------

Message: 6
Date: Wed, 29 Jun 2005 13:47:30 -0700
From: Wolfgang Hoschek <whoschek AT lbl.gov>
Subject: Re: [XOM-interest] Serializer performance patches
To: Elliotte Harold <elharo AT metalab.unc.edu>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <1F7278F1-1DD7-4C02-BFA2-1D78DC59EFC5 AT lbl.gov>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

On Jun 29, 2005, at 1:30 PM, Elliotte Harold wrote:

> After investigating this further, I'm fairly convinced that the
> performance gain Wolfgang found is real, but I don't think my
> initial hypothesis as to where that came from is correct. The "slow
> path" is doing more than simply looking for characters that need to
> be escaped. It's also counting characters for purposes of line
> breaks and indents and so forth;

I'm not sure I can follow. As far as I can see, the patch is only
taking the fast path when no chars need to be counted for line breaks
and indents and so forth. When there's something to count it takes
the slow path. But that is a very uncommon case. Can you please clarify?

Wolfgang.




------------------------------

Message: 7
Date: Wed, 29 Jun 2005 17:41:10 -0400
From: Robert Koberg <rob AT koberg.com>
Subject: Re: [XOM-interest] Serializer performance patches
To: Elliotte Harold <elharo AT metalab.unc.edu>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <42C31576.80003 AT koberg.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Is Wolfgang a committer for XOM?

If not, shouldn't he be?

Are there other commiters to XOM than ERH? (especially after the latest
saxon post to his blog (which I agree with!))

best,
-Rob


Elliotte Harold wrote:
> After investigating this further, I'm fairly convinced that the
> performance gain Wolfgang found is real, but I don't think my initial
> hypothesis as to where that came from is correct. The "slow path" is
> doing more than simply looking for characters that need to be escaped.
> It's also counting characters for purposes of line breaks and indents
> and so forth; and I think most of the speed gain in this proposal comes
> from avoiding that. The failing unit tests post-patch all seem to
> indicate places where the the fast path failed to update fields the slow
> path would have caught.
>
> I've had a FastSerializer class in my pantry (experimental code I don't
> necessarily make public) for a while now, and it shows numbers on my
> system that are close to what this patch from Wolfgang provides; not
> quite as fast but within 20%, and definitely faster than the usual
> Serializer. FastSerializer is not a subclass of Serializer. It's a
> complete reimplementation that only does UTF-8 and provides no line
> breaking or indenting or other pretty-printing features. Possibly
> something like this needs to be published, but don't think it's a
> replacement for the standard Serializer class.
>
> There is still about a 20% improvement from Wolfgang's work beyond what
> my own FastSerializer does. I'm not sure where that's coming from. I
> need to explore the differences between the two classes a little more to
> see what he may have done that I didn't.
>
> However, whichever way we go, I don't think this patch can work with the
> existing Serializer. It breaks the pretty printing too badly. What it
> makes clear though is just how much pretty printing (even the
> possibility of it, whether you do it or not) costs in serialization
> time. It might be worth releasing a much simpler, non-configurable
> serializer.
>



------------------------------

Message: 8
Date: Wed, 29 Jun 2005 14:54:13 -0700
From: Wolfgang Hoschek <whoschek AT lbl.gov>
Subject: Re: [XOM-interest] Serializer performance patches
To: elliotte Harold <elharo AT metalab.unc.edu>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <7AE7F773-C2C6-4AB1-B028-69F4DAF151C0 AT lbl.gov>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

Ah, I can now see the issue. Here is a revised patch that retains
essentially the same performance I've mentioned before. It runs
through all xom-1.0 tests (yes, I'm sticking with 1.0 for the time
being).

final void writePCDATA(String s) throws IOException {

if (normalize) {
s = Normalizer.normalize(s, Normalizer.NFC);
}
// if (isUnicodeWriter && !mightNeedPCDATAEscaping(s)) { // WH
int columns;
if (isUnicodeWriter && (columns = mightNeedPCDATAEscaping(s))
>= 0) { // WH
out.write(s);
if (columns > 0) {
column += columns;
lastCharacterWasSpace = false;
skipFollowingLinefeed = false;
justBroke=false;
}
return;
}
int length = s.length();
for (int i=0; i < length; i++) {
writePCDATA(s.charAt(i));
}

}

final private static int mightNeedPCDATAEscaping(String s) { // WH
int columns = 0;
int length = s.length();
for (int i = 0; i < length; i++) {
char c = s.charAt(i);
if (c < 0xd800 || c > 0xDBFF) columns++;
switch (c) {
case '&': return -1;
case '<': return -1;
case '>': return -1;
case '\r':return -1;

case '\t': return -1;
case '\n': return -1;
case ' ': return -1;
}
}
return columns;
}




final void writeAttributeValue(String s)
throws IOException {

if (normalize) {
s = Normalizer.normalize(s, Normalizer.NFC);
}
// if (isUnicodeWriter && !mightNeedAttributeEscaping(s)) { // WH
int columns;
if (isUnicodeWriter && (columns = mightNeedAttributeEscaping
(s)) >= 0) { // WH
out.write(s);
if (columns > 0) {
column += columns;
lastCharacterWasSpace = false;
skipFollowingLinefeed = false;
justBroke=false;
}
return;
}
int length = s.length();
for (int i=0; i < length; i++) {
writeAttributeValue(s.charAt(i));
}

}

private static final int mightNeedAttributeEscaping(String s) { // WH
int columns = 0;
int length = s.length();
for (int i = 0; i < length; i++) {
char c = s.charAt(i);
if (c < 0xd800 || c > 0xDBFF) columns++;
switch (c) {
case '\t': return -1;
case '\n': return -1;
case '"': return -1;
case '\r': return -1;
case '&': return -1;
case '<': return -1;
case '>': return -1;

case ' ': return -1;
}
}
return columns;
}


final void writeMarkup(String s) throws IOException {

if (normalize) {
s = Normalizer.normalize(s, Normalizer.NFC);
}
int columns;
if (isUnicodeWriter && (columns = mightNeedMarkupEscaping(s))
>= 0) { // WH
out.write(s);
if (columns > 0) {
column += columns;
lastCharacterWasSpace = false;
skipFollowingLinefeed = false;
justBroke=false;
}
return;
}
int length = s.length();
for (int i=0; i < length; i++) {
writeMarkup(s.charAt(i));
}

}

final private static int mightNeedMarkupEscaping(String s) { // WH
int columns = 0;
int length = s.length();
for (int i = 0; i < length; i++) {
char c = s.charAt(i);
if (c < 0xd800 || c > 0xDBFF) columns++;
switch (c) {
case '\t': return -1;
case '\n': return -1;
case ' ': return -1;
}
}
return columns;
}






------------------------------

Message: 9
Date: Wed, 29 Jun 2005 23:36:25 +0100
From: "Michael Kay" <mike AT saxonica.com>
Subject: Re: [XOM-interest] Serializer performance patches
To: "'Robert Koberg'" <rob AT koberg.com>, "'Elliotte Harold'"
<elharo AT metalab.unc.edu>
Cc: xom-interest AT lists.ibiblio.org
Message-ID: <20050629223635.7B6D84C00B AT lists.ibiblio.org>
Content-Type: text/plain; charset="us-ascii"


> Are there other commiters to XOM than ERH? (especially after
> the latest
> saxon post to his blog (which I agree with!))

Reference please?

Michael Kay
http://www.saxonica.com/




------------------------------

Message: 10
Date: Wed, 29 Jun 2005 16:33:21 -0700
From: Wolfgang Hoschek <whoschek AT lbl.gov>
Subject: Re: [XOM-interest] Serializer performance patches
To: xom-interest AT lists.ibiblio.org
Message-ID: <6608B713-14B7-4F1B-9F41-B9A7742B594E AT lbl.gov>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed

>
> Are there other commiters to XOM than ERH? (especially after the
> latest saxon post to his blog (which I agree with!))
>
> best,
> -Rob
>
>

The Saxon code base may not win beauty contests but IMO it's by far
the most standards compliant, mature and efficient product in that
area. In practise, it get's the job done better than anything else
out there. In a crowded place, there's rather limited value in (re)
implementing some spec for the 100th time, this time just a little
better or cleaner. Saxon is a race horse wrt. evolution and
development approach, not a turtle for dummies. Wrt. the ill-advised
unit test craze: Writing unit tests with exhaustive coverage for
every line of code may work well for small toys or projects with many
committers, but would be a complete waste of time for the Saxon one
man show. While early interactive experimental tests and automated
integration tests (of which Saxon has a gazillion) do have a useful
cost/benefit ratio, fine grained 100% unit test coverage mania is ill
advised. It would take *years* to write and maintain such tests,
while interesting development and evolution would almost completely
stagnate. Saxon is solid and at the same time far ahead leading the
pack, getting the priorities right. Just my take.

P.S.:
Some of us learned programming on historic batch mainfraimes where a
submitted hello world job would take one hour to complete, only to
spit out a laconic "syntax error" on some printer in the computer
center on the other side of campus. For most folks, this learning
experience led to thinking hard about a program before doing
anything, a style of programming that produces programs that are
mostly bug free to begin with. Adding tests as a particular need
arises seems sensible, thinking before testing is better, and
refraining from mostly clueless arguments is best. Ah well, enjoy...

Wolfgang.



------------------------------

_______________________________________________
XOM-interest mailing list
XOM-interest AT lists.ibiblio.org
http://lists.ibiblio.org/mailman/listinfo/xom-interest


End of XOM-interest Digest, Vol 29, Issue 17
********************************************





  • Re: [XOM-interest] XOM-interest Digest, Vol 29, Issue 17, Warwick Bailey, 06/30/2005

Archive powered by MHonArc 2.6.24.

Top of Page