Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] Serializer performance patches

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Wolfgang Hoschek <whoschek AT lbl.gov>
  • To: Elliotte Harold <elharo AT metalab.unc.edu>
  • Cc: xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] Serializer performance patches
  • Date: Thu, 30 Jun 2005 15:29:08 -0700

On Jun 30, 2005, at 5:46 AM, Elliotte Harold wrote:

Wolfgang Hoschek wrote:


Some of us learned programming on historic batch mainfraimes where a submitted hello world job would take one hour to complete, only to spit out a laconic "syntax error" on some printer in the computer center on the other side of campus. For most folks, this learning experience led to thinking hard about a program before doing anything, a style of programming that produces programs that are mostly bug free to begin with. Adding tests as a particular need arises seems sensible, thinking before testing is better, and refraining from mostly clueless arguments is best. Ah well, enjoy...


Just a thought, but maybe the software development practices that worked best in an era of 1 MHz batch processing and punch cards aren't necessarily the same as the ones that work well for 2 GHz computers with GUIs and acres of RAM?

Surely the practices are completely different nowadays. For example, rapid interactive prototyping inside an IDE with experimental test snippets inside an IDE can be very productive, as is quick IDE refactoring of entire subsystems, etc. Phantastic! But that doesn't change the fact that experience and lessons learned from history remain useful.

Or maybe those 1970s era techniques never worked so well in the first place. It was just the best that could be done with the available tools?

Right, it was just the best that could be done under the circumstances. But even lessons learned the hard way are valuable lessons. Code that "just works" is cheapest.

I believe such programs are mostly bug free to begin with about as much as I believe it really was the tooth fairy leaving all those quarters under my pillow.

Your software development style seems to be very theoretical. That probably made sense in 1975, but in 2005 it's a lot quicker and easier to try three different approaches and measure which one works better than it is to carefully consider the details of the algorithm.

I'm not sure what you're trying to say. I'm always combining historic experience, theory, practical experiment, and an open mind for future change. None of this is mutually exclusive.


You've complained about some of the changes I've made in 1.1 that are hurting performance. It's not a coincidence that performance is the one area that's really not tested at all by XOM's unit test suite. Since performance tests still have have to be run manually, they tend not to get run. Consequently things tend to break. If performance testing could be integrated directly into XOM's automated test suite, then I could be a lot more confident about applying patches designed to fix and improve performance.

As has been pointed out by others, the cost/benefit ratio of automated performance regression tests is poor, unfortunately. There are just too many variables that need careful consideration in order to reach meaningful interpretations. So automation doesn't seem to work well; more than a semi-automatic approach doesn't seem feasible. At least that's what I found. If a better realistic solution can be found, all the better.


Perhaps in small 1970s Fortran programs that rarely exceeded 10,000 lines, it was possible to be reasonably confident in the code purely from an analysis of the algorithms. But in today's much larger object-oriented programs? I don't believe it's possible. Even with object boundaries, data encapsulation, careful attention to minimizing coupling, and more, there's still just way too much interconnectedness in even good code for a human to be able to carefully follow all possible paths through the code. The advantage of having a test suite that exercises most of the code base on many different paths is that you can be reasonably confident whether or not a change to fix one issue is likely to break anything else. Without a solid test suite, I'd be scared to touch anything in XOM for fear of what might break.

Fine grained unit tests tend to catch the easy/cheap bugs, and tend to fail to detect the hard/expensive ones coming from integration into systems consisting of many subsystems, and layers. That is, fine grained tests help, but only so much.

Your proposed performance patch is a good example of that. The test suite made it immediately obvious that there were problems and it couldn't be accepted as is. The test suite also showed us exactly what broke and how so that you could fix the patch. Neither you nor I spotted the problems with failing to escape markup from a simple analysis of the code, but the test suite found it immediately. The larger and more complex a system is the more important it is that it be comprehensively tested. Anything less makes it impossible to safely change.

It's nice to have those tests for a rather complex animal such as the Serializer. Again, I have no objections to testing per se. In the end, it's a question of goals and priorities given a finite amount of time/resources, and the answers depend.

Wolfgang.




Archive powered by MHonArc 2.6.24.

Top of Page