XOM API for Processing XML with Java

[Wolfgang Hoschek <whoschek AT lbl.gov>]

On Jun 30, 2005, at 5:46 AM, Elliotte Harold wrote:

> Wolfgang Hoschek wrote:
>
>
> > Some of us learned programming on historic batch mainfraimes where  
> > a  submitted hello world job would take one hour to complete, only  
> > to  spit out a laconic "syntax error" on some printer in the  
> > computer  center on the other side of campus. For most folks, this  
> > learning  experience led to thinking hard about a program before  
> > doing  anything, a style of programming that produces programs  
> > that are  mostly bug free to begin with. Adding tests as a  
> > particular need  arises seems sensible, thinking before testing is  
> > better, and  refraining from mostly clueless arguments is best. Ah  
> > well, enjoy...
>
> Just a thought, but maybe the software development practices that  
> worked best in an era of 1 MHz batch processing and punch cards  
> aren't necessarily the same as the ones that work well for 2 GHz  
> computers with GUIs and acres of RAM?

Surely the practices are completely different nowadays. For example,  
rapid interactive prototyping inside an IDE with experimental test  
snippets inside an IDE can be very productive, as is quick IDE  
refactoring of entire subsystems, etc. Phantastic! But that doesn't  
change the fact that experience and lessons learned from history  
remain useful.

> Or maybe those 1970s era techniques never worked so well in the  
> first place. It was just the best that could be done with the  
> available tools?

Right, it was just the best that could be done under the  
circumstances. But even lessons learned the hard way are valuable  
lessons. Code that "just works" is cheapest.

> I believe such programs are mostly bug free to begin with about as  
> much as I believe it really was the tooth fairy leaving all those  
> quarters under my pillow.
>
> Your software development style seems to be very theoretical. That  
> probably made sense in 1975, but in 2005 it's a lot quicker and  
> easier to try three different approaches and measure which one  
> works better than it is to carefully consider the details of the  
> algorithm.

I'm not sure what you're trying to say. I'm always combining historic  
experience, theory, practical experiment, and an open mind for future  
change. None of this is mutually exclusive.

> You've complained about some of the changes I've made in 1.1 that  
> are hurting performance. It's not a coincidence that performance is  
> the one area that's really not tested at all by XOM's unit test  
> suite. Since performance tests still have have to be run manually,  
> they tend not to get run. Consequently things tend to break. If  
> performance testing could be integrated directly into XOM's  
> automated test suite, then I could be a lot more confident about  
> applying patches designed to fix and improve  performance.

As has been pointed out by others, the cost/benefit ratio of  
automated performance regression tests is poor, unfortunately. There  
are just too many variables that need careful consideration in order  
to reach meaningful interpretations. So automation doesn't seem to  
work well; more than a semi-automatic approach doesn't seem feasible.  
At least that's what I found. If a better realistic solution can be  
found, all the better.

> Perhaps in small 1970s Fortran programs that rarely exceeded 10,000  
> lines, it was possible to be reasonably confident in the code  
> purely from an analysis of the algorithms. But in today's much  
> larger object-oriented programs? I don't believe it's possible.  
> Even with object boundaries, data encapsulation, careful attention  
> to minimizing coupling, and more, there's still just way too much  
> interconnectedness in even good code for a human to be able to  
> carefully follow all possible paths through the code. The advantage  
> of having a test suite that exercises most of the code base on many  
> different paths is that you can be reasonably confident whether or  
> not a change to fix one issue is likely to break anything else.   
> Without a solid test suite, I'd be scared to touch anything in XOM  
> for fear of what might break.

Fine grained unit tests tend to catch the easy/cheap bugs, and tend  
to fail to detect the hard/expensive ones coming from integration  
into systems consisting of many subsystems, and layers. That is, fine  
grained tests help, but only so much.

> Your proposed performance patch is a good example of that. The test  
> suite made it immediately obvious that there were problems and it  
> couldn't be accepted as is. The test suite also showed us exactly  
> what broke and how so that you could fix the patch. Neither you nor  
> I spotted the problems with failing to escape markup from a simple  
> analysis of the code, but the test suite found it immediately. The  
> larger and more complex a system is the more important it is that  
> it be comprehensively tested. Anything less makes it impossible to  
> safely change.

It's nice to have those tests for a rather complex animal such as the  
Serializer. Again, I have no objections to testing per se. In the  
end, it's a question of goals and priorities given a finite amount of  
time/resources, and the answers depend.

Wolfgang.