Skip to Content.
Sympa Menu

sm-discuss - [SM-Discuss] prometheus regression tests project

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: seth AT swoolley.homeip.net
  • To: sm-discuss AT lists.ibiblio.org
  • Subject: [SM-Discuss] prometheus regression tests project
  • Date: Sun, 23 Apr 2006 10:07:23 -0700

Since there are quite a few bugs to fix that we can't track down easily
but with prometheus and we have a lot of policies that are "stated but
not enforced" I think we should make a special grimoire of misbehaving
and good spells for prometheus to detect.

This would be in addition to a lint tester to catch stuff in the source
directly.

It would be in its own repo and there would be these sections to start
(for example):

grimoire-policies
prometheus-failure-handling
sorcery-failure-handling

We could put a series of spells that run "exit 1" (very bad) at various
stages of the install and put them in sorcery-failure-handling to detect
major problems with spells.

Also, we could fail with a regular return 1 or false at the end of the
file as a nicer test.

And throw in various kinds of syntax errors in spell files for good
measure.

For prometheus-failure-handling I would want things that don't make
sorcery do bad things like bad/missing md5s/gpg checks, invalid
dependencies/providers, bad urls, and various other things that we
should catch. I can then run prommie against these regression tests for
each update.

For grimoire policies, we could test out the lint checker and pm-seer
for various policy errors like WIP in test grimoire, build_api issues,
history file correctness, SPELL != spellpath and things that aren't
fatal, but not good.

A new file TEST_SPEC could have a FAILURE_STRING="regex here" indicating
what the failure report should match if the failure has been detected
and a SUCCESS_INSTEAD="yes" if the spell is supposed to be good.
Prometheus can be made to look for these variables and to only report
bugs if they are not what they are supposed to be (and to report against
prometheus itself). A FAIL_PRODUCT and FAIL_COMPONENT set of variables
would tell prometheus where to file bugs against if the tests fail
although I'll probably start with them all going to prometheus by
default until I can trust it to spot sorcery problems. The sorcery unit
test regression suite can be made to run from a spell in here as well
with a FAILURE_STRING set to detect the failure messages from its
stdout. :)

I think I'll write the harness around it and some sample spells before
fixing any other bugs in prometheus so I have a good debugging
environment to work from such that it's easy to fix and test my fixes.
:)



  • [SM-Discuss] prometheus regression tests project, seth, 04/23/2006

Archive powered by MHonArc 2.6.24.

Top of Page