Public SourceMage Discussion List

[seth AT swoolley.homeip.net]

Since there are quite a few bugs to fix that we can't track down easily 
but with prometheus and we have a lot of policies that are "stated but 
not enforced" I think we should make a special grimoire of misbehaving 
and good spells for prometheus to detect.

This would be in addition to a lint tester to catch stuff in the source 
directly.

It would be in its own repo and there would be these sections to start 
(for example):

grimoire-policies
prometheus-failure-handling
sorcery-failure-handling

We could put a series of spells that run "exit 1" (very bad) at various 
stages of the install and put them in sorcery-failure-handling to detect 
major problems with spells.

Also, we could fail with a regular return 1 or false at the end of the 
file as a nicer test.

And throw in various kinds of syntax errors in spell files for good 
measure.

For prometheus-failure-handling I would want things that don't make 
sorcery do bad things like bad/missing md5s/gpg checks, invalid 
dependencies/providers, bad urls, and various other things that we 
should catch.  I can then run prommie against these regression tests for 
each update.

For grimoire policies, we could test out the lint checker and pm-seer 
for various policy errors like WIP in test grimoire, build_api issues, 
history file correctness, SPELL != spellpath and things that aren't 
fatal, but not good.

A new file TEST_SPEC could have a FAILURE_STRING="regex here" indicating 
what the failure report should match if the failure has been detected 
and a SUCCESS_INSTEAD="yes" if the spell is supposed to be good.  
Prometheus can be made to look for these variables and to only report 
bugs if they are not what they are supposed to be (and to report against 
prometheus itself).  A FAIL_PRODUCT and FAIL_COMPONENT set of variables 
would tell prometheus where to file bugs against if the tests fail 
although I'll probably start with them all going to prometheus by 
default until I can trust it to spot sorcery problems.  The sorcery unit 
test regression suite can be made to run from a spell in here as well 
with a FAILURE_STRING set to detect the failure messages from its 
stdout. :)

I think I'll write the harness around it and some sample spells before 
fixing any other bugs in prometheus so I have a good debugging 
environment to work from such that it's easy to fix and test my fixes. 
:)