Public SourceMage Discussion List

[Remko van der Vossen <wich AT yuugen.jp>]

On Mon, May 18, 2009 at 07:41:13AM +0200, Andraž 'ruskie' Levstik wrote:
> :2009-05-17T14:50:Mark Bainter:
> 
> > Well, my plans to be present didn't work out, but I am reading through
> > the log, and I had a couple of thoughts to share.
> > 
> > On the topic of an itworks flag, What if we extended sorcery to support
> > an additional DETAIL item that would point at either a command or a
> > script (maybe with a standardized name) which could be run during
> > automated testing, and 'itworks' gets set based on the return value from
> > running that?
> > 
> > For most packages it should be fairly simple to at least try to run the
> > primary binary, and make sure it doesn't segfault or throw library or
> > other runtime dependency errors that aren't caught by the casting itself. 
> >  
> > 
> > If there's no pointer, it's just an unknown, but a hard failure or hard
> > success at least helps to prioritize issues.  And, it shouldn't add too
> > much additional labor for most spells, and it would be sufficiently
> > flexible that packages that typically break in a given set of ways can
> > have tests added to the script for those specific concerns.
> 
> As it was said you are offloading an it works to a "runs" check which is
> something completely different.

I like the solution though, escpecially as a $SPELL/QA script like Eric
proposed. We could have a check_self as default there to check at least
something. Then for any known trouble cases we can write more involved
testing scripts.

It would indeed be the most flexible solution. I know it doesn't test
"it works" for all spells we have, but I'm affraid that is an
unrealistic target, at least to begin with. With a QA script defaulting
to check_self we have at least a defined working basic QA system on the
way to an "it works" idea. Any future improvements to check_self the QA
would automatically also benefit from. And of course we could extend
default_qa with other tests besides check_self if there are any sensible
generic tests to do.

And yes, this doesn't take away the responsibility of the grimoire
developer to check their work, they should still verify that their
updated spell actually works beyond casting it. However we know it will
not be possible for everyone to test everything and sometimes things
just slip by, having an automated test system, especially one that will
do regression testing would be very good in my opinion.

In any case I would be all for adding a QA script to the spells. The
question however is what we will do with it for now, do we set up a
dedicated QA system casting spells and running QA scripts, do we rely on
the grimoire developers to run the QA scripts, do we make it a sorcery
option to run QA after every cast so that regular users can also help?

Remko van der Vossen

Attachment: smime.p7s
Description: S/MIME cryptographic signature