Skip to Content.
Sympa Menu

sm-discuss - Re: [SM-Discuss] Spell quality checking

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: David Kowis <dkowis AT shlrm.org>
  • To: SM-Discuss <sm-discuss AT lists.ibiblio.org>
  • Subject: Re: [SM-Discuss] Spell quality checking
  • Date: Wed, 30 Sep 2009 20:36:30 -0500

Andraž 'ruskie' Levstik wrote:
> I believe most of these occur due to our current policy of
> "if it casts commit"
>
> I would prefer if we extend this policy with:
> "if it runs commit"

I believe this is a good thing too, however we need to define a
methodology for determining, preferably automatically, what "running" is.

>
> This should generally catch most problems. I'm actually amazed I've hit
> bugs with building things as well. People seem to not be aware there are
> things such as: force_depends and sub depends and similar.

That is an issue with documentation as another has mentioned, and I'll
tackle that one in reply to that email.

>
> My proposed ideas are:
>
> a) cast test - this should also mean cast with older versions of
> libs - as in don't cast the lib first then the app - this way it
> ensures no force_depends is needed
> This should generally be easy, update the app before the lib.
>
> b) cast from a clean install - this would ensure there are no missing
> depends - this could probably be automated by prometheus, there are
> alternatives to finding missing depends of course. Some are even in
> guru tools.
>
> c) runs - does the built thing run, does it work as advertised
> i.e.: does vim edit files, save them, open them
> This would need to be defined per spell, maybe in a new spell file.
> Just launching might miss something critical. If parts can be
> automated it would cut down on time needed. But I think delaying an
> update for a few hours so that one can use the app/lib for a while
> would certainly be beneficial in the long run.

Whilst these are good ideas, they don't provide anything repeatable. How
do I know that it works? How can you prove it to me?

"I ran it on my box for weeks."
"Well, on my box, it doesn't run at all."

Some kind of automated testing script to verify libraries, verify
functionality, verify installation of a binary, or verify compliation of
a feature is needed. The reason I think it needs to be automated: We
don't have enough time to do it all. We've got lots of people (at least
two) that have dropped off recently because there's too much work to do.
It takes too much time to update spells. Any testing methods we come up
with need to be automated so that we can say "Go test!" and it happens,
and we get results. For the programmers out there, think unit tests. We
pick functionality we want to test for spells, and we write unit tests
for it. We may not cover it all on the first go, we may not cover it on
the second go. But we'll cover some, and that's better than what we've
got right now.

>
> What do all these 3 examples have in common?
> a) I run the app for a while(especially if it's a daemon or a more
> complex one. In case of complexity I'll try to use as many functions
> as I can in some time frame).
> b) I read the ChangeLog/commits
> c) I check where applicable for new configure options, new dependencies
> specific versions of them and so on.

B and C should be part of any update, although I'm sure I'm guilty of
not doing it all the time. I know I'm guilty of deliberately ignoring
some of the new optional dependencies with bugzilla, because building
perl module spells is a PITA. A is still not a good way of testing,
because we can't reproduce the test. We can't determine that we missed
something this time and need to write a better test to catch it the next
time.

> I would prefer if our spells offered various options that upstream
> provides(e.g.: kdeedu4 - marble can be built without KDE - only needs
> Qt) not just what is used most, or be scared of query spam etc.
>
> Query spam can be easily managed - query: Do you want to configure some
> more advanced options or just stick to defaults? y|N
>

I don't think this is the appropriate solution, I think the appropriate
solution is to be able to cast a config-set. I would love to be able to
store an option set and reuse it again later. Better yet, it'd be great
to be able to create an option set and share it with others. "Dave's KDE
Konfig" etc.

If you couldn't tell, I'm big on automating this testing process.
Anything we can automate, even if it takes a bit more work initially,
saves us time in the long run. And it makes it verifyable and
repeatable. If we really want to improve the quality, we need to do
this. Then we can easily catch regressions in versions due to dependency
changes and other such evil things.

I don't think a QA file in the grimoire is the right solution. We will
add significant levels of data to the grimoire that end users don't
need. Unless, as part of the release process, we remove them from the
stable grimoire. I think having a QA grimoire (or some other repository
for the scripts) is the solution. I don't have an RFC or anything for
how to handle the testing and the verifying yet either. I've been busy
making something to make my perl-cpan spells (automatically even ;) ).

--
David Kowis
SourceMage GNU/Linux -- www.sourcemage.org

Attachment: signature.asc
Description: OpenPGP digital signature




Archive powered by MHonArc 2.6.24.

Top of Page