Public SourceMage Discussion List

[Seth Alan Woolley <seth AT positivism.org>]

Excellent crowd-pleasing speech follows:

Source Magers!  Lend me your eyeballs!

Everybody here knows we've got an excellent package manager that's undergone 
years of evolution, redesign, and use in the field.

Everybody here knows that our grimoire team is excellent at keeping 
programs up to date.

Everybody here knows that we don't advertise ourselves and thus don't 
have as many users as the big guys.

Everybody here knows that the big guys have many eyes and the corporate 
guys have many resources to buy expensive testing farms for QA.

Few may not know that there are two reasons why I find it difficult to 
use Source Mage: stable grimoire instability through a lack of testing 
and the lack of frequent ISO releases.  Being able to trust the stable 
grimoire to be well-tested and to trust that the ISO puts out a bootable 
system in many configurations _even after updating the grimoire_ are our 
two most _visible_ efforts.  The ISO update issue ties in well with the 
stable grimoire issue since many of the issues come from using an old 
ISO and then being upgraded to radically new basesystem programs.  More 
frequent ISO updating with automated and consistent build scripts that 
people can peer review is essential to this effort.  Perhaps the ISO can 
be regenerated with each stable release.

So as your QA team leader, first priority is the stable grimoire.  The 
stable grimoire is tested (in test grimoire), but not for corner cases.  
That last 5% is crucial.  If we can get it down to 1% (made up numbers 
of course), our users' administration time fixing problems is cut by 
80%.

Though we don't have the resources of the big guys, we do have some 
resources, although it may not seem like it.  Our resources are 
decentralized machines that spend very little time actually compiling.  
I call these machines: "your computer".

A few years ago Andrew of sorcery fame wrote a tool called prometheus 
whose intent was to test the grimoire over and over and over and report 
bugs automatically.  It was written, and it was good.  It has since been 
left to bit-rot and doesn't work quite perfectly with sorcery anymore.

This is about to change.  Since stepping down as Security Team Leader 
and hopefully becoming your QA Team Leader, I intend to take prometheus, 
relight his fire, and enhance him to work again.  I intend to run 
specified prometheus runs at regular intervals on the stable grimoire 
before it is released (as an upgrade from the previous stable and to 
itself), and to test all non-security-related updates (where Thomas is 
responsible for them) if possible.  I intend to start a team to build a 
website that tracks our grimiore quality (I'd especially like volunteers 
for this part to help distribute the load) and keeps track of where we 
need to work on updating the grimoire.  I've done this for build_api, 
for team member credits, and at one point for the md5 checking, which 
used to be really broken (although since source code auditing was 
integrated into sorcery, I took that down from the web -- actually lost 
a harddrive in a crash and didn't find time to but the backups back up).  
As examples of some QA-related scripting I have some stuff I've done on 
the side:

smgl-credits, sums history bullets per contributor
http://smgl.positivism.org:8080/files/smgl-credits

build_api statistics:
http://smgl.positivism.org:8080/files/searchedapi
http://smgl.positivism.org:8080/files/searchedapistat
http://smgl.positivism.org:8080/files/searchedapistattotal

md5 auditing:
gaze checkmd5s

historyfix script:
http://perforce.smee.org:8080/@@//sgl/misc/scripts/historyfix

duplicate dependency searching script:
(fixing these makes dependency resolution faster)
http://perforce.smee.org:8080/@@//sgl/misc/scripts/searchdupes.sh
http://smgl.positivism.org:8080/files/searchdupes
http://smgl.positivism.org:8080/files/searched <-- results

(I have other credits such as tracking and committing hundreds of 
security updates and improved libunpack code in sorcery and other bug 
fixes, but these aren't entirely QA team related.)

Second, as QA team leader, I'd be responsible to lead an effort to smoke 
check a lot of our existing scripting.  Since Paul/dufflebunk has begun 
work on unit testing for sorcery, he's found a number of interesting 
corner-cases and fixed problems that nobody has yet run into.  This is 
kind of like prometheus for sorcery.  This will help make our stable 
releases rock-hard stable.  We already rarely have major issues with 
sorcery, especially since Andrew's been in charge and compared to the 
mess that is gentoo python scripting with their scripts that have a 
penchant for backtracing (I'm sure all you ex-gentooers ;) know what I'm 
talking about here).

Third, as QA team leader, I'd be responsible for making sure sorcery and 
the grimoire work well together.  When sorcery's API changes, we need to 
make sure that the grimoires are updated and gurus are made aware of any 
outstanding issues in their spells' use of the sorcery API.  We need to 
make processes that let the stable grimoire update process know about 
not only WIP issues but the update process has to be 
update-dependency-aware.  When we integrate to stable we need to know 
what fixes in devel need other spells in devel when integrated.  Since 
sorcery is not designed to handle these kinds of dependency issues, we 
need to provide them in the update process until such time as sorcery 
can allow spells to be self-describing in this manner.  I'd like to 
perhaps have a file that gurus can write, similar to WIP, that contains 
keywords like: "UPDATE-WITH apr" or "UPDATE-WHEN sorcery=1.13.* and 
gcc=3.4.*".  That way the stable update process can be less error prone 
and prometheus will have less work to do to find integration issues.  
We also need to have versioned FUNCTIONS files as well.

Fourth, as QA team leader, I'd be responsible for unit testing the ISO 
creation process.  Right now the ISO creation process is rife with 
uncertainty.  The various ISO build methods don't put out consistent 
installs, and while this is nice from a perspective that it means 
sorcery and spells have to be especially vigilant, we need to have more 
stability in this area if we are going to get anywhere.  I applaud the 
recent work on revolutionizing our ISO build processes to break free 
from the shackles our old processes created and hope we can have a 
consistent build process for ISO creation.  I also applaud the effort 
past ISO-makers such as Benoit and Low have done to at least give us 
something to showcase our work with.  They have added a number of 
features that are not trivial features to the ISO and thus I thank them 
again.  But, when things start to settle down, we need to make this more 
consistently tested.

These are just some of the ideas and directions the QA Team can work on, 
and you know I won't go away since I've been here pretty much since the 
beginning.  :)

So, I end with the appeal, of course, like they say: please vote for me!

(And volunteer for the team if you're not a developer yet. ;) )

Seth

On Mon, Apr 18, 2005 at 02:42:11PM -0700, Seth Alan Woolley wrote:
> Here is where I give a little motivational speech:
> 
> <insert excellent crowd-pleasing speech here>
> 
> Oh and don't forget to vote.  Failing to vote counts as a no vote in our 
> strange voting system (kinda like school levy bonds where I grew up).
> 
> Actually, I would give a real speech but I'm at work and my time is 
> limited.
> 
> Seth

-- 
Seth Alan Woolley [seth at positivism.org], SPAM/UCE is unauthorized
Key id 00BA3AF3 = 8BE0 A72E A47E A92A 0737  F2FF 7A3F 6D3C 00BA 3AF3
Security Team Member Source Mage GNU/Linux http://www.sourcemage.org
Elected Coordinating Committee Member, Pacific Green Party of Oregon

Attachment: pgpUvBXjckWNX.pgp
Description: PGP signature