Skip to Content.
Sympa Menu

sm-discuss - [SM-Discuss] ISO Build Complexities

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: flux <flux AT sourcemage.org>
  • To: SM-Discuss <sm-discuss AT lists.ibiblio.org>
  • Subject: [SM-Discuss] ISO Build Complexities
  • Date: Thu, 18 Sep 2008 21:01:43 -0400

I'm starting this under its own thread because it's clearly not about
distrowatch ;) Please don't consider this to be a rant, even if I do not
clearly communicate myself and end up sound like I am ranting. I just
want to raise awareness of what is involved in the process of generating
ISO releases.

There are a lot of factors that affect ISO development. Some of these
are apparent to everyone. Some of them are not. I'm going to try to
explain some of the finer problems that ISO development faces which I
think less people are aware of.

The current ISO test series hasn't gone to stable yet. This is not
because the installer isn't exactly ready (there are things to fill in,
but frankly I don't care since it works and people have not had serious
problems installing due to the installer). It hasn't gone to stable
because it isn't booting where it is supposed to. When I updated the ISO
kernel config to support SATA and PATA devices, all kinds of problems
cropped up. Essentially the root of all these evils is due to a poor
driver implementation in the mainline kernel. The ata_piix driver is
greedy and attaches itself to all devices it possibly can, even when it
shouldn't. To my knowledge this has not been fixed in the kernel yet.
The workaround to this was to make the drivers built as modules, and add
udev to the initrd on the ISO so that it could udev could autodetect the
hardware and load the correct modules instead of having race conditions
between the modules. This actually works, and so far works quite well.
However, this leads to another problem, which is guaranteeing that the
ISO kernel can still boot a machine without the aid of an initrd. This
is possible, but harder to pin down, and it wouldn't help to have 2
separate kernels on the ISO, since the problem is in the kernel itself.
There are instructions on how to build one's own kernel during the ISO
install, so at least users can do that until I am able to tighten down
the kernel config.

Although the 0.10.0-test3 ISO actually works rather well (SATA issues
aside), I'm not considering it stable until it boots on *all* the
hardware it is actually supposed to, without intervention from the user.
If the Project Lead suggests for me to do otherwise, then I will of
course release a new stable. Otherwise, the new stable will wait until
these kernel issues have been resolved.

This leads us to actual cauldron development. I'm not saying this to
step on anyone's toes, but cauldron development is more complicated and
precarious than either grimoire or sorcery work for the following
reasons. First, sorcery and grimoire work involves only software,
whereas cauldron work involves both software and hardware. If there is a
problem with a hardware configuration and the drivers that manage it,
this will halt ISO production just as much as having software conflicts
(like the grub bug where grub couldn't boot ext2/3 with >128 inodes and
the defaults for ext2/3 changed to 256 inodes). Second, the grimoire
depends on sorcery, and sorcery depends on the grimoire, but cauldron
depends on both. If something breaks in either sorcery or the grimoire,
it will hit cauldron. If something breaks in cauldron, sorcery and the
grimoire see nothing of it. Third, the grimoire does not necessarily (in
practice) require that all spells will build successfully from scratch.
Even end users don't require this, as typically a user will only need
this to hold true in one instance: installation. From there they can
just continually roll updates like they are supposed to be able to.
Cauldron, however, requires spells to build from scratch every time,
since each ISO is essentially a "clean slate" of spells.

These additional constraints imposed on cauldron development, coupled
with the fact that we don't have a working prometheus, result in an
interesting situation: cauldron essentially gets the role of prometheus
by default. There are a lot of problems that show up in ISO generation
that don't show up when cutting grimoire releases. For example, the grub
bug mentioned above shows up a lot more often in the ISO, since
typically users create new filesystems when they are installing, and not
often when they already have a working system and are just doing an
update. Cauldron development has even led to the detection of bugs in
the basesystem chroots, currently generated from the grimoire team. I
don't think anyone else hit those bugs before, even though advanced
users were installing from those chroots. The fact that cauldron hits
more bugs means that cauldron development is slowed even more, as those
additional bugs need to be resolved before a stable release can be made.

The generation process for the 0.10.0 series of ISOs is mainly manual.
Various separate stages can and have been automated (casting of all the
additional spells beyond basesystem that are needed, cleaning out of
files/binaries that aren't needed, etc.), but it is still a very
time-consuming process and requires a lot of human intervention due to
the dependencies and bugs referenced above. Thus, current cauldron
development is focusing on automating the process. A lot of work has
already been done on this, but it's not yet at a point where it can be
used to produce an ISO without intervention. Currently, only myself and
Karsten are working on this portion, and Karsten is currently MIA due to
schoolwork (I assume). Andrew Stitt is currently helping to automate
going from a basesystem chroot to basesystem+needed_spells_for_ISOs, but
he's hitting a lot of bugs with that (he just found a sorcery bug by
doing so). We are all there is for the cauldron team (and Andrew is not
even official yet). If people want to see more ISOs released more often,
then we will need more contributors.

One area of possible contribution can just be testing the ISOs that are
released. This means more than just seeing if it boots, or "yeah I
installed" however. We need people who will have the patience to file
bug reports, and to try to work with us to find the causes of those
problems. This is harder for cauldron, since some bugs may be hardware
dependent, and thus requires working with the person who filed the bug
in order to resolve the bug. Of course if anyone is willing to spend
time helping out with ISO generation or cauldron coding, I'd be more
than happy for the help. Just send me an email or let me know on IRC and
I'll help point you in the right directions to get you started.
--
Justin "flux_control" Boffemmyer
Cauldron wizard and general mage
Source Mage GNU/Linux
http://www.sourcemage.org

Attachment: pgpFL0_SNGO8a.pgp
Description: PGP signature




Archive powered by MHonArc 2.6.24.

Top of Page