Source Mage code commit list

[Andrew Stitt <scm AT sourcemage.org>]

GIT changes to devel-andrew cauldron by Andrew Stitt <acedit AT armory.com>:

 data/depends/ncurses.p         |    1 
 doc/release.recipe             |   76 
+++++++++++++++++++----------------------
 scripts/cleaniso.sh            |    4 +-
 scripts/spellcaster.sh         |   71 ++++++++++++++++++++++++++++++++++++++
 scripts/twelve-step-program.sh |   11 ++---
 5 files changed, 116 insertions(+), 47 deletions(-)

New commits:
commit 06901b35dab8f6368eecb1ac5a9546ec86fad612
Author: Andrew Stitt <acedit AT armory.com>
Commit: Andrew Stitt <acedit AT armory.com>

    scripts/twelve-step-program.sh: fix errors from last run
    
    fix kernel /lib/modules directory name
    copy compile_config for parallel compilation options

commit 3028ce7fd6e58d4c834afaf9c9463424ef5af095
Author: Justin Boffemmyer <flux AT sourcemage.org>
Commit: Justin Boffemmyer <flux AT sourcemage.org>

    scripts/cleaniso.sh: ignore missing files
    
    Added the force flag (-f) to rm to ignore missing files. If the file is
    already missing, then we don't need to worry about it, because we wanted
    to delete it anyway if it was there. Also removed two useless preceding
    spaces.

commit 3da36428ea25ab2c9a5aa6147599064b60991d9c
Author: Justin Boffemmyer <flux AT sourcemage.org>
Commit: Justin Boffemmyer <flux AT sourcemage.org>

    doc/release.recipe: updated recipe
    
    Updated the release recipe for the newer build processes.

commit ad8c6683ebdbbbc5026f4df592a3c84b32b2a4ec
Author: Justin Boffemmyer <flux AT sourcemage.org>
Commit: Justin Boffemmyer <flux AT sourcemage.org>

    scripts/spellcaster.sh: create depends,packages
    
    Generate the depends and packages state files for SYSDIR sorcery so that
    it can find the "installed" spells after the install.

commit 0f6fbee37ade295f45687061356dbb312e0aae5b
Author: Justin Boffemmyer <flux AT sourcemage.org>
Commit: Justin Boffemmyer <flux AT sourcemage.org>

    scripts/spellcaster.sh: install stable grimoire
    
    Install the same version of the stable grimoire used for casting in
    TARGET into SYSDIR.

commit aa87706537f6b482e97a68ec2904a8205597911f
Author: Justin Boffemmyer <flux AT sourcemage.org>
Commit: Justin Boffemmyer <flux AT sourcemage.org>

    Ensure ncurses is built with UTF8 enabled
    
    Added a "depends" file for ncurses to the repo, and added functionality
    to scripts/spellcaster.sh to install it into the TARGET to make use of
    it during casting.

diff --git a/data/depends/ncurses.p b/data/depends/ncurses.p
new file mode 100644
index 0000000..3014852
--- /dev/null
+++ b/data/depends/ncurses.p
@@ -0,0 +1 @@
+UTF8="y"
diff --git a/doc/release.recipe b/doc/release.recipe
index d1544f4..956aead 100644
--- a/doc/release.recipe
+++ b/doc/release.recipe
@@ -21,48 +21,46 @@ All cauldron ISO scripts (for the test ISOs) are located 
in the cauldron/scripts
 1.  Get a fresh basesystem tarball, unpack it into a directory, and rename 
that
     directory "build".
 
-2.  Cast all the spells that will go on the ISO (required and optional) via
-    cauldron/scripts/spellcaster.sh -c build x86. The list of standard x86
-    spells to cast for the official x86 isos can be found in the repo in
-    cauldron/data/rspells.x86 (required) and cauldron/data/ospells.x86
-    (optional).  Note: the terms "required" and "optional" here refer to what
-    is required vs.  optional for the user to install onto the target system.
-    All spells are required to be on official ISOs. Also, only the required
-    spells should be in the final system.tar.bz2, and optional spells should
-    only be present on the ISO itself.
-
-3.  build a linux kernel and modules, however you choose to do so (manually 
or
+2.  Build a linux kernel and modules, however you choose to do so (manually 
or
     by casting the linux spell). We have a kernel config that you can use in
     our git repo. (cauldron/data/config-2.6)
 
-4.  Check the build tree for sanity (make sure nothing left over from
-    basesystem is there that should not be there, for example
-    /etc/udev/rules.d/70-persistent-net.rules). Some of this will be taken 
care
-    of in steps #6 and #7 via the use of the cauldron/scripts/add-sauce.sh
-    script.
-
-5.  Make two copies of the build directory tree, call one iso and the other
-    system. Run cauldron/scripts/spellcaster.sh -d system x86 to remove all 
the
-    optional spells from the system tree (leaving only the required spells).
-
-6.  Make any necessary adjustments to the system tree. This includes running
-    cauldron/scripts/add-sauce.sh -s /path/to/system/tree and cleaning up
-    var/cache/sorcery and var/spool/sorcery (e.g. via cleanse), removing 
linux
-    source trees in /usr/src, and creating a symlink to the latest linux 
source
-    under /var/spool/sorcery e.g., ln -sf
-    /var/spool/sorcery/linux-VERSION.tar.bz2 /usr/src/ for easy access.
-
-7.  Make adjustments to the iso tree. This will be more extensive than what 
is
-    done for the system tree. First, the iso does not need a build 
environment,
-    so (most) of all the build tools can be completely wiped from the ISO. 
This
-    can be done by running cauldron/scripts/cleaniso.sh -a /path/to/iso/tree.
-    Additionally, run cauldron/scripts/add-sauce.sh -i /path/to/iso/tree.
-
-8.  Make a compressed tarball of the system tree, and copy it into
-    cauldron/iso/system.tar.bz2. This can be done easily by running
-    cauldron/scripts/mksystem.sh /path/to/system/tree 
/path/to/system.tar.bz2.
-    Don't forget to copy the tarball to the cauldron/iso tree after creating
-    it!
+3.  Cast all the spells that will go on the ISO (required and optional) via
+    cauldron/scripts/spellcaster.sh build x86. The list of standard x86 
spells
+    to cast for the official x86 isos can be found in the repo in
+    cauldron/data/rspells.x86 (required), cauldron/data/ispells.x86 (iso), 
and
+    cauldron/data/ospells.x86 (optional). Note: the terms "required" and
+    "optional" here refer to what is required vs. optional for the user to
+    install onto the target system. All spells are required to be on official
+    ISOs. Also, only the required spells should be in the final 
system.tar.bz2,
+    and optional spells should only be present on the ISO itself.
+
+4.  Check the iso and sys trees for sanity. Especially make sure that all
+    the requisite directories are present (smgl-fhs) and there is nothing
+    astray in their etc/ directories. Some of this will be taken care of in
+    steps #5 and #7 via the use of the cauldron/scripts/add-sauce.sh script.
+
+5.  Make any necessary adjustments to the system tree (/tmp/cauldron/sys). 
This
+    includes running cauldron/scripts/ add-sauce.sh -s /tmp/cauldron/sys and
+    cleaning up var/cache/sorcery and var/spool/sorcery (e.g. via cleanse),
+    removing linux source trees in /usr/src, and creating a symlink to the
+    latest linux source under /var/spool/sorcery e.g., ln -sf /
+    var/spool/sorcery/linux-VERSION.tar.bz2 /usr/src/ for easy access.
+
+6.  Prune the iso tree (/tmp/cauldron/iso). The iso does not need a build
+    environment, so (most of) the build tools can be completely wiped from 
the
+    ISO. This can be done by running cauldron/scripts/cleaniso.sh -a
+    /path/to/iso/tree.
+
+7.  Make adjustments to the iso tree (/tmp/cauldron/iso) by running
+    cauldron/scripts/add-sauce.sh -i /tmp/cauldron/iso.
+
+8.  Make a compressed tarball of the system tree, and copy it to
+    /tmp/cauldron/iso/system.tar.bz2. Creation of the tarball can be done
+    easily by running cauldron/scripts/mksystem.sh /tmp/cauldron/sys
+    /tmp/cauldron/iso/system.tar.bz2. Don't forget to copy the tarball to the
+    iso tree after creating it if you save it to a path other than the root 
of
+    the iso tree!
 
 9.  Make an initrd, and copy it to cauldron/iso/boot/initrd.gz. This can be
     done via cauldron/scripts/mkinitrd.sh /path/to/iso/tree 
iso_kernel_version
diff --git a/scripts/cleaniso.sh b/scripts/cleaniso.sh
index 94f8995..769ee27 100755
--- a/scripts/cleaniso.sh
+++ b/scripts/cleaniso.sh
@@ -100,11 +100,11 @@ do
                        then
                                # chroot and clean a directory using rmdir
                                echo "Attempting to remove directory 
$ISOCHROOT/$DIRT"
-                                chroot "$ISOCHROOT" rmdir 
--ignore-fail-on-non-empty "$DIRT"
+                               chroot "$ISOCHROOT" rmdir 
--ignore-fail-on-non-empty "$DIRT"
                        else
                                # chroot and clean an individual file
                                echo "Attempting to delete $ISOCHROOT/$DIRT"
-                                chroot "$ISOCHROOT" rm "$DIRT"
+                               chroot "$ISOCHROOT" rm -f "$DIRT"
                        fi
                fi
        done
diff --git a/scripts/spellcaster.sh b/scripts/spellcaster.sh
index 9bec322..9a99a87 100755
--- a/scripts/spellcaster.sh
+++ b/scripts/spellcaster.sh
@@ -140,6 +140,11 @@ function prepare_target() {
        cp "$CAULDRONDIR/ispells.$TYPE" "$TARGET"/ispells
        cp "$CAULDRONDIR/ospells.$TYPE" "$TARGET"/ospells
 
+       # Copy any spell-specific required options into the TARGET
+       [[ -d "$TARGET"/etc/sorcery/local/depends/ ]] ||
+               mkdir -p "$TARGET"/etc/sorcery/local/depends/
+       cp "$CAULDRONDIR"/depends/* "$TARGET"/etc/sorcery/local/depends/
+
        # generate basesystem casting script inside of TARGET
        cat > "$TARGET"/build_spells.sh <<-'SPELLS'
 
@@ -185,10 +190,53 @@ SPELLS
        chmod a+x "$TARGET"/build_spells.sh
 }
 
+function install_kernel() {
+       local SRC=$1
+       local DST=$2
+       local kconfig=$3
+       local version=
+       local kernel=
+
+       # Try to autodetect the location of the kernel config based on whether
+       # the linux spell was used or not.
+       if [[ -z $kconfig ]]
+       then
+       fi
+
+       if gaze -q installed linux &> /dev/null
+       then
+               version=$(gaze -q installed linux)
+               kernel=/boot/vmlinuz
+
+       # Try to autodetect the linux kernel version using spell version or
+       # kernel config
+       if [[ -z $version ]]
+       then
+               version="$(zgrep 'Linux kernel version:')"
+               version="${version#*version: }"
+       fi
+
+       # Try to guess the location of the kernel itself
+       kernel=
+
+       cp "$kconfig" "$DST"/boot/config-$version
+       cp "$SRC"/ "$DST"/
+       cp "$SRC"/ "$DST"/
+       cp "$SRC"/ "$DST"/
+       cp "$SRC"/ "$DST"/
+       cp "$SRC"/ "$DST"/
+}
+
 function setup_sys() {
        local SPOOL="$TARGET/var/spool"
        local SORCERY="sorcery-stable.tar.bz2"
        local SORCERYDIR="$TARGET/usr/src/sorcery"
+       local gvers=$(head -n1 "$TARGET"/var/lib/sorcery/codex/stable/VERSION)
+       local stable="stable-${gvers%-*}.tar.bz2"
+       local syscodex="$SYSDIR/var/lib/sorcery/codex"
+       local tablet="$SYSDIR/var/state/sorcery/tablet"
+       local packages="$SYSDIR/var/state/sorcery/packages"
+       local depends="$SYSDIR/var/state/sorcery/depends"
 
        # unpack the sys caches into SYSDIR
        for cache in $(<"$TARGET"/sys-list)
@@ -207,6 +255,29 @@ function setup_sys() {
        pushd "$SORCERYDIR" &> /dev/null
        ./install "$SYSDIR"
        popd
+
+       # install the stable grimoire used for build into SYSDIR
+       (
+               cd "$TARGET/tmp"
+               wget http://download.sourcemage.org/codex/$stable
+       )
+       [[ -d "$syscodex" ]] || mkdir -p $syscodex &&
+       tar jxf $stable -C "$syscodex"/
+       mv "$syscodex"/${stable%.tar.bz2} "$syscodex"/stable
+
+       # generate the depends and packages info for sorcery to use
+       . "$SYSDIR"/etc/sorcery/config
+       for spell in "$tablet"/*
+       do
+               for date in "$spell"/*
+               do
+                       tablet_get_version $date ver
+                       tablet_get_status $date stat
+                       tablet_get_depends $date dep
+                       echo "${spell##*/}:${date##*/}:$stat:$ver" >> 
"$packages"
+                       cat "$dep" >> "$depends"
+               done
+       done
 }
 
 function setup_iso() {
diff --git a/scripts/twelve-step-program.sh b/scripts/twelve-step-program.sh
index 458fd56..5699e1d 100755
--- a/scripts/twelve-step-program.sh
+++ b/scripts/twelve-step-program.sh
@@ -15,11 +15,10 @@
   test -d /root/build || 
   echo 'step 1 failed' >> /var/log/sorcery/activity
 
-  # step 1.5 (add resolv.conf and sorcery url configuration)
+  # step 1.5 (add resolv.conf and host sorcery url/compiler configuration)
   echo "step 1.5"
   cp /etc/resolv.conf /root/build/etc/resolv.conf
-  cp -fa /lib/modules/* /root/build/lib/modules
-  cp -fa /usr/src/* /root/build/usr/src
+  cp /etc/sorcery/local/compile_config 
/root/build/etc/sorcery/local/compile_config
   echo LEAPFORWARD_URL=http://10.0.0.11/smgl/spool/ > 
/root/build/etc/sorcery/local/url
 
   echo step 2 build kernel
@@ -34,7 +33,7 @@
     make -j 4 && make modules -j 4 && make modules_install &&
   popd &&
   ls /lib/modules &&
-  cp -fav /lib/modules/2.6.24 /root/build/lib/modules &&
+  cp -fav /lib/modules/2.6.24-SMGL-iso /root/build/lib/modules &&
   cp -fav /usr/src/linux-2.6.24 /root/build/usr/src ||
   echo 'step 2 failed' >> /var/log/sorcery/activity
 
@@ -44,7 +43,7 @@
 
   step 3.5 copy kernel sources to iso and sys tree
   # may be handled by step 3 later on
-  cp -fav /lib/modules/2.6.24 /tmp/cauldron/iso/lib/modules &&
+  cp -fav /lib/modules/2.6.24-SMGL-iso /tmp/cauldron/iso/lib/modules &&
   cp -fav /usr/src/linux-2.6.24 /tmp/cauldron/iso/usr/src ||
   echo 'step 3.5 failed' >> /var/log/sorcery/activity
 
@@ -79,7 +78,7 @@
   echo 'step 8 failed' >> /var/log/sorcery/activity
 
   echo step 9 make initrd
-  bash /root/cauldron/scripts/mkinitrd.sh /tmp/cauldron/iso 2.6.24 &&
+  bash /root/cauldron/scripts/mkinitrd.sh /tmp/cauldron/iso 2.6.24-SMGL-iso 
&&
   cp initrd.gz /root/cauldron/iso/boot/initrd.gz ||
   echo 'step 9 failed' >> /var/log/sorcery/activity