Skip to Content.
Sympa Menu

sm-commit - [SM-Commit] GIT changes to master cauldron by Justin Boffemmyer (cedadc6bff0d048e0122dbd8066209e63ba57dad)

sm-commit AT lists.ibiblio.org

Subject: Source Mage code commit list

List archive

Chronological Thread  
  • From: Justin Boffemmyer <scm AT sourcemage.org>
  • To: sm-commit AT lists.ibiblio.org
  • Subject: [SM-Commit] GIT changes to master cauldron by Justin Boffemmyer (cedadc6bff0d048e0122dbd8066209e63ba57dad)
  • Date: Sun, 14 Mar 2010 13:32:45 -0500

GIT changes to master cauldron by Justin Boffemmyer <flux AT sourcemage.org>:

API | 12 ++++
DESIGN | 163
+++++++++++++++++++++++++++++++++++++++++++++++++---------------
HACKING | 150 ++++++++++++++++++++++++++++++++++++++++++++++++----------
README | 3 -
TODO | 30 ++++++++---
5 files changed, 283 insertions(+), 75 deletions(-)

New commits:
commit cedadc6bff0d048e0122dbd8066209e63ba57dad
Author: Justin Boffemmyer <flux AT sourcemage.org>
Commit: Justin Boffemmyer <flux AT sourcemage.org>

documentation: updated for potion and enchantment

Updated all the documentation to include/be more complete for the new
potion as well as changes to enchantment.

diff --git a/API b/API
index 0824408..0af48c8 100644
--- a/API
+++ b/API
@@ -28,12 +28,22 @@ Anything pertaining to installers is referred to
collectively as "enchantment"

* All cauldron API functions are to be prefixed with cauldron_
* All cauldron API variables are to be prefixed with CAULDRON_
+
* All enchantment API functions are to be prefixed with enchant_
* All enchantment API variables are to be prefixed with ENCHANT_
+
+ * All potion API functions are to be prefixed with enchant_potion_
+ * All potion API variables are to be prefixed with ENCHANT_POTION_
+
* Other API variables and functions are to be prefixed in accordance with
what
makes sense for where those functions and variables are defined. For
example, in the common library liberror, functions are prefixed with
liberror_.
+
+ * Functions which logically group together (i.e., functions that operate
+ specifically on modules themselves) should share a common prefix (i.e.,
+ enchant_module_enter and enchant_module_exit)
+
* Functions should be preceded by verbose comments to describe the use of
the
function, formatted for use with bashdoc

@@ -86,5 +96,5 @@ prefixed by "cauldron_" since this is a cauldron function
(incidentally, this
function is defined in libcauldron). Local variable copies of the API-named
parameters are made as shorthand notations for the full API names.

-Though it is not required, all contributing developers are strongly urged to
+** Though it is not required, all contributing developers are strongly urged
to
document functions internally as well, as shown in the example.
diff --git a/DESIGN b/DESIGN
index ad7941d..af84dd9 100644
--- a/DESIGN
+++ b/DESIGN
@@ -1,27 +1,50 @@
-The basic toolchain can/should be gotten from the running host system. This
-can be accomplished by using binary caches from /var/cache/sorcery. If the
-cache file for a given toolchain spell does not exist, it should NOT be
-taken from a repository somewhere, or from a basesystem tarball. Nor should
the
-iso builder have to re-cast one of the tool-chain components in order to
-generate the cache file for use. Rather, it should be taken from the already
-existing installed files on the system. This will require sorcery to generate
-binary tarballs of installed spells. If it doesn't do this already, a
solution
-can be had by grabbing files from the installwatch logs (/var/state/sorcery?)
-and rolling those into a tarball. (I believe devel sorcery handles this
-situation.)
+In essence, Cauldron consists of two sub-components: cauldron (which is for
+creating installation ISOs, and manages building packages for the ISO and
+making sure everything fits together), and enchantment (which is for the
+installer interfaces that run during the ISO run-time). The two are related
to
+each other, but perform very distinct tasks, and are used at different times
+(cauldron is for build-time, enchantment is for run-time). To think of it
+another way, cauldron could be likened to "make cdrecord", while enchantment
+would be a front-end to cdrecord.
+
+===
+ISO Generation - cauldron
+===
+
+What follows below is a rough outline of the process Cauldron needs to go
+through in order to create ISOs. For the implementation, each separate part
of
+the process should be as modular as possible. Additionally, although each
part
+should be written so that user intervention is not needed, manual processing
+should be allowed for at every stage. This is to allow for fixes/workarounds
to
+be applied in the event of bugs in the code (which will invariably happen),
as
+well as to allow for more complicated processing that we either can't handle
in
+an automated way, or haven't had the foresight to account for ahead of time.
+
+The basic toolchain can/should be gotten from the running host system if
+possible. This can be accomplished by using binary caches from
+/var/cache/sorcery. If the cache file for a given toolchain spell does not
+exist, it should NOT be taken from a repository somewhere, or from a
basesystem
+tarball, unless absolutely necessary (see next paragraph). Nor should the iso
+builder have to re-cast one of the tool-chain components in order to generate
+the cache file for use. Rather, it should be taken from the already existing
+installed files on the system. This will require sorcery to generate binary
+tarballs of installed spells. If it doesn't do this already, a solution can
be
+had by grabbing files from the installwatch logs (/var/state/sorcery?) and
+rolling those into a tarball. (I believe devel sorcery handles this
situation.)

In the event that the host system does not have a build toolchain itself (an
-embedded system, for example, or a system geared for security), then a
-basesystem tarball will be needed. However, for the intents and purposes of
-Cauldron, that basesystem will then be considered the host system, so the
same
-discussion as above still applies.
+embedded system, for example, a system geared for security), then a
basesystem
+tarball will be needed. This will also be the case if cauldron is being run
+from a non-SMGL host system. However, for the intents and purposes of
Cauldron,
+that basesystem will then be considered the host system, so the same
discussion
+as above still applies.

Once the basic tool-chain tarball is generated/found/copied, it can be
unpacked
inside of the build directory. This, along with a copy of sorcery, should
-provide everything needed to generate the iso (aside from the spell packages
+provide everything needed to generate the iso aside from the spell packages
themselves, which should be gotten over the net or copied in manually from
pre-downloaded files. For grabbing over the net, a proper resolv.conf will be
-needed inside the iso build chroot).
+needed inside the iso build chroot.

The settings for sorcery should be set for the target system (target cpu,
etc.). For optimizations, strip should be used. The question of whether -Os
@@ -35,35 +58,35 @@ of the iso.

Afterwards, a basic system can be built inside the build chroot using
sorcery and the configured tool-chain. This basic system should be absolutely
-minimal, as it will be what is installed onto the target system. After the
base
-system is generated/built, the chroot should be exited, and the cache files
for
-all the spells that are to be included in the target install (non-optional
-spells) should be unpacked into a separate directory. That directory will
then
-be archived and compressed as system.tar.bz2 to be placed on the ISO. Don't
-forget to clean out unnecessary files from the new separate directory, _not_
-the original build directory, before making the bzipped tarball.
-[system.tar.bz2] (unnecessary files will include the host (cross-compile)
-tool-chain used to build the base system, as well as [possibly] sorcery files
-in /var, anything in /usr/src, etc.)
+minimal, as it will be what is non-optionally installed onto the target
system.
+After the base system is generated/built, the chroot should be exited, and
the
+cache files for all the spells that are to be included in the target install
+(non-optional spells) should be unpacked into a separate directory. That
+directory will then be archived and compressed as system.tar.bz2 to be placed
+on the ISO. Don't forget to clean out unnecessary files from the new separate
+directory, _not_ the original build directory, before making the bzipped
+tarball. [system.tar.bz2] (unnecessary files will include the host
+(cross-compile) tool-chain used to build the base system, as well as
[possibly]
+sorcery files in /var, anything in /usr/src, etc.)

Chroot back into the main build dir (i.e. not the system install copy).
Build the iso system by casting additional spells on top of the base system
generated in the previous step. This includes a kernel (config for the kernel
-SHOULD be something downloaded from a repository), filesystem tools
-(reiserfstools, xfstools, e2fstools, etc.), and anything else required not
only
-to boot the target system but manipulate and install onto it. This means that
-all features desired in the installer must be present or provided for through
-the spells cast in this step or the previous step (though very little should
-come from the previous step, as the previous step should really only provide
an
-absolute minimal tool-chain generated for the target system).
+SHOULD be taken from Cauldron itself), filesystem tools (reiserfstools,
+xfstools, e2fstools, etc.), and anything else required not only to boot the
+target system but manipulate and install onto it. This means that all
features
+desired in the installer must be present or provided for through the spells
+cast in this step or the previous step (though very little should come from
the
+previous step, as the previous step should really only provide an absolute
+minimal tool-chain generated for the target system).

Exit the iso build chroot. Unpack the necessary spell caches to a separate
directory for the iso system. Remove any unnecessary files/packages/etc. from
the iso (i.e., run the cleaners on it). Copy in the bzipped tarball of the
system install (system.tar.bz2).

-Generate an initrd/initramfs to be used to boot the ISO, and place it boot/
in
-the iso system directory just created and populated from the build spell
+Generate an initrd/initramfs to be used to boot the ISO, and place it in
boot/
+in the iso system directory just created and populated from the build spell
caches. Configure the boot/init process for the iso. This includes modifying
isolinux/isolinux.cfg, as well as various scripts/configs in /etc, as well as
generating an initrd/initramfs for the ISO. Pay particular attention to the
@@ -71,5 +94,69 @@ init system (including what will happen "before init").
(Whether or not other
bootloaders, such as grub, can be used as viable alternatives to isolinux
will
be researched later.)

-Generate a bootable iso using mkisofs. Compress the iso using bzip2. [e.g.
-smgl-devel.iso.bz2]
+Copy whatever installers are going to be used during the ISO runtime into the
+ISO. Make sure that it is set to run the installer(s) upon booting.
+
+Generate a bootable iso using mkisofs (or whatever other ISO creation tool
you
+prefer). Compress the iso using bzip2. [e.g. smgl-devel.iso.bz2]
+
+
+===
+Installers - enchantment
+===
+
+All common functionality for installers should be abstracted into libraries
+that can then be sourced by each installer. This is especially true for any
+back-end processing (any processing which is not directly related to
+interfacing with the user - compressing a file, keeping track of the current
+installation step, etc. are all back-end processes). Output messages should,
to
+the extent possible, be separated from code to allow for easier translation
and
+integration into i18n.
+
+To achieve the above goals, the installers are set up as follows. The common
+library is libenchantment, which is responsible for tracking the current
step,
+maintaining the history of steps (both forward and backward), getting the
list
+of modules, etc. For state tracking, this is left to lib.potion.
+
+The modules (steps) themselves are placed in their own directory, which will
be
+common to all installers. Each step will have a short summary description as
+$module.info, while the actual instructions are in $module. Because there may
+be processing specific to a module that really isn't shared with other
modules,
+or we may need to do processing specific to the start of a module (which
would
+be too tedious to test for each time in libenchantment), the each module will
+also have it's own enter, exit, and potion subdirectories, which can contain
+scripts for processing each of those events. Any scripts placed in those
+directories should be named based on the function of the script (fdisk if it
is
+to run fdisk one state information, for example). We may eventually want to
+make a common directory for such scripts, and then symlink them into the
module
+subdirectories, and only have actual script files in the module
subdirectories
+when they are truly unique to that module.
+
+Each installer will have its own directory. There will be required files that
+each installer is required to have (such as the list of modules it uses and
+their ordering), though this may be changed later if they are all forced to
+have the file contents in common. No installer should conflict with other
+installers (they should all be able to run in parallel, so that the user can
+switch from one to the other seamlessly).
+
+All installer state files are to be flat files. That means they should simply
+list filenames, spell names, mount points, etc. directly in the file, one per
+line. For organization, either use a directory or a separate file. Do not
+implement file formats unless you really, really have to, and even then don't
+do it without permission from the Cauldron Lead.
+
+Everything that could conceivably be user-configurable, should be. This
+includes what compression program is used, what language the installer is
+running in, and even where the installer files are located. This should all
+also be configurable at run-time, via the enchantment config file. If a user
+implemented their own installer, but didn't want to make a whole ISO just to
+include the couple of shell scripts comprising their new installer, as long
as
+they can mount a disk (or other device) containing the new scripts it should
be
+easy for the user to point enchantment at them and run them from the ISO. If
+the user speaks a language other than English, seeing their native language
+should require something as easy as changing ENCHANT_LANGUAGE="en" to
+ENCHANT_LANGUAGE="zh", for example. Ensuring user customizability will
require
+a little extra effort initially, but will make the installers far more
flexible
+and powerful, which will in the long-run help minimize the effort to add new
+code/features as well as make users more happy.
+
diff --git a/HACKING b/HACKING
index d358f3f..18f08c8 100644
--- a/HACKING
+++ b/HACKING
@@ -42,9 +42,9 @@ caches from the host system are only used to set up an
initial build toolchain,
which will then be used to produce a build toolchain specific to the target
system the ISO is to be built for.

-***********
+************
* lib.init *
-***********
+************
The lib.init module handles the init system for the ISO. It provides
functions for creating an initramfs/initrd and for setting up the ISO inittab
for booting into the correct enchant setup (i.e. putting the menu installer
on
@@ -61,12 +61,14 @@ and functions as they are needed within cauldron. This
includes functions which
source sorcery files in the target, as well as functions to download and
install sorcery into the target.

+
**************
* errorcodes *
**************
Defines the cauldron-specific error codes and corresponding messages
to
be used for error-checking via liberror functions.

+
*************
* cleaners/ *
*************
@@ -79,9 +81,10 @@ though the build tools themselves are not. No build tools
should be on the ISO!
Their spell caches are provided instead, and they are to be accessed from the
installed platform.

-***********************
-* etc/cauldron/config *
-***********************
+
+**************
+* etc/config *
+**************
The config file contains all the global variables used by the cauldron
scripts used to build an ISO that can be user-configured. Like the sorcery
config file(s), etc/cauldron/config is the main (default) config file, and
@@ -89,35 +92,35 @@ etc/cauldron/local/config is meant to contain modified or
custom values. For
any setting found in both config files, the etc/cauldron/local/config file
overrides what is in etc/cauldron/config.

-*****************************
-* etc/cauldron/$ARCH/spells *
-*****************************
+********************
+* etc/$ARCH/spells *
+********************
A list of all the spells necessary for the ISO runtime to function.
These are all basic spells, like bash for example. Later we will likely also
have arch-specific spell lists, since different archs will need, for example,
different boot loaders (grub and lilo are only for x86). The spell list is
used
by libcauldron.

-*****************************
-* etc/cauldron/$ARCH/caches *
-*****************************
+********************
+* etc/$ARCH/caches *
+********************
A list of the spells which are necessary for a basic installed target
system. This should be as close to a minimal basesystem as possible. Anything
which is truly optional should be provided as spell sources rather than built
caches. All spells whose caches are provided on the ISO must also have their
sources provided on the ISO.

-******************************
-* etc/cauldron/$ARCH/sources *
-******************************
+*********************
+* etc/$ARCH/sources *
+*********************
A list of the spells which are not necessary for a basic installed
target system. These sources are provided for additional functionality, like
networking. Exclusion of a spell from this list does not imply exclusion of
the
spell's sources from the ISO (cf. iso-caches.$ARCH, mentioned above).

-********************************
-* etc/cauldron/$ARCH/toolchain *
-********************************
+***********************
+* etc/$ARCH/toolchain *
+***********************
This is a list of all the spells needed for building the binaries to
go
onto the ISO. It includes things like gcc, as well as arch-specific spells
such
as bin86 for x86.
@@ -130,11 +133,18 @@ lib/
Code common to both cauldron and enchantment.

************
+* libcolor *
+************
+ This library defines all the color codes and message type definitions
+to be used by the cauldron and enchantment scripts, and functions to handle
+setting the colors for the needed message types.
+
+************
* liberror *
************
This library defines all the error codes recognized by the cauldron
-scripts, and functions to handle the printing of error messages using these
-codes. This is used by all the other scripts in the project.
+and enchantment scripts, and functions to handle the printing of error
messages
+using these codes. This is used by all the other scripts in the project.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


@@ -151,6 +161,16 @@ ISO installers. Currently sourced by libcauldron, but
likely will be sourced by
cauldron instead of libcauldron soon. It is also sourced by the different
installers, as it provides common functionality for them.

+**************
+* lib.potion *
+**************
+ This library provides state tracking functionality for enchantment.
The
+state tracking can be used for error-proofing installation procedures,
+automating installs (or even portions of them, as it can function over ranges
+of modules), and undo functionality if the functions are linked with the
+history functionality provided by libenchantment. This library is sourced by
+libenchantment.
+

**************
* errorcodes *
@@ -158,17 +178,18 @@ installers, as it provides common functionality for
them.
Defines the enchantment-specific error codes and corresponding
messages to
be used for error-checking via liberror functions.

-********************
-* enchantment/menu *
-********************
+
+*********
+* menu/ *
+*********
The menu installer (look at older ISOs (0.9.6.3 or earlier) for an
idea
of what this is). Everything relevant to the menu installer goes under this
directory, to be copied onto the ISO later if the ISO is intended to make use
of this module.

-*********************
-* enchantment/shell *
-*********************
+**********
+* shell/ *
+**********
The shell installer (as seen on the 0.10.0 ISOs). Everything relevant
to the shell installer goes under this directory, to be copied over to the
ISO
later if the ISO is intended to use this installer. (Note: The ISO really
@@ -178,9 +199,86 @@ resources. It's a good backup if something goes wrong
with another installer,
and has proven capable of being the only installer on the ISO as well. Other
installers are just eye-candy on top of this one.)

+**********************
+* $installer/modules *
+**********************
+ Each installer is required to have this file, which lists which
modules
+it will make use of, and the ordering of them.
+
+
+**********************
+* modules/help/index *
+**********************
+ When the user calls for help, the contents of this file should be
+displayed, which show what commands are available to the user and what they
do.
+This should probably be moved into installer-specific files.
+
+************************
+* modules/help/editors *
+************************
+ This is a list of all the editors that are available on the ISO.
+Different folks, different editing preferences. Never force your users to use
+one editor.
+
+***************************
+* modules/welcome/welcome *
+***************************
+ This is the welcome message displayed directly after the ISO finishes
+booting. It mentions how to proceed to the next steps, and also how to access
+different keymaps. This should probably be moved to installer-specific files.
+
+
+********************
+* modules/$module/ *
+********************
+ Each module directory houses all the files needed for that given
+module. Each of the files/directories associated with a module are dealt with
+in turn below.
+
+***************************
+* modules/$module/$module *
+***************************
+ These are the actual instructions that would be presented to the user.
+The file includes a very simplistic markup to designate text which should be
+highlighted (commands, headings, etc.). The markup is replaced at runtime to
+display in the desired state (for the shell, escape codes either representing
+bold or a color if libcolor is loaded).
+
+********************************
+* modules/$module/$module.info *
+********************************
+ This is the short summary info for the module displayed when the list
+of modules is given (i.e., the shell installer's "steps" command).
+
**************************
-* etc/enchantment/config *
+* modules/$module/enter/ *
**************************
+ This is a collection of scripts to run when the user enters the
module.
+Typically these scripts are used for preparatory work, such as ensuring that
+module-specific temporary files exist and have needed start values, but could
+do anything that needs to be executed at the start of a module. These scripts
+are not passed any arguments, so they must be fully autonomous.
+
+*************************
+* modules/$module/exit/ *
+*************************
+ This is a collection of scripts to run when the user exits the module.
+Typically these scripts are used for cleanup work, such as writing out final
+values to the module-specific temporary files, but could do anything that
needs
+to be executed at the end of a module. These scripts are not passed any
+arguments, so they must be fully autonomous.
+
+***************************
+* modules/$module/potion/ *
+***************************
+ This is the collection of scripts responsible for automating the
+process of the module. Each script in this directory must take one argument,
+which is the path to the potion to use for automating the module.
+
+
+**************
+* etc/config *
+**************
The config file contains all the global variables used by the
enchantment
scripts used during the ISO runtime that can be user-configured. Like the
sorcery config file(s), etc/cauldron/config is the main (default) config
file,
diff --git a/README b/README
index 06da3fb..2a0e841 100644
--- a/README
+++ b/README
@@ -3,7 +3,8 @@ general cauldron tools.

If you would like to contribute to Cauldron development, please read HACKING
and API first. DESIGN will explain the general principles motivating how
-Cauldron is set up and what needs to be done to build an ISO.
+Cauldron is set up and the general processes involved in building an ISO and
+installing from one.

If you are looking for an area that you can contribute to, feel free to read
TODO, or ask around in #sourcemage-cauldron on irc.freenode.net.
diff --git a/TODO b/TODO
index 90e6a75..3e58db9 100644
--- a/TODO
+++ b/TODO
@@ -7,11 +7,10 @@ Things to do in order to achieve 0.1
- perform the build
- clean the chroot out (cleaners)
- add installer(s)
- - extra sanity checks on boot and etc in chroot (add_sauce stuff)
- - switch initramfs/initrd to use busybox
+ - extra sanity checks on boot and etc in chroot (add_sauce stuff for
+ those familiar with the 0.10 series)
+ - switch initramfs to use busybox
o will need a scrubbed busybox config for this
-* set up how to do chroots in libcauldron (${CAULDRON_CHROOT}) (mostly done)
-* write code to install sorcery into the CAULDRON_BUILD chroot
* write a cauldron spell so that cauldron can install itself
* write code to set up the init process on the ISO
- this needs to be sensitive to what installers are present
@@ -21,13 +20,26 @@ Things to do in order to achieve 0.1
~ Not sure if this should really be for 0.1, but a lot of
people have asked for it already.
- i18n
- o fbterm + fontconfig + freetype2 for font displaying
- o uim (uim-fep) for input
+ o translate error messages, instructions, etc. and implement
an
+ easy way of calling them based on what language is selected
+ ($ENCHAN_LANGUAGE in the config file, plus either a file
+ suffix (errorcodes.en) or a separate dir (en/errorcodes) for
+ each language supported) - the framework is mostly in place,
+ now we need translators
+ o display (fonts): fbterm + fontconfig + freetype2
+ o input: uim (uim-fep) + m17n-lib + m17n-db
* LOTS AND LOTS OF TESTING!
* More sanity checks and error testing (use liberror!)
- - liberror should get a function liberror_chkerror or something, which
- will take an error code and do different things depending on the
- debug level, etc.
+ - this is done, but could be more complete
+ - currently errors are emitted, but they aren't yet really handled -
we
+ want them handled at higher levels, because how an error is
displayed
+ to the user will partly depend on what installer they are currently
+ using, but we shouldn't really "ignore" the errors at the lower
+ levels either
+ - we need to determine at what levels which errors are really fatal,
+ which errors are warnings, and which errors are more informational
+ debug messages - once we determine all this we can start properly
+ handling, passing, or ignoring errors

====================
Things to do for 1.0



  • [SM-Commit] GIT changes to master cauldron by Justin Boffemmyer (cedadc6bff0d048e0122dbd8066209e63ba57dad), Justin Boffemmyer, 03/14/2010

Archive powered by MHonArc 2.6.24.

Top of Page