Skip to Content.
Sympa Menu

sm-discuss - Re: [SM-Discuss] Teh Future

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: flux <flux AT sourcemage.org>
  • To: SM-Discuss AT lists.ibiblio.org
  • Subject: Re: [SM-Discuss] Teh Future
  • Date: Sun, 27 Jan 2013 01:03:30 +0900

tl;dr: I agree with the general impetus, but I think some of the
specifics need some additional thought/input from everyone so we
can improve it further.

David Kowis (dkowis AT shlrm.org) wrote [13.01.26 08:28]:
</snip>
> Single spell file:
> ------------------
> Opening 6 files is difficult. Opening one file is easy. I can maintain a
> much better context regarding what my variable names are, and what I'm
> going to do with them. There's no reason we *must* split things up into
> many files, and I don't think there's any benefit to it. I believe this
> is a change we must implement.
</snip>

I believe you were considering the developer's perspective, but not the
computation perspective. There is a very clear computational reason to
have separate files, especially with the way sorcery is currently set
up. The various stages of casting are truly separate stages, and can
actually be run separately (see delve). This is, IMHO, actually a good
thing. If everything for a spell is in a single file, it means that
single file will be loaded every time sorcery enters a different stage
of processing. Now, that may not matter much if each file is very small
and doesn't do much. However, it will mean that sorcery *must* load and
parse the *entire* file for every stage. Currently, if a file doesn't
exist at all, sorcery performs the default action for that stage. A test
if the file exists is much cheaper than parsing a file to find out we're
just going to run the default anyway. Also, if the single file is large
enough (unlikely, but possible for monster spells like the linux spell),
it would consume more RAM/time/etc. to load the file, and you'll be
loading it multiple times so it stacks. This second argument might be
moot on most machines where resources are nowadays far more ample than
they used to be, but there it is.

Of course from a developer's point of view David's assessment is, IMHO,
quite accurate.

<snip>
> Chroot from packages:
> ---------------------
> I need to be able to run a script and end up with a chroot that is a
> base system for SMGL. Those packages should be sourced from the Core
> grimoire.
</snip>

Although it's certainly not perfect, this script already exists in the
Cauldron repo. The main issue is that the spells themselves aren't
necessarily repeatable, and we don't have individual spells as supported
binary caches on a server, only chroots. The chroots that are currently
on the servers were not generated with a script (to my knowledge), so
their own repeatability can't be guaranteed. Thus, it's a sort of
chicken and egg problem due to the lack of the supported and official
binaries, but the (start of the) script is already there, so if we get
the binary stuff in place then the rest of the work is (at least mostly)
already done.

<snip>
> Updated init system:
> --------------------
> Simpleinint-msb works, but it's old and crappy. I'm a fan of systemd,
> because it makes things amazingly easy to do, and is ridiculously fast.
> It's also been adopted by several linux distros, and large projects.
> However, that's not set in stone, we just need something newer, and
> preferably something that other distros are using, so we can take
> advantage of the work that others are doing as well.
</snip>

Old? Yes. Unmaintained? Yes. Crappy? That depends on how you define
"crappy". All the systems that use simpleinit with SMGL boot (to my
knowledge), so it clearly works. Although there has been some work to
get an agnostic init setup, and a select few have done custom setups to
run alternative inits, there hasn't been a widespread or large push to
get a different init system. That implies that for most SMGL users
simpleinit more than "works", it "works well enough". This, I think, is
like unix compared to plan9: it works well enough and the newer
alternative doesn't bring *significantly* more to the table that people
are sticking with the old and tried-and-true.

I'm not advocating that simpleinit is great and we should stay with it.
I'm not advocating that systemd isn't great. I'm simply stating that
you're making a subjective but baseless claim that simpleinit is "bad"
without saying/showing in what way(s) it's bad.

<snip>
> Chroot based build process:
> ---------------------------
> For building binary packages, I want to take advantage of a chroot and
> unionfs (or rsync and hardlinks or something.) Inspiration from this
> page:
> https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot
> It's probably the sanest way to produce a package that we can ensure
> isn't melding in dependencies we don't want and such. By ensuring that
> we build things into binary packages as well, we can catch leaky
> installs, or missing dependencies when we're building the chain of
> packages. Additionally, it'll give the system itself protection from a
> stupid installer doing bad things, or a partially failed install.
</snip>

I have actually already been working on something like this for a while
in the newer version of cauldron. Better binary support in sorcery would
certainly be a plus for this, but at least we can get the basics by
having bare "defaults" for everything. If you only use the defaults from
the spells, then you can verify/repeat the binaries by getting the state
of the grimoire that matched what a given chroot/ISO was built against
(via git) and running the same spells with the defaults.

Note that chrooting won't give perfect security against stupid
installers, unless your chroots are *very* tight, and even then it's
possible an intentionally malicious install might have a trick to
circumvent something. This is always true as a security issue in general
though, and adding another layer is a good idea.

However, there is a issue for doing *all* installs via chroot: you will
be casting spells over and again even when they are already installed,
unless you first graft them in from the host system. This can get
complicated, but it is possible to do it, except you will only be able
to do so when the version in the host system matches the options
requested by the chrooted spell cast. You'll also need a smarter way to
handle conflicts/merges/updates between versions in the chroot vs.
versions in the host system. I.e., if you have gcc without g++ in the
host system, and cast a spell in a chroot that forces/requests gcc with
g++, you'll likely need the g++ enabled version in the host system (for
libstdc++ at runtime). That means updating the host gcc. In this case,
there's probably no issue and you can just do it, but in some other
cases it might cause existing spells in the host system to break due to
library changes (especially if a spell forces a dependency without a
feature that's enabled in the host). This can be done smartly, but will
need to be planned out and accounted for.

<snip>
> Declarative spell config:
> -------------------------
> Spell configuration needs to not be procedural. I should be able to say
> "cast kde" and get a menuconfig style interface where I can toggle
> things off and on and know what the effects of my selections are going
> to be without having to restart the entire process again. I should also
> be able to store a config to a file "Dave's KDE Desktop Config" and load
> that in, and be notified of new options somehow. This is critical not
> only to making it easier for people to construct systems, but to have
> repeatable builds. When someone complains that their package doesn't
> build, we can ask for their config, throw it in a chroot, and duplicate
> the problem, either finding a patch, or finding out that their config is
> simply broken. Finally, having stored configs allows us to package those
> up with a binary package, and should you already have a binary package
> with the proper config, you can just extract that rather than rebuild it
> again.
</snip>

You don't need a declarative config to make this happen. You just need
to have a system that can handle dynamically-updated menus. And yes, I
think a menu system would be a smart thing to do for sorcery, at least
as an option over the current presentation, and has been generally
requested by others in the past.

> That's my brain dump of high level things I think SMGL needs to
> accomplish to remain relevant in the world of Linux Distros. I can't
> promise that I have time to do all these things, as I don't really. I
> can help with planning, and I can help with direction, but I will need
> others to step up and help start with the design of these things.
> In order of importance:
> 1. Declarative spell config
> 2. Binary package
> 3. Chroot generation
> 4. ISO generation
> 5. Updated init
> 6. Single spell file (although, I think this basically happens if we
> implement declarative spell configs)
>
> Doing the things in this order gives us clear goals, and we can then get
> onto shipping ISOs, and it will make it significantly easier for people
> to start using the distro, as well as easier for us to continue to use
> the distro and build/maintain packages.
>
> Or we can leave it at 1.0 and be done, moving on to different things.
>
> --
> David Kowis

For the points I skipped, I basically agree with them as they are and
don't really have any input.

Personally, I think having individual official/supported binaries of
core spells (gcc, glibc, etc.) needed to put together a base system are
more important at this stage than anything else. Many times that users'
systems have gone somewhat belly-up they have had to resort to pulling
binaries off of ISOs or out of basesystem chroots. The problem there is
that a) the ISOs and chroots don't always match the versions of the
spells the users need, especially because they stale and don't get
replenished often enough (due to the generation/repeatability issues),
and b) it's slow and resource intensive to (possibly first) download the
entire ISO/chroot and then mount/unpack the file to copy the (usually
only one) binary out of. Having supported, official individual spell
binaries for truly core spells will enable both system rescue and
generation of fresh chroots/ISOs more easily. This in turn would
facilitate more experimentation/testing, as it would be easier to set up
test environmentsi, which would help get everything else into place
faster.

Whether we really want to have declarative spell configs depends on how
it's actually implemented. There are trade-offs that need to be properly
weighed. Non-declarative gives flexibility and power, but declarative
gives robustness and (much easier) repeatability.

I think there's another issue hiding behind this one though: the
metadata we collect/store for spell configuration. Regardless of how the
options are presented to the user, ultimately the repeatability question
comes down to how the spell is finally configured, after getting user
input. How we store user configurations is an area where I think there
is much more room for improvement, even over how spells are written by
the developers. It'd be nice to be able to write a user configuration
file *by hand* and propagate it to different machines in order to cast a
spell with the same options on different machines. That doesn't require
the spell files themselves to be declarative. If enough information (and
of the right kind(s)) is stored in the final configuration file, then
sorcery could even outright bypass the spell files and use the metadata
config file instead to build a spell. IMHO this is a better direction to
go down first, and the issue of declarative vs. procedural spell files
can wait.

In any event, from both David's and my arguments, it seems to me that
what's most needed is (not in order of importance):

* core grimoire spells that are tightly controlled/tested and also
offered as official binary caches (and possibly as a fully separate
core grimoire)

* improvements to sorcery to handle binary caches better and with more
repeatability/testability

* improvements to sorcery to (better) recreate spell builds from a
given configuration, which are (more) repeatable/testable

--
Justin "flux_control" Boffemmyer
Cauldron wizard and general mage
Source Mage GNU/Linux
http://www.sourcemage.org

Attachment: pgp2mux7ghrs0.pgp
Description: PGP signature




Archive powered by MHonArc 2.6.24.

Top of Page