Public SourceMage Discussion List

[flux <flux AT sourcemage.org>]

David Kowis (dkowis AT shlrm.org) wrote [13.01.28 06:07]:
> > <snip>
> >> Updated init system:
> >> --------------------
> >> Simpleinint-msb works, but it's old and crappy. I'm a fan of systemd,
> >> because it makes things amazingly easy to do, and is ridiculously fast.
> >> It's also been adopted by several linux distros, and large projects.
> >> However, that's not set in stone, we just need something newer, and
> >> preferably something that other distros are using, so we can take
> >> advantage of the work that others are doing as well.
> > </snip>
> > 
> > Old? Yes. Unmaintained? Yes. Crappy? That depends on how you define
> > "crappy". All the systems that use simpleinit with SMGL boot (to my
> > knowledge), so it clearly works. Although there has been some work to
> > get an agnostic init setup, and a select few have done custom setups to
> > run alternative inits, there hasn't been a widespread or large push to
> > get a different init system. That implies that for most SMGL users
> > simpleinit more than "works", it "works well enough". This, I think, is
> > like unix compared to plan9: it works well enough and the newer
> > alternative doesn't bring *significantly* more to the table that people
> > are sticking with the old and tried-and-true.
> > 
> > I'm not advocating that simpleinit is great and we should stay with it.
> > I'm not advocating that systemd isn't great. I'm simply stating that
> > you're making a subjective but baseless claim that simpleinit is "bad"
> > without saying/showing in what way(s) it's bad.
> 
> It's *far* easier to build initscripts for systemd than it is for
> simpleinit-msb or any other init system I've used. I am generally
> inclined to agree that systemd isn't traditional UNIX philosophy, but I
> really like how easy systemd makes it to build complicated and powerful
> init systems. Simpleinit-msb doesn't do any of those things, and us as a
> distro will have to reinvent all that stuff. Primarily, however, I would
> like to move to something that's more common, and used by other distros,
> so we can leverage the users of other distros.

While it may be easier to create init scripts for systemd, you still
haven't pointed out what simpleinit cannot do that systemd can (if we
aren't looking at how easy it is to do so). You claimed that
"simpleinit-msb doesn't do any of those things", but you failed to
clarify what "those things" refers to. If you meant building complicated
and powerful init systems, then neither simpleinit-msb or systemd do
this: they don't *build* init systems, they *are* init systems. If you
meant building init setups, then both allow for this. You can use any
language you want to write init scripts for simpleinit, since all it
cares about is whether a file in a given "runlevel" directory is
executable or not. This means that your init script can be written in C,
python, Lua, Java, etc. If you are unable to build something complicated
and powerful given that, then perhaps you are doing things wrong. If
your argument is that we don't necessarily supply something more
powerful/complicated out of the box, then you'd be right, but I'd argue
that's not a problem since ultimately that should be up to the admin.

Again, I'm not arguing against systemd. You seem to be focusing your
argument a little more on pro-systemd and anti-simpleinit. I'm just
looking to clarify anti-current_setup vs. pro-new_setup, regardless of
what new system we end up going with. Change is fine, as long as we have
clear goals and targets for the change.

Also, although it's harder to implement, I'd like to see IMP get more
fully implemented. The philosophy and spirit of our distro is choice for
the user/admin, and not to get in the admin's way. I'd prefer for a user
to be able to use whatever init system he/she wants. There are
consequences for that, both for the user and for us as developers, but I
think it's worth it. Compare it with the choice of udev vs. manual /dev
tree management. Many distros have thrown away this choice for the user
and force udev, but we allow the choice, and I think we're better for
it.

> > 
> > <snip>
> >> Chroot based build process:
> >> ---------------------------
> >> For building binary packages, I want to take advantage of a chroot and
> >> unionfs (or rsync and hardlinks or something.) Inspiration from this
> >> page:
> >> https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot
> >> It's probably the sanest way to produce a package that we can ensure
> >> isn't melding in dependencies we don't want and such. By ensuring that
> >> we build things into binary packages as well, we can catch leaky
> >> installs, or missing dependencies when we're building the chain of
> >> packages. Additionally, it'll give the system itself protection from a
> >> stupid installer doing bad things, or a partially failed install.
> > </snip>
> > 
> > I have actually already been working on something like this for a while
> > in the newer version of cauldron. Better binary support in sorcery would
> > certainly be a plus for this, but at least we can get the basics by
> > having bare "defaults" for everything. If you only use the defaults from
> > the spells, then you can verify/repeat the binaries by getting the state
> > of the grimoire that matched what a given chroot/ISO was built against
> > (via git) and running the same spells with the defaults.
> 
> Until the spells are updated, and you have to go re-verify all the
> defaults, and you have no way of knowing what defaults are new/changed
> without manually going through the process. This hearkens back to the
> people time is more important than computer time, and so we should make
> the computer do this work forus.

Yes, of course. I wasn't saying that no further change was needed. I was
only pointing out that we can take a first step in this direction
already, even with unmodified sorcery/grimoire/etc. Modifying sorcery
would get us even further. In other words, I was agreeing with you. I
said as much above, but perhaps I worded it poorly.

> > 
> > Note that chrooting won't give perfect security against stupid
> > installers, unless your chroots are *very* tight, and even then it's
> > possible an intentionally malicious install might have a trick to
> > circumvent something. This is always true as a security issue in general
> > though, and adding another layer is a good idea.
> > 
> > However, there is a issue for doing *all* installs via chroot: you will
> > be casting spells over and again even when they are already installed,
> > unless you first graft them in from the host system. This can get
> > complicated, but it is possible to do it, except you will only be able
> > to do so when the version in the host system matches the options
> > requested by the chrooted spell cast. You'll also need a smarter way to
> > handle conflicts/merges/updates between versions in the chroot vs.
> > versions in the host system. I.e., if you have gcc without g++ in the
> > host system, and cast a spell in a chroot that forces/requests gcc with
> > g++, you'll likely need the g++ enabled version in the host system (for
> > libstdc++ at runtime). That means updating the host gcc. In this case,
> > there's probably no issue and you can just do it, but in some other
> > cases it might cause existing spells in the host system to break due to
> > library changes (especially if a spell forces a dependency without a
> > feature that's enabled in the host). This can be done smartly, but will
> > need to be planned out and accounted for.
> 
> I plan that storing the spell config in the binary tarball will indicate
> that you need something specific installed.

How is storing spell config in the binary tarball related to the
pitfalls of building in a chroot?

Note that I am not advocating against chroot builds, I'm merely pointing
out that we need to carefully plan how to implement them so that they
are done properly and handle potential pitfalls well.

> > 
> > <snip>
> >> Declarative spell config:
> >> -------------------------
> >> Spell configuration needs to not be procedural. I should be able to say
> >> "cast kde" and get a menuconfig style interface where I can toggle
> >> things off and on and know what the effects of my selections are going
> >> to be without having to restart the entire process again. I should also
> >> be able to store a config to a file "Dave's KDE Desktop Config" and load
> >> that in, and be notified of new options somehow. This is critical not
> >> only to making it easier for people to construct systems, but to have
> >> repeatable builds. When someone complains that their package doesn't
> >> build, we can ask for their config, throw it in a chroot, and duplicate
> >> the problem, either finding a patch, or finding out that their config is
> >> simply broken. Finally, having stored configs allows us to package those
> >> up with a binary package, and should you already have a binary package
> >> with the proper config, you can just extract that rather than rebuild it
> >> again.
> > </snip>
> > 
> > You don't need a declarative config to make this happen. You just need
> > to have a system that can handle dynamically-updated menus. And yes, I
> > think a menu system would be a smart thing to do for sorcery, at least
> > as an option over the current presentation, and has been generally
> > requested by others in the past.
> 
> I believe that making it strictly declarative will make it easier to
> deal with configurations. Having logic makes it much more difficult for
> a computer to compare differences in config, since we cannot know what
> it is without executing that path.

I believe I see where you were going with this argument, so I will
attempt to clarify it here. Please correct me if I am mistaken. What you
meant by "deal with configurations" was pointed not at creating the
configuration initially (i.e. getting user input in the first place
during an actual cast), but in verifying prior configurations so that
they can be repeated. In this case, declarative clearly wins over
procedural.

The menu system could, theoretically, be implemented in sorcery as it is
now though. The menu system would just have to also be procedural and
dynamic, so that it can update menu options as needed to reflect the
logic followed in the spell's configuration. We actually had just such a
menu system in the older ISOs (0.9.5 for example).

> > I think there's another issue hiding behind this one though: the
> > metadata we collect/store for spell configuration. Regardless of how the
> > options are presented to the user, ultimately the repeatability question
> > comes down to how the spell is finally configured, after getting user
> > input. How we store user configurations is an area where I think there
> > is much more room for improvement, even over how spells are written by
> > the developers. It'd be nice to be able to write a user configuration
> > file *by hand* and propagate it to different machines in order to cast a
> > spell with the same options on different machines. That doesn't require
> > the spell files themselves to be declarative. If enough information (and
> > of the right kind(s)) is stored in the final configuration file, then
> > sorcery could even outright bypass the spell files and use the metadata
> > config file instead to build a spell. IMHO this is a better direction to
> > go down first, and the issue of declarative vs. procedural spell files
> > can wait.
> 
> I disagree, since figuring out how to load in that config is critical to
> procedural vs. declarative. It's much easier to load in a config from
> somewhere else if it's declarative. Much harder if it's procedural.
> Especially if we need to compare things, like what options are new, etc.

I think you misread me. I wasn't making an argument about declarative
vs. procedural at all. Metadata is generally inherently declarative (if
you know of non-declarative metadata, feel free to let me know). I
claimed that both how sorcery stores final configuration information as
well as what specific information sorcery chooses to store is, IMHO, far
more important than the question of spell files being declarative vs.
procedural (at least initially). How you load a static config is
actually the same in both procedural and declarative options: you just
load it. The difference comes in how they operate on the loaded config.
Regardless, it's the config that's really more important here. To
reiterate what I pointed out above, if you have a good enough config
stored, you should be able to actually completely bypass the spell files
altogether for successive casts of the same spell using the same config.

When you mention comparing what options are new, you'd be comparing a
config for one version of a spell to a (potential) config for a new
version of a spell. Again, this is not a question of procedural vs.
declarative in the system that generates such a config, but only a
question of the config itself. Take for example the linux kernel .config
file. The system that generates this .config file is procedural (unless
you do allyesconfig or similar). The .config file itself is declarative,
but that's because it's pure config data. This file is then used for
comparing with new options, even in the procedural config generation
(drop an older .config into a kernel source tree, run make menuconfig or
similar, and options are populated accordingly with new ones marked as
(NEW) in the menus).

Again, I'm not advocating for or against declarative spell files right
now. I'm arguing that before we muck about with spell files we need to
make sure we get the metadata (a spell's .config, if you will) right. If
we have that right, then we should be able to do the right thing whether
we do declarative or procedural config generation, and then we can even
allow for multiple alternative implementations.

> > 
> > In any event, from both David's and my arguments, it seems to me that
> > what's most needed is (not in order of importance):
> > 
> > * core grimoire spells that are tightly controlled/tested and also
> >   offered as official binary caches (and possibly as a fully separate
> >   core grimoire)
> > 
> > * improvements to sorcery to handle binary caches better and with more
> >   repeatability/testability
> > 
> > * improvements to sorcery to (better) recreate spell builds from a
> >   given configuration, which are (more) repeatable/testable
> 
> I would reverse the order of these, because the last will contribute
> greatly to the first, as well as the second contributing to the first.
> Given those bits of infrastructure, we can more easily produce tightly
> controlled/tested binaries.

Please reread. I stated explicitly that my list was unordered.

-- 
Justin "flux_control" Boffemmyer
Cauldron wizard and general mage
Source Mage GNU/Linux
http://www.sourcemage.org

Attachment: pgp8ttW7krmJv.pgp
Description: PGP signature