Public SourceMage Discussion List

[Jaka Kranjc <smgl AT lynxlynx.info>]

On Friday 25 of January 2013 17:27:54 David Kowis wrote:
> /.../
> A bit more detail on the individual items:
> 
> Single spell file:
> ------------------
> Opening 6 files is difficult. Opening one file is easy. I can maintain a
> much better context regarding what my variable names are, and what I'm
> going to do with them. There's no reason we *must* split things up into
> many files, and I don't think there's any benefit to it. I believe this
> is a change we must implement.
The vast majority of spells are very short and simple automations, so I find 
your reasoning and maybe ignorance absurd. Consider using an IDE if you 
really 
have problems.

The bad aspects were already outlined, so let me just mention that the 
inheritance we use could be adapted, but it would all be at a cost to clarity 
and scriptability (quill would also need a rewrite).

> Well defined binary package format:
> -----------------------------------
> Our ISO is insanely difficult to generate, because we have no binary
> packages at all. We have no way of bootstrapping a system. We've had
> attempts at this in the past with official chroots, but there was never
> a particularly repeatable way to generate them. I want to make a smaller
> Core grimoire that we do provide binary packages for that can be used to
> build a completely new system. Not enough to install all of KDE or
> whatever, or even X, but a booting system that you can log in to, and
> start building your own packages. To do this, we need a binary package
> format that can be acquired, verified, and examined easily. I need to
> know how the package was configured, when it was built, and of course, I
> need the files to install. This is also necessary for building an ISO.
> Acquiring and extracting a known config for a spell to build the
> official ISO is necessary to have repeatable, reliable ISOs. As well,
> this enables sharing of built packages across many machines easily,
> especially if they're all built the same way. The binary package format
> will need to contain enough information to be able to verify the config
> of the binary package, and compare it to what was requested to be built.
What's actually missing now? We store most of the state that was used when 
the 
spell started casting and you can even have config.log appended to the 
compile 
log (which is external, I know). Reuse is as simple as unpacking it, exactly 
what resurrect does.
 
> Core grimoire:
> --------------
> We need to pare down the amount of stuff that is well supported. We
> don't have enough manpower to take care of all the things. The Core
> grimoire will provide binary packages of the things we support. Those
> configs are what we support. This way we can guarantee that someone will
> be able to get a working system fired up based on the configs we've got.
> We should have no problem supporting configs for i686 as well as x86_64
> eventually, but we can target only one for the beginning.
Is there a buildbot planned, would everyone touching it need a clean chroot 
for generation, would it be a two-step process or something else?

How would it be versioned/developed? I guess it would replace stable/rc/test 
and all the others would remain in what currently test is.
 
> Chroot from packages:
> ---------------------
> I need to be able to run a script and end up with a chroot that is a
> base system for SMGL. Those packages should be sourced from the Core
> grimoire.
> 
> ISO from packages:
> ------------------
> I need to be able to run a script and end up with an ISO containing
> enough stuff to install a new system. Those packages need to come from
> the Core grimoire.
> 
> Updated init system:
> --------------------
> Simpleinint-msb works, but it's old and crappy. I'm a fan of systemd,
> because it makes things amazingly easy to do, and is ridiculously fast.
> It's also been adopted by several linux distros, and large projects.
> However, that's not set in stone, we just need something newer, and
> preferably something that other distros are using, so we can take
> advantage of the work that others are doing as well.
Hasn't IMP gotten far enough? IIRC we already have a working base (upstart, 
systemd).

> Chroot based build process:
> ---------------------------
> For building binary packages, I want to take advantage of a chroot and
> unionfs (or rsync and hardlinks or something.) Inspiration from this
> page:
> https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroo
> t It's probably the sanest way to produce a package that we can ensure isn't
> melding in dependencies we don't want and such. By ensuring that we build
> things into binary packages as well, we can catch leaky
> installs, or missing dependencies when we're building the chain of
> packages. Additionally, it'll give the system itself protection from a
> stupid installer doing bad things, or a partially failed install.
castfs is ran in a chroot. I still see no reason why it couldn't be extended 
to also include the BUILD stage. Cleanliness can be ensured by the chroot 
system itself (one of the previous goals) and removal of the new after 
casting 
(the same dance prometheus did).

Only a sane chroot setup can help with catching extra libraries being linked 
in or similar, but it may be a bit much to request everyone use it, I don't 
know. For this "core" grimoire definitely, for the rest, I'm not so sure it 
should be a requirement. Any packager using non-generic optimisations would 
have to build the updates twice.

> Declarative spell config:
> -------------------------
> Spell configuration needs to not be procedural. I should be able to say
> "cast kde" and get a menuconfig style interface where I can toggle
> things off and on and know what the effects of my selections are going
> to be without having to restart the entire process again. I should also
> be able to store a config to a file "Dave's KDE Desktop Config" and load
> that in, and be notified of new options somehow. This is critical not
> only to making it easier for people to construct systems, but to have
> repeatable builds. When someone complains that their package doesn't
> build, we can ask for their config, throw it in a chroot, and duplicate
> the problem, either finding a patch, or finding out that their config is
> simply broken. Finally, having stored configs allows us to package those
> up with a binary package, and should you already have a binary package
> with the proper config, you can just extract that rather than rebuild it
> again.
Configs are already stored, extractable and reproducible.

As for the declarative part, some solution to be able to see and set 
everything up front would definitely be cool. Especially for any test 
scripting 
without Expect.
I don't see a way to make it completely declarative though, unless you leave 
the dependency resolution out of it or (re)generate all the possible trees. :S

A lot can be done with smarter PREPARE/CONFIGURE/DEPENDS — you can optionally 
add any dialog gui code over it. The two core dependency functions (used for 
the 4 exposed ones) can be used to make noninteractive versions of 
optional/suggest_depends.

LP
-- 
To err is humor