Public SourceMage Discussion List

[David Kowis <dkowis AT shlrm.org>]

On 1/27/2013 2:55 PM, Jaka Kranjc wrote:
> On Friday 25 of January 2013 17:27:54 David Kowis wrote:
>> /.../
>> A bit more detail on the individual items:
>>
>> Single spell file:
>> ------------------
>> Opening 6 files is difficult. Opening one file is easy. I can maintain a
>> much better context regarding what my variable names are, and what I'm
>> going to do with them. There's no reason we *must* split things up into
>> many files, and I don't think there's any benefit to it. I believe this
>> is a change we must implement.
> The vast majority of spells are very short and simple automations, so I 
> find 
> your reasoning and maybe ignorance absurd. Consider using an IDE if you 
> really 
> have problems.
> 
> The bad aspects were already outlined, so let me just mention that the 
> inheritance we use could be adapted, but it would all be at a cost to 
> clarity 
> and scriptability (quill would also need a rewrite).

Not all spells are so simple. It gets much more complicated when one
tries to deal with multi-version spells, or packages with complex
sub-dependencies. It would be significantly easier to open only one
file, than it it to open six, especially when shelled into a box using
vim. I believe that people efficiency is important, and I would rather
us change the way that we do things than provide a solution that is "Go
use an IDE, if you don't like it."

> 
>> Well defined binary package format:
>> -----------------------------------
>> Our ISO is insanely difficult to generate, because we have no binary
>> packages at all. We have no way of bootstrapping a system. We've had
>> attempts at this in the past with official chroots, but there was never
>> a particularly repeatable way to generate them. I want to make a smaller
>> Core grimoire that we do provide binary packages for that can be used to
>> build a completely new system. Not enough to install all of KDE or
>> whatever, or even X, but a booting system that you can log in to, and
>> start building your own packages. To do this, we need a binary package
>> format that can be acquired, verified, and examined easily. I need to
>> know how the package was configured, when it was built, and of course, I
>> need the files to install. This is also necessary for building an ISO.
>> Acquiring and extracting a known config for a spell to build the
>> official ISO is necessary to have repeatable, reliable ISOs. As well,
>> this enables sharing of built packages across many machines easily,
>> especially if they're all built the same way. The binary package format
>> will need to contain enough information to be able to verify the config
>> of the binary package, and compare it to what was requested to be built.
> What's actually missing now? We store most of the state that was used when 
> the 
> spell started casting and you can even have config.log appended to the 
> compile 
> log (which is external, I know). Reuse is as simple as unpacking it, 
> exactly 
> what resurrect does.

They're stored separately, and not tied together. We're like 90% of the
way there, yes. I would just like to be able to
distribute/share/whatever one file with the config and such tied to it.

>  
>> Core grimoire:
>> --------------
>> We need to pare down the amount of stuff that is well supported. We
>> don't have enough manpower to take care of all the things. The Core
>> grimoire will provide binary packages of the things we support. Those
>> configs are what we support. This way we can guarantee that someone will
>> be able to get a working system fired up based on the configs we've got.
>> We should have no problem supporting configs for i686 as well as x86_64
>> eventually, but we can target only one for the beginning.
> Is there a buildbot planned, would everyone touching it need a clean chroot 
> for generation, would it be a two-step process or something else?

I'm not completely sure how to go about doing this, we'd have to see how
it pans out.

> How would it be versioned/developed? I guess it would replace 
> stable/rc/test 
> and all the others would remain in what currently test is.

It would be separate from the existing stable/rc/test setup, and we
could define a different workflow for it, when we update it and such.
And spells would have to meet certain criteria in order to be accepted
into this grimoire. Everything else would go in the existing grimoires
as they are, making it easy for things to be kept up to date for people
that want them updated while giving us a solid core grimoire that's
stable and reliable.

>> Updated init system:
>> --------------------
>> Simpleinint-msb works, but it's old and crappy. I'm a fan of systemd,
>> because it makes things amazingly easy to do, and is ridiculously fast.
>> It's also been adopted by several linux distros, and large projects.
>> However, that's not set in stone, we just need something newer, and
>> preferably something that other distros are using, so we can take
>> advantage of the work that others are doing as well.
> Hasn't IMP gotten far enough? IIRC we already have a working base (upstart, 
> systemd).

That might be true, but the grimoire still operates on the assumption of
simpleinit-msb, and no changes have been made to support IMP out of the box.

> 
>> Chroot based build process:
>> ---------------------------
>> For building binary packages, I want to take advantage of a chroot and
>> unionfs (or rsync and hardlinks or something.) Inspiration from this
>> page:
>> https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroo
>> t It's probably the sanest way to produce a package that we can ensure 
>> isn't
>> melding in dependencies we don't want and such. By ensuring that we build
>> things into binary packages as well, we can catch leaky
>> installs, or missing dependencies when we're building the chain of
>> packages. Additionally, it'll give the system itself protection from a
>> stupid installer doing bad things, or a partially failed install.
> castfs is ran in a chroot. I still see no reason why it couldn't be 
> extended 
> to also include the BUILD stage. Cleanliness can be ensured by the chroot 
> system itself (one of the previous goals) and removal of the new after 
> casting 
> (the same dance prometheus did).
> 
> Only a sane chroot setup can help with catching extra libraries being 
> linked 
> in or similar, but it may be a bit much to request everyone use it, I don't 
> know. For this "core" grimoire definitely, for the rest, I'm not so sure it 
> should be a requirement. Any packager using non-generic optimisations would 
> have to build the updates twice.

That's a valid point, I hadn't considered it that way. We should,
however, be able to allow people to establish for themselves a core
grimoire built using their non-generic opts easily, and thus only update
that core chroot when those updates come out. That would considerably
reduce the amount of double-building that was necessary. For the other
grimoires, it should be possible to build the package in the chroot, but
since we won't be distributing those packages, it won't matter what
optimizations people use, since it's on them to build it themselves.

> 
>> Declarative spell config:
>> -------------------------
>> Spell configuration needs to not be procedural. I should be able to say
>> "cast kde" and get a menuconfig style interface where I can toggle
>> things off and on and know what the effects of my selections are going
>> to be without having to restart the entire process again. I should also
>> be able to store a config to a file "Dave's KDE Desktop Config" and load
>> that in, and be notified of new options somehow. This is critical not
>> only to making it easier for people to construct systems, but to have
>> repeatable builds. When someone complains that their package doesn't
>> build, we can ask for their config, throw it in a chroot, and duplicate
>> the problem, either finding a patch, or finding out that their config is
>> simply broken. Finally, having stored configs allows us to package those
>> up with a binary package, and should you already have a binary package
>> with the proper config, you can just extract that rather than rebuild it
>> again.
> Configs are already stored, extractable and reproducible.

And reloadable? And I can update a spells config, and know what options
have changed, even within if/then logic? No, that's not possible.

> 
> As for the declarative part, some solution to be able to see and set 
> everything up front would definitely be cool. Especially for any test 
> scripting 
> without Expect.
> I don't see a way to make it completely declarative though, unless you 
> leave 
> the dependency resolution out of it or (re)generate all the possible trees. 
> :S

It'd be work to generate the possible trees, but I think it'd have
advantages, as I've detailed in other mails.

Thanks for your responses,
David Kowis

Attachment: signature.asc
Description: OpenPGP digital signature