Public SourceMage Discussion List

[Remko van der Vossen <wich AT yuugen.jp>]

SUMMARY

What I feel should be done;

- Everyone should move to as-needed aggressively; switch on as-needed
  and sorcery rebuild your systems
- Stealth should push test into a new stable-rc and prepare delivering a
  new stable grimoire
- Devel branch owners should create a good UP_TRIGGERS for spells with
  ABI incompatible changes compared to test and prepare for pushing to
  test after stable-rc has been prepared

Who is with me?






THE WHOLE STORY

Everyone,

As we all know SMGL has been stagnating the last couple of years, we've
been holding off on pushing from test to stable, we have multiple devel
branches which have not been pushed to test in a long time, there's been
practically no sorcery or cauldron development in a long time, we could
go on and on.

This has had several reasons but the two main ones are;
1) our inability to deal properly with ABI incompatible changes 
2) lack of time/motivation of SMGL developers

ABI incompatible changes have been a major headache for us, which, in
the past has led to complete system breakages for many people and has
turned a lot of people away from SMGL. Lately we've been dealing with
the problem by basically not dealing with it at all, we've kept ABI
incompatible changes in separate development branches without any way
forward to get these newer versions into test and ultimately into
stable. Now people are dissatisfied that everything is ancient. We need
to break this pattern, we need a solution to the actual problem.

In order to deal with the problem we need accurate and complete
dependency trees, we need a way to trigger recasts of dependers upon ABI
incompatible changes and we need a way of automatically breaking cyclic
dependencies. Realizing all of this takes a lot of effort, effort that
the SMGL developers as a group are not able or willing to spend, at
least in the short term. Therefore the time has come to cheat as much as
we can and to bite some bullets where we cannot.

The first problem we have are wildly inaccurate dependency trees. These
inaccuracies lead to casting order problems as stuff that depends on
some spell X gets cast before spell X. The problems are exacerbated in
cleanse as cleanse will not try to fix stuff in the right order causing
either outright failure or the need for many iterations of cleanse.

Why are our dependency trees so inaccurate?
1) we choose to omit dependencies that are covered by basesystem
2) we don't check our DEPENDS files for correctness, as long as it
   compiles we assume it's correct
3) our DEPENDS files are based on direct usage dependencies, not on link
   dependencies, but casting order needs to be based on link dependencies

How can we resolve this problem? We cheat! We remove as many link
dependencies as we can with as-needed, which makes our dependency tree
much more accurate without changing our dependency tree, yay for us.
Some years ago I didn't want to go the as-needed route as it means
making a choice for our users, which we generally regard as a bad thing.
But as I've been looking at the problem from time to time, seeing the
vast scale of the problem and seeing the general stagnation and apathy
within SMGL leads me to believe that we have no other recourse.

Switching to as-needed already takes care of a very, very large part of
the dependency tree problem. We should still work on improving our
DEPENDS files, but that can now take a back seat to getting ABI
incompatible updates into test and finally stable.

The next problem we have to tackle is triggering recasts of dependers
upon ABI incompatible changes. Why can't we do this? Because we don't
have a good way to get all dependers as we have a wildly inaccurate
dependency tree. Additionally we do not have any automated ways in
sorcery to detect ABI incompatible changes, nor do we have an automated
way to trigger recasts of all dependers.

So, what do we do about this problem? Again, we cheat! We find all
dependers, not by our dependency tree, but by checking all installed
spells for dynamic ELF objects (that is both binaries and shared libs)
that link to the library which has ABI incompatible changes. We still
need to find ABI incompatible changes ourselves however, although
http://upstream-tracker.org/ can help with that. We already know quite a
few; libpng, icu, libxcb, libpthread-stubs, openssl, linux-pam to name
just a few. Also we'll need to trigger the recasts ourselves using
UP_TRIGGERS. All in all it isn't that much work though, we're not going
to create static lists of all dependers, we only have to create an
UP_TRIGGERS once that does all the work and then just make sure that
that UP_TRIGGERS does its work whenever there is an ABI incompatible
update. An example of such an UP_TRIGGERS can be found in the libpng
spell in devel-libpng, (not the devel-libpng-1.6 branch.)

Lastly we come to the cyclic dependencies. Cyclic dependencies are a
major pain in the ass for which we really need something in sorcery to
deal with it automatically. Unfortunately this isn't going to happen on
short notice and holding out for it to happen will likely result in
another one or two years of stagnation ending in yet another call to
action which results in nothing. So, I propose we start biting some
bullets and just get on with it. It is worth noting though that the
dramatic reduction in link dependencies with as-needed has made
identifying and resolving cyclic dependencies during upgrade problems
much easier and might even have reduced the number of cyclic
dependencies themselves.

All of this will, I think, give us a reasonable way of dealing with the
problems at hand. Users will still have to deal with the odd problem
during upgrades, but they should be reasonably manageable to the point
where we can actually write down some advice for specific upgrades in
order to deal with some known cyclic dependencies for example. So, how
do we go about it all? I propose the following sequence of steps.

1) We have to get as-needed everywhere. Without as-needed we have no
hope in hell of getting any reasonable casting orders and thus always
end up in upgrade hell. So, to everyone I say, get latest devel sorcery
on your box, (maybe we should make a new stable sorcery,) switch on
as-needed in your optimization options, and essentially; sorcery rebuild
your box. Yes, the sorcery rebuild is necessary, as you would still have
all the problematic link dependencies if you don't. To everyone who
doesn't enable as-needed; fine it's your own choice, but be it at your
own peril.

2) We have to get current test into a new stable-rc. Current stable is
ancient, current test is more like our old stable and the various devel
branches are like our old test. We need to shift gears and get back to
what stable and test were intended to be. So, let's get test into a
stable-rc which we can actually test once we have all our systems
as-needed-i-fied. If all is well, good, if we encounter ABI
incompatibility problems we can deal with them as described above.

3) We need to get the devel-* branches up to scratch so that we can get
them merged into master after a new stable-rc has been created. Whether
we all merge them at once and go for a big bang or whether we do them
one by one between successive stable-rc cycles we should still discuss.
What does it mean to get them up to scratch? The ABI incompatible
changes need to be identified and good UP_TRIGGERS scripts need to be
created as described above. Then upgrading from master to the devel
branch has to be tested on a few as-needed-i-fied systems and major
manual steps necessary to upgrade should be recorded.

As for the devel branches, I would like to see the following devel
branches merge into master;

devel-icu
devel-libpng
devel-xorg-modular

Are any others blocking due to ABI incompatibility issues?

I'll take care of devel-libpng, who will take ownership of the other
two?

Regards, Remko