Public SourceMage Discussion List

[Remko van der Vossen <wich AT yuugen.jp>]

On Tue, Aug 26, 2014 at 08:17:42PM -0400, Sukneet Basuta wrote:
> On Sat, Nov 9, 2013 at 12:41 PM, Remko van der Vossen <wich AT yuugen.jp> 
> wrote:
> 
>     I think the only proper way to solve this is to have sorcery collect the
>     link dependencies from the installed spells, in the case of ELF dynamic
>     objects, the spells that provide each of the listed DT_NEEDED entries in
>     the dynamic objects, even if the listed entry is not directly used, as
>     building anything that links to that object will fail if the dynamic 
> object
>     in the entry does not exist anymore. Then, when casting anything, 
> sorcery
>     needs to know about incompatible upgrades, build a tree of link 
> dependent
>     spells ignoring dependencies in DEPENDS for this tree, and immediately 
> cast
>     those spells in the correct order. We still have to see about resolving 
> any
>     cyclic dependencies if there are any, and it should be seen whether 
> these
>     spells should be cast from the grimoire or from tablet. Then and only 
> then
>     should sorcery continue casting its normal queue, meaning that the link
>     dependency triggered queue should be separate from the normal 
> cast/install
>     queue.
> 
>     This would of course mean a big change in sorcery though and given the 
> lack
>     of time and motivation for any big sorcery work, I don't think this is
>     going to happen anytime soon.
> 
> 
> Honestly, I'd love to get around to implementing this, but I don't have the
> time currently.

While it would still be worthwhile to create this it is;
1) not easy at all
2) far less pressing when we make as-needed the default

Why is it difficult to do right?
1) this only covers ELF DT_NEEDED, there are many more ways in which
   dependencies can fail which are not covered at all, trying to cover
   everything is extremely difficult
2) sonames are, contrary to the original idea, not unique, many packages
   use wrapper scripts that set LD_LIBRARY_PATH or other nonsense that 
   can mess everything up

When we use as-needed we already get rid of >90% of the problem as most
of the direct dependencies missing from DEPENDS are these indirect
dependencies which are not directly used. This makes the dependency
trees based on DEPENDS info much more accurate. The missing dependencies
are now mainly;
1) basesystem dependencies that we choose not to include in DEPENDS, a
   decision we should change in my opinion
2) indirect dependencies which are also direct dependencies.

This means we can still have cast ordering problems and it means that we
will not be able to get a full list of spells to recast upon ABI
incompatible spell upgrades. However these problems are greatly reduced
compared to the situation where we did not use as-needed. To address the
problems we should, in my opinion, start working on better quality
DEPENDS files. I had been working on tooling to find missing
dependencies, but I soon ran into the whole libtool hell as well as the
difficulties in finding the right library to which something actually
links at runtime. If we however switch over to as-needed completely I
would be interested in pursuing this further and develop tooling that
allows us to improve the quality of out DEPENDS files. Then, when we
have good quality DEPENDS files we don't really need to record link
dependencies in tablet like I suggested long ago.

Regards, Remko