Public SourceMage Discussion List

["Remko van der Vossen" <wich AT stack.nl>]

> On Sunday 22 of March 2009 13:27:50 Remko van der Vossen wrote:
>> Hi everyone,
>>
>> As some of you may know I'm working on a new and improved script to find
>> missing library dependencies, which is much faster than the old
>> findLibDepends.sh and uses readelf to read the dynamic section instead
>> of relying on ldd which also lists second and higher level dependencies.
>>
>> The script is not yet fully up to working order, I still need to handle
>> sorcery providers correctly and I still need to get my index updates
>> watertight, though using the script I have made a list of all missing
>> dependencies of the spells installed on one of the boxes I maintain. I
>> have put this list up on the wiki[0] so that we can start resolving
>> these missing dependencies, thus improving the quality of our dependency
>> tree which is currently lacking.
>>
>> I would request everyone who has a bit of time to spare to look at the
>> list, pick one or more spells and resolve its missing dependencies. Of
>> course doing so requires checking whether the dependency is optional or
>> not and correctly adding the dependencies to the DEPENDS files. There
>> are bound to be some optional dependencies which cannot be configured
>> through a package's configure script or similar for which we would need
>> to file upstream bugs.
>>
>> I hope working together we can all help to improve the quality of our
>> dependency tree, hopefully lowering the possibility of breakages
>> significantly.
>
> Would it be hard to generate a subset list of spells excluding missing
> dependencies provided indirectly? Those are the undoubtedly missing
> dependencies (once you sort out providers) and have the highest impact.
> Seeing how the list is long, I think it would be good to prioritise on
> them and whatever needs to be done for libxcb.

It should be doable, but I'd need a fully qualified dependency listing,
which shoould be easy enough to generate, though one question here is how
you qualify indirect dependency chains that contain optional dependencies
at certain links whereas the true dependency between the ends of the chain
might be a non-optional dependency... I hope I can look into it tomorrow
night, but I can't make any promises just yet.

As an aside, I was musing about the possibility of a way to have sorcery
track the additional (unneeded, but real) dependencies imparted by libtool
and friends. Let a,b and c be spells such that a depends on b and b
depends on c, where depends can either be a regular dependency or an
enabled optional dependency, and b.la contains a "dependency c.so" leading
to a.so having a "needed b.so" as well as a "needed c.so". However there
is no direct dependency necessary from a to c, i.e. a.so uses no symbols
from c.so.

I think the most graceful solution would be to let spell b specify that
any dependees will implicitly depend on c as well. When spell b gets cast,
it is recorded in the tablet that c is an ancillary spell. Then when spell
a gets cast, the ancillary spells listed for spell b in the tablet are
recorded as ancillary dependencies of spell a in the tablet.

So, in short, if a spell installs a .la file that includes dependency
listings on other spells we indicate these in the spell somehow such that
they end up in the tablet allowing us to track those kind of dependencies.

There is however one drawback to this; every dependee of spell b will
automagically have an ancillary dependency on c, even though it might not
even need it. For instance if spell a only uses a particular library from
spell b, a will also get all ancillary dependencies incurred by the other
libraries installed by b. Also if spell a actually does things right, i.e.
uses --as-needed when linking, explicitly links only to the libraries
required, or whichever other way, it will still get the ancillary
dependency on c.

An alternative to this would be for us to explicitly list all ancillary
dependencies, i.e. in the DEPENDS of spell a we would have an
ancillary_depends b c, indicating that spell a has an ancillary dependency
on c, subject to the condition that b is enabled dependency of a, and c is
an enabled dependency of b, where an enabled dependency is either a
regular dependency or an enabled optional dependency. Of course this would
mean quite a bit more work on our end, but will reduce the number of
ancillary dependencies if and when we get upstream so far as to improve
their building process, linking only with needed libraries.

An open question is whether this mechanism would need to cascade
recursively. I don't think it does, as libtool only looks at the .la files
of the libraries specified and doesn't do this recursively as far as I am
aware, but I'm not sure on that.

Okay, this turned into a bit of a bigger aside as I was originally
thinking... As is obvious this is quite a complex solution and would
involve quite a bit of work to implement correctly in both sorcery and
codex, but I believe there isn't really any way to do it much simpler yet
still correct. If anyone has ideas to simplify this whole mess, please
share.

Remko van der Vossen.