Public SourceMage Discussion List

[Thomas Orgis <thomas-forum AT orgis.org>]

Am Thu, 1 Oct 2020 22:39:47 +0200
schrieb Ismael Luceno <ismael AT iodev.co.uk>: 

> If you build project A and B using the same target directory it re-uses
> anything it had built before.

OK, that would be the goal, then.

> > I guess a .crate can contain sources or binary libs, and you don't know
> > from the outside?  
> 
> AFAIK, always sources.

Nope. Winapi is also a crate:

$ tar -tzf 
registry/cache/github.com-1ecc6299db9ec823/winapi-i686-pc-windows-gnu-0.4.0.crate
 |  head
winapi-i686-pc-windows-gnu-0.4.0/Cargo.toml.orig
winapi-i686-pc-windows-gnu-0.4.0/Cargo.toml
winapi-i686-pc-windows-gnu-0.4.0/build.rs
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_aclui.a
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_activeds.a
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_advapi32.a
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_advpack.a
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_amsi.a
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_api-ms-win-net-isolation-l1-1-0.a

$ tar -xzf …
$ file winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_advapi32.a 
winapi-i686-pc-windows-gnu-0.4.0/lib/libwinapi_advapi32.a: current ar archive

It's a question how common is that.

> Only to remove pinned versions, the .lock files, so that you don't get
> different versions of the same library to different "final" binaries.

I see the idea, but aren't the pinned versions what keeps the house of
cards on unstable APIs from collapsing? You want to be responsible for
the foundations?

> The problem is we would then need to provide packages and duplicate the
> dependencies in the DEPENDS file for every version mentioned in a .lock
> file.

Yes. No fun. It goes back to the question if we want to integrate rust
package management for real or not … I mean, also with cpan, we only
handle a very little part of it.

> You need to compile with the "approved" version of rust for each app.

So you want to cater for differing versions of rust itself, but hope to
get away with one version for each dependency? Looks like selective
optimism to me:-/

> > (Heck, how big would be a dump of CTAN, CPAN, CRAN, pipy, npm, … each?
> > Maybe separate grimoires. Or just storing info on just-cast spells on
> > demand.)  
> 
> It is a lot of duplication and we can do very little about it... other
> than automate the heck out of the conversion, no matter how it's
> implemented.

And the question of how much we want to atomatically put into the
grimorie. Is it practical to make a big sweep and have thousands if not
millions of automatically generated spell files? Well, I guess it could
be automated in a way that all the data is there, but the actual spell
files are created on-demand, as I see lots of redundancy. I guess the
list of source hashes will be rather big already. 

- CPAN: 200k modules
- CTAN: 6k
- CRAN: 16k
- npm: ??? Unable to get a number through mere web browsing.
- pypi: 265k
- crates.io: 47k (that was quick, eh?) … (Also, 4 billion downloads
  claimed … easy, if each new deployment re-downloads sources.)

So, we're indeed approaching a million spells. Of course, only a tiny
amount of those are relevant. But when we want to package the open
source ecosystem, we'd need to give people access to all.

Of course, the more realistic approach is to package only that tiny
subset that we need to get some relevant functionality for the system.

But it would be kinda nice if installations from CPAN would be tracked
in sorcery, and even versioned with the grimoire offering a snapshot of
the various registries. And some heavy mirroring like

        https://mirror.sobukus.de/files/cpan/

That's 28G, by the way. It's 1137 categories. 173249 files, including
duplicates due to versions. Hm. That doesn't really add up to 200k
modules. I'm not sure if others are counting differing versions. The
file and URL lists amount to 1M gzipped, 10M uncompressed. Not that
much data, after all. I  guess we'd need to version the uncompressed
package indices in the grimoire. Or we have a spell named rust-index
(cpan, etc.) that installs a compressed archive of the index we mirror
into the system. Sorcery then creates spells on-the-fly from that. That
way, those rather strictly packaged external units don't even bloat the
grimoire. Would need some magic to write DEPENDS with spell names that
will exist after sorcery created the virtual grimoire for them. But I
sense a concept here.

Versioning all the package registries in our grimoire git in detail
seems like a waste.

With such a fully automatic approach, we would forgo the option of
patching up things individually. But I guess we can get away with that
for this higher-level stuff. We'll just globally decide about a
separate prefix and mechanisms that are valid for all. Special
packages that should install binaries into /usr/bin, for example,
could/should (?) still be explicitly written as spells (like cbindgen
now) that depend on the virtual rust/whatever grimoire.

I could start trying to mirror those other registries, btw. Not all of
them are easily accessible via web scraping (npm?!).

> The problem isn't the design of the system itself but the fact it's
> meant to operate on it's own and always connected to the net.

… and the question if we are brave/stupid enough to battle that.
Really, what do other distros do?

> Yes, cargo provides an offline mode, but it still requires you to
> connect first to a repository to get everything, so you would need
> to duplicate that to fully work offline.

… a fake repository served by sorcery.

> 
> In the end, you need an alternate implementation of cargo to make all
> of it work.

An alternative of all the net-facing stuff. Not the building itself. We
don't have to do _all_ the work. And we could upstream changes to
support offline mode better.

> Maybe a way would de to edit the Cargo.toml files to provide local
> paths to the dependencies, and in that way avoid the registry, which
> means the CARGO_HOME could be readonly maybe.

Since it's structured (better than for go, at least the early days),
such an approach should be feasible. What we need to fake is the
updating of that CARGO_HOME with built to-be-dependencies …


Alrighty then,

Thomas