Public SourceMage Discussion List

[Thomas Orgis <thomas-forum AT orgis.org>]

Am Thu, 1 Oct 2020 18:46:31 +0200
schrieb Ismael Luceno <ismael AT iodev.co.uk>: 

> Is it only the sources though?
> 
> If you have A and B depending on C, C only gets built once...

Well, let's see what's inside 
http://sobukus.de/rust/cbindgen-0.14.6-cargo.tar.gz:

.cargo$ find . -type f|xargs file -iN |grep -v winapi|grep -v text
./.package-cache: inode/x-empty; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/serde-1.0.104.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/rand-0.7.2.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/redox_syscall-0.1.56.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/clap-2.33.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/vec_map-0.8.1.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/syn-1.0.11.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/remove_dir_all-0.5.2.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/rand_core-0.5.1.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/proc-macro2-1.0.6.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/rand_hc-0.2.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/log-0.4.8.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/getrandom-0.1.13.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/itoa-0.4.4.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/unicode-xid-0.2.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/ansi_term-0.11.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/unicode-width-0.1.7.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/strsim-0.8.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/libc-0.2.66.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/serde_json-1.0.44.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/toml-0.5.5.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/cfg-if-0.1.10.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/ryu-1.0.2.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/quote-1.0.2.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/unicode-segmentation-1.6.0.crate:
 application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/c2-chacha-0.2.3.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/rand_chacha-0.2.1.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/wasi-0.7.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/tempfile-3.1.0.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/serde_derive-1.0.104.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/heck-0.3.1.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/atty-0.2.13.crate: 
application/gzip; charset=binary
./registry/cache/github.com-1ecc6299db9ec823/bitflags-1.2.1.crate: 
application/gzip; charset=binary
./registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.6.0/src/tables.rs:
 application/octet-stream; charset=binary

That's a plain text source file. The File tool is confused.

./registry/src/github.com-1ecc6299db9ec823/heck-0.3.1/no_step_on_snek.png: 
image/png; charset=binary
./registry/src/github.com-1ecc6299db9ec823/syn-1.0.11/src/keyword.rs: 
inode/x-empty; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.last-updated: inode/x-empty; 
charset=binary
./registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack-eab3369a28e264fdf3bc419d070cf1c3731dfa56.idx:
 application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack-eab3369a28e264fdf3bc419d070cf1c3731dfa56.pack:
 application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/3/l/log: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/3/s/syn: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/3/r/ryu: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/se/rd/serde_json: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/se/rd/serde_derive: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/se/rd/serde: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/c2/-c/c2-chacha: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/un/ic/unicode-width: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/un/ic/unicode-xid: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/un/ic/unicode-segmentation:
 application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/wa/si/wasi: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/he/ck/heck: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/pr/oc/proc-macro2: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/li/bc/libc: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/cl/ap/clap: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/ve/c_/vec_map: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/cf/g-/cfg-if: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/ra/nd/rand: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/ra/nd/rand_core: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/ra/nd/rand_hc: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/ra/nd/rand_chacha: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/at/ty/atty: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/te/mp/tempfile: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/ge/tr/getrandom: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/an/si/ansi_term: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/st/rs/strsim: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/bi/tf/bitflags: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/qu/ot/quote: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/it/oa/itoa: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/pp/v-/ppv-lite86: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/to/ml/toml: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/re/mo/remove_dir_all: 
application/octet-stream; charset=binary
./registry/index/github.com-1ecc6299db9ec823/.cache/re/do/redox_syscall: 
application/octet-stream; charset=binary

Most files are plain-text .rs sources (grep -v text). Then, there's the
winapi binaries that shouldn't bother us, I guess. That leaves those
crates. I imagine that such crates could be what we store in a
system-wide registry … but it would help if we had anyone here who
actually uses rust to comment on how things work. Well, back to those
crates.

$ tar -tzf ./registry/cache/github.com-1ecc6299db9ec823/serde-1.0.104.crate
serde-1.0.104/Cargo.toml.orig
serde-1.0.104/Cargo.toml
serde-1.0.104/LICENSE-APACHE
serde-1.0.104/LICENSE-MIT
serde-1.0.104/README.md
serde-1.0.104/build.rs
serde-1.0.104/crates-io.md
serde-1.0.104/src/de/from_primitive.rs
serde-1.0.104/src/de/ignored_any.rs
serde-1.0.104/src/de/impls.rs
serde-1.0.104/src/de/mod.rs
serde-1.0.104/src/de/utf8.rs
serde-1.0.104/src/de/value.rs
serde-1.0.104/src/export.rs
serde-1.0.104/src/integer128.rs
serde-1.0.104/src/lib.rs
serde-1.0.104/src/macros.rs
serde-1.0.104/src/private/de.rs
serde-1.0.104/src/private/macros.rs
serde-1.0.104/src/private/mod.rs
serde-1.0.104/src/private/ser.rs
serde-1.0.104/src/ser/impls.rs
serde-1.0.104/src/ser/impossible.rs
serde-1.0.104/src/ser/mod.rs
serde-1.0.104/src/std_error.rs
serde-1.0.104/.cargo_vcs_info.json

Looks like sources to me. Others seem to also contain .rs sources.

Cargo gets all dependencies, then builds them in some order … then
lumps everything into the target binary. Where does your A, B, C
picture come into play?


> rlibs are static and cdynlibs are dynamic libs with static dependencies
> built-in.

Ah, cdylibs. No n.

        https://doc.rust-lang.org/reference/linkage.html

I guess a .crate can contain sources or binary libs, and you don't know
from the outside?

> And the problem is we need to tell which versions are used to apply
> policies.

> 
> If you start pinning commits from git repos then it's chaos.

To be clear: Are you intending to take over control of dependencies
from rust packages? Apparently so …

> We need to force versions for sure (replace/edit .lock files), because
> upstream often can't be trusted to make the right choices, but we need to
> be able to install multiple versions too.

You trust us to make better choices? Sure, often the list of deps will
be outdated. But if we're the only ones messing around, our builds will
differ from everything the people use out there. Nobody tests our setup
(not even as little as it's the case for 'normal' stuff right now).

Anyone got a hint on how other distros handle this? Is rust stuff
integrated at all? What about go?

I don't see a conflict with the versions … the directories/crate names
contain version numbers. Do rust packages know compile-time options,
though? Well, no issue if we have only source crates. It could be fun
working with a central CARGO_HOME, though. We'd need to use it as
upstream to pull copies from into the spell's CARGO_HOME, as things are
built there. See where we went from the craze about out-of-tree builds
before? ;-)

> Not a problem if we let cargo manage it under a special target
> directory, I guess, one per rust version.

Different versions of rust?! More madness?

> rust-{ver}-{pkg}-{ver} spells
> 
> And we copy them all at each upgrade; the rust spell should also be
> versioned.

Well, a section could be handy for that … with the addition that the
section name becomes part of the spell identification. But without
that, you need to automate the hell out of this. Well, like Cargo
automates it right now. Should the spells even exist as classic spell
files? Should sorcery generate them on the fly? Same for CPAN and
friends actually?

> Looking more in depth it looks like we need to provide a synthetic
> CARGO_HOME.

I have doubts that we manage that without a person onboard who

a) knows, uses, (loves?) rust.
b) is willing to violate cargo like that.

Who steps up?

> This will be produced at each PRE_BUILD from the downloaded sources.
> 
> Then I guess the spells only need to keep build artifacts around (that's
> CARGO_TARGET_DIR), and somehow workaround cargo's dependency management,

Would be nice to have things built once only that way. This does put us
roughly back into the comfort level of static libraries (with us having
to write a good deal of the linker).

In generall: With all these existing ecosystem packaging systems, I
feel it is a huge waste of time if we spend time managing spells
mirroring the respective package databases. Grimoire releases could
contain dumps of those databases though, to have reproducible builds
from mirrored sources even if those database services are gone.

(Heck, how big would be a dump of CTAN, CPAN, CRAN, pipy, npm, … each?
Maybe separate grimoires. Or just storing info on just-cast spells on
demand.)



Alrighty then,

Thomas