Public SourceMage Discussion List

[Thomas Orgis <thomas-forum AT orgis.org>]

Am Fri, 16 Jun 2017 13:32:11 -0400
schrieb Vlad Glagolev <stealth AT tiberian.ru>: 

> On Fri, 16 Jun 2017 13:25:37 +0200
> Thomas Orgis <thomas-forum AT orgis.org> wrote:

> > To make this really robust, we should add versioning of files to this.
> > At least we should be able to catch the case where the first found file
> > is the wrong one and then the verify stage fails.
> 
> Shouldn't it be caught on the mirrors themselves and not the origin
> (download.sourcemage.org)? 

Indeed … for files coming from the grimoire, the mirror could check
itself. It's a bit tricky though when you don't have every version of
the grimoire at hand for the whole history of files that clients might
want. Also, there are time frames where the grimoire has the old hash,
upstream changes file, we update grimoire … what used to be the right
file is now not the right file anymore. My mirror so far does not
handle that case automatically: If it has a file, it does not try to
download it again.

Once I add checksum support, the grimoire will be one first source of
them and the server can know if it got the correct one or should try
again or from a different source.

> verifying it against the checksum in the db.

That checksum db … it only exists in some sense of existence.
Especially the history is not that structured. And then we have files
without checksums. When there is a spell that uses GPG with the sig as
unchecked SOURCE2 … you don't even realise that you get a different
re-released file or an older previous same-name release file! That's
why I say checksums should always be added (at least for the sig in
that case). I shy away from an automatism checking the gpg sigs … as
that depends on having up-to-date keyrings available and complicates
matters a lot more than hash checking.

> > Do you intend to eventually populate the fallback directory with _all_
> > sources we ever used?  
> 
> Ideally -- yes.

Theroretically, filesystems should be OK with many files in one dir
nowadays … but the tradition says that you try to partition things a
bit. I opted for at least using the spell categories as substructure in

        http://mirror.sobukus.de/files/grimoire/

to keep things manageable. With all files in one basket, I presume we
would at least do some pruning old old stuff from time to time.

Since my mirror is specially built to preserve upstream for eternity, I
try to use a structure that promises to scale a bit better. Note that
files/grimoire and files/sourcemage is just one part. What I initially
started with is

        http://mirror.sobukus.de/files/src/

with URL scraper scripts from project websites, projects being added
manually by me. This includes things like

        http://mirror.sobukus.de/files/src/linux/

with a mere 269G of all files I found worthwhile from kernel.org . I
made separate trees for other big things like gnu, gnome, cpan (fun
fact: GNOME is bigger than KDE, CPAN, and Gnu combined, with 140G vs.
23G+26G+64G).

> However I'm not sure 50 gigs I have would be enough for

Yeah, I see that I got a slightly different approach here;-) I got fed
up with unreliable upstream and wanted to solve that problem for me
once and for all. I rent a physical server since many years now, for
mail and websites. The last time I just made sure I got one with two 3T
disks in there, for not much more monthly fee than I'd have anyway.
Downside of this setup is that there is no backup for the source
archive per se … but I got that other mirror set up at work on a much
more redundant storage device. And probably I'll make copies to spare
disks lying around from time to time. (I feel so old, actually taking
care of my data myself.)

> that. The idea was to have all grimoires from 0.4 to 0.63 added to the
> Codex and use a script powered by 'summon -g <grimoire> <spell>' to
> fetch all tarballs/patches/signatures.

Indeed, I could extend that. For now I got a set of grimoires and
extract the URLs via `summon -g $grimoire --print raw $spell` … store
those and fetch them in a separate step. I could add the old grimoires
to the chroot doing the summon without much hassle. And then there is
Ismael's awk scripting to directly extract spell URLs from grimoires
without summon. Since performance does not matter for my process, I
opted for the safer full-blown method.

But getting back at the fully populated fallback directory (tree):
Since this stuff is in a larger context on my server, I'd need to do
some special treatment there, avoiding copies of all the files. So far
I avoid copies of files by placing a symlink instead of downloading
twice. Would I check for hashes, I'd have to download from each source
at least once, I guess …

Since I actually mirror the process of extracting sources from the
grimoires, I should get away with omitting the fallback directory and
have my redirect magic do its work, assuming that I got a mirror of the
primary source already.

> Right. We probably need to come up with a smarter automated solution to
> rely on upstream mirrors as much as possible on the mirror servers
> themselves to save net i/o.

Luckily, SMGL is a really small distro (user base) and people are not
that interested in old sources in general, so that the load on my
server is of no concern. I only get warnings when it downloads things (new
releases of huge stuff) or I sync to my other mirror.

> > By the way … requests is an awfully generic name … we should really
> > prefix all these scripting language/runtime libs/modules to avoid
> > clashes.

> Totally agree. It'll need _a lot_ of deprecational work though, and I'm
> not sure it's worth it _right now_.

A clear rule for new spells would be a start. I'm no fan of
deprecation. But if we do it, a one-time big switch might be a good
idea, so that the annoyance doesn't extend too much. We got other
things to do right now.

> Great, we should probably create a table on this page[0] with all
> features supported

Yes. Hm, I did activate an account for that site some time ago, didn't
I? Hm … didn't work. First firefox complained about the security and
then my password is not accepted (anymore?). I guess I should also add
content to the fallback directory index page on my mirror to give a
hint about you also getting files that are not in there …


Alrighty then,

Thomas

Attachment: pgpcnukhV_qb5.pgp
Description: Digitale Signatur von OpenPGP