Public SourceMage Discussion List

[Vlad Glagolev <stealth AT tiberian.ru>]

On Fri, 16 Jun 2017 13:25:37 +0200
Thomas Orgis <thomas-forum AT orgis.org> wrote:

> I applaud moves to counter our helpless situation regarding flaky
> upstream sources.
> 
> Am Thu, 15 Jun 2017 13:30:13 -0400
> schrieb Vlad Glagolev <stealth AT tiberian.ru>: 
> 
> > I have built a small service to redirect to up-to-date sources if
> > they're found on any of our mirrors. Currently they are:
> > 
> > - http://mirror.belltower.us/sourcemage/fallback/
> > - http://mirror.sobukus.de/files/sourcemage/fallback/
> 
> Hm, so this works well once I make
> 
>       http://mirror.sobukus.de/files/sourcemage/fallback/
> 
> a front to the full search over the source files I host instad of just
> the tiny amount of files in the sourcemage/fallback directory. You
> should be able to get just about every file via
> 
>       http://mirror.sobukus.de/$FILE
> 
> from my collection of upstream mirrors, including automatic downloads
> of everything in the grimoire (at least static source URLs not
> depending on fancy spell config).
> 
> To ease the thinking, I'll code a redirect of
> 
>       http://mirror.sobukus.de/files/sourcemage/fallback/$FILE
> 
> to do the same search over the whole repository if the file does not
> exist there. The benefit would also be that we can put a specific
> version of $FILE into the fallback directory and that will be preferred
> over any other version I have. The elephant in the room is that with
> re-releases, there is a set of $FILEs that have different versions.
> 
> > What it does is this:
> > 
> > 1. Gets the source file name
> > 2. Goes through the list of mirrors and performs HEAD against them
> > 3. On 200 OK it saves the file name and mirror in Redis with
> > expiration time of 1 week
> > 4. Right after #3 it 303-redirects to the corresponding mirror and
> > returns the URL to the client
> > 5. In case all mirrors return 404, it yields with 404 error as well
> > 6. Next time the same file is requested, the mirror is taken from Redis
> > quickly without any need to do the full hunt
> 
> To make this really robust, we should add versioning of files to this.
> At least we should be able to catch the case where the first found file
> is the wrong one and then the verify stage fails. If you remember only
> that one, but somewhere else (even on the same mirror) a correct
> version exists, we have permanent spell failure. This is not many
> spells, but it is more than a few were release file names are not
> unique.

Shouldn't it be caught on the mirrors themselves and not the origin
(download.sourcemage.org)? Otherwise our entry point will get a ton of
useless traffic trying to download every single source possible and
verifying it against the checksum in the db.

> 
> I have it on my schedule to add a checksum-based query to the mirror
> (once I figured out versioned storage with the same file name) … like
> 
>       
> http://mirror.sobukus.de/checksum/sha512/7b77b543bd6a127095c6a6a2e7f9b5933f5a391734cc6dcbc160f786008afcd74646a7c304e1be682042221648b0fcae7efafda4bbe66ba30a7021740abddc34
> 
> That would ensure that we really get the correct files. For that to
> work, we'd need to add hashes to all sources, even if they're verified
> by GPG sig. We're playing catch-up here with other distros that mirror
> upstream sources since ages, also with hash sums.
> 
> > the logic above is valid for fallback at download.sourcemage.org. it
> > also means mirrors can be not in sync and have different sources until
> > we want to sync them and have enough capacity to store data.
> 
> Do you intend to eventually populate the fallback directory with _all_
> sources we ever used?

Ideally -- yes. However I'm not sure 50 gigs I have would be enough for
that. The idea was to have all grimoires from 0.4 to 0.63 added to the
Codex and use a script powered by 'summon -g <grimoire> <spell>' to
fetch all tarballs/patches/signatures.

> Could become a rather huge directory, and would
> mean that I indeed should stop syncing that one and rather use the
> redirect to the regular upstream file database I have on my mirror
> (also with sync scripts separate from what's in the grimoire).

Right. We probably need to come up with a smarter automated solution to
rely on upstream mirrors as much as possible on the mirror servers
themselves to save net i/o.

> 
> > It's only 50 LoC
> 
> 50 lines? Including comments? ;-)

Haha, mostly newlines, no comments yet.

> 
> >, but I wanted to polish it with config file support
> > (where you could configure mirror list and other stuff like timeouts
> > and expiration time) before submitting it to our wand repository.
> > 
> > Its dependencies are:
> > 
> > - python
> > - bottle
> > - flup (for FastCGI)
> > - redis-py (could be even faster with hiredis-py attached to redis-py)
> > - requests
> 
> By the way … requests is an awfully generic name … we should really
> prefix all these scripting language/runtime libs/modules to avoid
> clashes. Things should be named py-$something, perl-$something … pkgsrc
> has this neat scheme with the python prefix: There is one source
> package named py-blabla and you can install py27-blabla and
> py34-blabla, each for the respective python version.

Totally agree. It'll need _a lot_ of deprecational work though, and I'm
not sure it's worth it _right now_. I haven't checked what other
distributions decided on 2-3-4-5 Python versions installed at the same
time and duplication of modules for each, but this is something that we
really need to keep in mind. Because from what I see in the industry
Python 2 is not gonna leave any time soon.

> 
> > Now I'm trying to collect sources from my systems (which are about
> > 20 and all running stable) and put them on Belltower. The next step
> > would be to try to collect all sources for current stable grimoire
> 
> Back when I did the dance of rebuilding and updating out chroots (from
> 0.60 on I think), I had stuff pushed to my mirror already. It should be
> rare cases where files are missing. The question is … does anyone want
> to mirror _my_ mirror? It's around 800G sources so far … I don't have
> deployment tools right now … am just tinkering with a slave mirror that
> pulls the the scripts from my svn repo and the collecton of URLs via
> git+http (lazy serving via apache). Hm. The URL collection itself is a
> 164M git repo. And I still need to fix the way the CPAN mirrors are
> handled (currently each run of the update script adds new URLs with
> different mirrors, hence I don't push that part to git yet). Mirroring
> mirrors can become complex.

800 gig? Woah, that's huge. I won't have such space in the cloud until
mid-autumn, I think, when/if I get a dedicated box with 2-4 TB disks.

> 
> Anyhow, I'll turn
> 
>       http://mirror.sobukus.de/files/sourcemage/fallback/
> 
> into the redirect thing now so that I don't have to educate folks about
> not using that instead of a full search.

Thank you, Thomas!

> 
> Ah, regarding protocols: My mirror talks HTTP and HTTPS over IPv4 and
> IPv6, of course. I didn't bother with FTP, nor rsync. Mirroring of my
> mirror is currently handled by checking out the same scripts and the
> URL database, then doing the normal download which pulls sources from
> upstream, optionally with my mirror set as a primary source to try
> first. Especially for an initial sync, trying the original servers
> first is nicer. At some point my ISP might get edgy.

Great, we should probably create a table on this page[0] with all
features supported to make it clear for everyone what download options
are available for every mirror.

> 
> 
> Alrighty then,
> 
> Thomas

[0] http://sourcemage.org/Mirrors

--
Vlad Glagolev <stealth AT tiberian.ru>

Attachment: pgpEOaLDdk579.pgp
Description: PGP signature