Skip to Content.
Sympa Menu

sm-discuss - Re: [SM-Discuss] Source Mage mirroring and infrastructure status p.2

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: Thomas Orgis <thomas-forum AT orgis.org>
  • To: sm-discuss AT lists.ibiblio.org
  • Subject: Re: [SM-Discuss] Source Mage mirroring and infrastructure status p.2
  • Date: Fri, 16 Jun 2017 13:25:37 +0200

I applaud moves to counter our helpless situation regarding flaky
upstream sources.

Am Thu, 15 Jun 2017 13:30:13 -0400
schrieb Vlad Glagolev <stealth AT tiberian.ru>:

> I have built a small service to redirect to up-to-date sources if
> they're found on any of our mirrors. Currently they are:
>
> - http://mirror.belltower.us/sourcemage/fallback/
> - http://mirror.sobukus.de/files/sourcemage/fallback/

Hm, so this works well once I make

http://mirror.sobukus.de/files/sourcemage/fallback/

a front to the full search over the source files I host instad of just
the tiny amount of files in the sourcemage/fallback directory. You
should be able to get just about every file via

http://mirror.sobukus.de/$FILE

from my collection of upstream mirrors, including automatic downloads
of everything in the grimoire (at least static source URLs not
depending on fancy spell config).

To ease the thinking, I'll code a redirect of

http://mirror.sobukus.de/files/sourcemage/fallback/$FILE

to do the same search over the whole repository if the file does not
exist there. The benefit would also be that we can put a specific
version of $FILE into the fallback directory and that will be preferred
over any other version I have. The elephant in the room is that with
re-releases, there is a set of $FILEs that have different versions.

> What it does is this:
>
> 1. Gets the source file name
> 2. Goes through the list of mirrors and performs HEAD against them
> 3. On 200 OK it saves the file name and mirror in Redis with
> expiration time of 1 week
> 4. Right after #3 it 303-redirects to the corresponding mirror and
> returns the URL to the client
> 5. In case all mirrors return 404, it yields with 404 error as well
> 6. Next time the same file is requested, the mirror is taken from Redis
> quickly without any need to do the full hunt

To make this really robust, we should add versioning of files to this.
At least we should be able to catch the case where the first found file
is the wrong one and then the verify stage fails. If you remember only
that one, but somewhere else (even on the same mirror) a correct
version exists, we have permanent spell failure. This is not many
spells, but it is more than a few were release file names are not
unique.

I have it on my schedule to add a checksum-based query to the mirror
(once I figured out versioned storage with the same file name) … like


http://mirror.sobukus.de/checksum/sha512/7b77b543bd6a127095c6a6a2e7f9b5933f5a391734cc6dcbc160f786008afcd74646a7c304e1be682042221648b0fcae7efafda4bbe66ba30a7021740abddc34

That would ensure that we really get the correct files. For that to
work, we'd need to add hashes to all sources, even if they're verified
by GPG sig. We're playing catch-up here with other distros that mirror
upstream sources since ages, also with hash sums.

> the logic above is valid for fallback at download.sourcemage.org. it
> also means mirrors can be not in sync and have different sources until
> we want to sync them and have enough capacity to store data.

Do you intend to eventually populate the fallback directory with _all_
sources we ever used? Could become a rather huge directory, and would
mean that I indeed should stop syncing that one and rather use the
redirect to the regular upstream file database I have on my mirror
(also with sync scripts separate from what's in the grimoire).

> It's only 50 LoC

50 lines? Including comments? ;-)

>, but I wanted to polish it with config file support
> (where you could configure mirror list and other stuff like timeouts
> and expiration time) before submitting it to our wand repository.
>
> Its dependencies are:
>
> - python
> - bottle
> - flup (for FastCGI)
> - redis-py (could be even faster with hiredis-py attached to redis-py)
> - requests

By the way … requests is an awfully generic name … we should really
prefix all these scripting language/runtime libs/modules to avoid
clashes. Things should be named py-$something, perl-$something … pkgsrc
has this neat scheme with the python prefix: There is one source
package named py-blabla and you can install py27-blabla and
py34-blabla, each for the respective python version.

> Now I'm trying to collect sources from my systems (which are about
> 20 and all running stable) and put them on Belltower. The next step
> would be to try to collect all sources for current stable grimoire

Back when I did the dance of rebuilding and updating out chroots (from
0.60 on I think), I had stuff pushed to my mirror already. It should be
rare cases where files are missing. The question is … does anyone want
to mirror _my_ mirror? It's around 800G sources so far … I don't have
deployment tools right now … am just tinkering with a slave mirror that
pulls the the scripts from my svn repo and the collecton of URLs via
git+http (lazy serving via apache). Hm. The URL collection itself is a
164M git repo. And I still need to fix the way the CPAN mirrors are
handled (currently each run of the update script adds new URLs with
different mirrors, hence I don't push that part to git yet). Mirroring
mirrors can become complex.

Anyhow, I'll turn

http://mirror.sobukus.de/files/sourcemage/fallback/

into the redirect thing now so that I don't have to educate folks about
not using that instead of a full search.

Ah, regarding protocols: My mirror talks HTTP and HTTPS over IPv4 and
IPv6, of course. I didn't bother with FTP, nor rsync. Mirroring of my
mirror is currently handled by checking out the same scripts and the
URL database, then doing the normal download which pulls sources from
upstream, optionally with my mirror set as a primary source to try
first. Especially for an initial sync, trying the original servers
first is nicer. At some point my ISP might get edgy.


Alrighty then,

Thomas

Attachment: pgpZtOEu0I587.pgp
Description: Digitale Signatur von OpenPGP




Archive powered by MHonArc 2.6.24.

Top of Page