Public SourceMage Discussion List

[Vlad Glagolev <stealth AT tiberian.ru>]

Hi everyone,

We've been testing our scm/website mirroring for quite some time, and
seems it works well enough to leave it as is for now.

Today I have some more updates on the next part of our mirroring infra:

UPSTREAM SOURCE CODE.

In the world of cloud servers, deprecating protocols and always changing
hosting providers upstream mirrors can become stale pretty quickly, so
we really need to have our own mirror of upstream source tarballs since
they're core of the distribution we cherish.

Currently, our master fallback mirror is quite useless: we don't have
many tarballs there, but even those which exist are mostly from 2006 :)

To prepare for new stable-rc cut I decided to fix our mirroring
workflow so we could always get the sources, especially for the stable
grimoire.

A note for Belltower: Belltower is a mirror server currently residing in
NJ I built to support this part of the solution. It also provides
mirrors for other distros I trust.

As of today, it contains full up-to-date (syncing every hour) mirror of
our master download location. The mirror supports HTTP and HTTPS, FTP
and RSYNC:

- http://mirror.belltower.us/sourcemage/
- https://mirror.belltower.us/sourcemage/
- ftp://mirror.belltower.us/sourcemage/

It is also fully IPv6-capable. RSYNC access is going to be limited to
other Source Mage mirrors who wanted to sync from it as I planned, so
it's not announced.

I have built a small service to redirect to up-to-date sources if
they're found on any of our mirrors. Currently they are:

- http://mirror.belltower.us/sourcemage/fallback/
- http://mirror.sobukus.de/files/sourcemage/fallback/

Think of it as a poor's man CDN, heh.

What it does is this:

1. Gets the source file name
2. Goes through the list of mirrors and performs HEAD against them
3. On 200 OK it saves the file name and mirror in Redis with
expiration time of 1 week
4. Right after #3 it 303-redirects to the corresponding mirror and
returns the URL to the client
5. In case all mirrors return 404, it yields with 404 error as well
6. Next time the same file is requested, the mirror is taken from Redis
quickly without any need to do the full hunt

the logic above is valid for fallback at download.sourcemage.org. it
also means mirrors can be not in sync and have different sources until
we want to sync them and have enough capacity to store data.

We don't (and probably won't for some time) have a lot of sources
served (and also since our user base is not huge), but I did some
stress-testing on the service, trying to DL axel's source file (which
was just around 50K):

Transactions:                  23556 hits
Availability:                 100.00 %
Elapsed time:                  59.14 secs
Data transferred:             569.53 MB
Response time:                  0.25 secs
Transaction rate:             398.31 trans/sec
Throughput:                     9.63 MB/sec
Concurrency:                   99.66
Successful transactions:       23640
Failed transactions:               0
Longest transaction:            0.76
Shortest transaction:           0.09

Pretty impressive, so the service is fully optimized. I completely
removed apache from geri, so scmweb.sourcemage.org should be doing
faster too now. Memory footprint and CPU consumption are on the lowest.

It's only 50 LoC, but I wanted to polish it with config file support
(where you could configure mirror list and other stuff like timeouts
and expiration time) before submitting it to our wand repository.

Its dependencies are:

- python
- bottle
- flup (for FastCGI)
- redis-py (could be even faster with hiredis-py attached to redis-py)
- requests

Now I'm trying to collect sources from my systems (which are about
20 and all running stable) and put them on Belltower. The next step
would be to try to collect all sources for current stable grimoire, and
then jump to upcoming stable-rc.

If you want to contribute and accelerate the process, please let me
know. Speaking of which, http://smgl.rawfeddogs.net/ might be a good
candidate. Kevin?

Thanks.

--
Vlad Glagolev <stealth AT tiberian.ru>

Attachment: pgpEiOmDq3SSu.pgp
Description: PGP signature