Public SourceMage Discussion List

[Vasil Yonkov <bustervill AT yahoo.com>]

Ever since I tried Source Mage for the first time, the update process is something that always bugged me. Maybe I was spoiled by Debian, but the 'scribe update' was taking a lot of time. Now, I finally decided to look for a faster way to update the spells, so I made a little study and a simple benchmark.

The machine is:

AthlonXP 1700+
7200rpm HDD - 42MB/s on the hdparm test and 8.5ms average seek time
ext3 filesystem
max of 1.2MB/s international internet bandwidth

Debian update:

ftp.debian.org - 160ms average ping
3 "grimoires" - main, contrib and non-free = 26 items, totalling 37,4 MB

the times are:
45s with 174 changes
2,5s with no changes

Source Mage update:

sorcery version 1.14.2-rc5
dbg.download.sourcemage.org - 175ms average ping
test grimoire = 39162 items, totalling 48,7 MB

Turned out that there are three ways to update:

tarball = 1m55s
rsync = 14m40s
git = from 20s to 45s


The first thing to notice is the number of items (files & folders) 26 vs 39K. This is also the main reason for updates to be so slow - huge number of small files. This is because of the very nature of Source Mage - the packages contain only the source, and the spells must contain every other thing needed, which makes them very descriptive. On the other hand, Debian is a binary distro, the compilation is already done and the stuff needed for the program's installation & configuration are in the packages, and that makes their "spells" very simple, so they can be combined in a single file without increasing complexity.
I did not studied Debian's update process in detail, but I think they also make files only with the changes in current update towards the state after the previous update - like patches. When you make an update, only are downloaded the "patches" since last update and applied consecutively. While in the tarball update process in Source Mage, you have to download all spells each time. That's the reason I actually thought the rsync update will be faster :) Moreover, with rsync method, the authentication failed every time (might be a temporary issue), and with git method, Sorcery was telling me every time "there is no verification method!". Well, I thought git has a built in sha1 checksum functionality. Am I wrong?

This "benchmark", that everybody can do, points git as the fastest update method. The only "downside" is that you need git besides bash :) If we have to be technically correct - for tarball update you need bzip2 and for rsync update - rsync. So the dependencies can not be pointed as a downside.

This raises an another question - after git is such a good method for update, why do we stick to the tarballs? It's a sluggish method. I don't know how it is on the server side, but I'm pretty sure that it uses a lot of bandwidth (compared to git) - everyone, on every update have to download around 6MB.

Wouldn't be easier to branch the codex with git and make it the default update method?