Public SourceMage Discussion List

[Bor Kraljič <pyrobor AT ver.si>]

On 12. of Sep 2011 11:31:38 David Kowis <dkowis AT shlrm.org> wrote:
> On 09/12/2011 10:01 AM, Bor Kraljič wrote:
> > Hellow.
> > 
> > I have been checking source download and verification of spells sources
> > for quite a while now. We have 2 master bugs
> > (http://www.sourcemage.org/issues/53 and
> > http://www.sourcemage.org/issues/55) on our issue tracker about that.
> > 
> > But bug http://www.sourcemage.org/issues/176 got my attention. The
> > source from OUR fallback server failed the verification. (just from
> > one. The other mirror is OK... Strange).
> > 
> > That how I started to wonder how our fallback mirrors are getting
> > updated? And how often?
> > 
> > I researched and found http://dbg.download.sourcemage.org/fallback/. Our
> > fallback mirror. To my surprise it was updated very recently :) Which is
> > GOOD! However when I looked what files were "last modified" I noticed
> > that there are SCM sources on our fallback mirrors. Which is totally
> > pointless. As far as I know sorcery doesn't even look for SCM sources
> > on fallback mirrors.
> The SCM sources are there because when someone does a summon of all the
> spells in the grimoire, it downloads the SCM sources as well.
> 
> > The second problem I see is that sources don't get verified after/before
> > are put on fallback mirror (as seen on
> > http://www.sourcemage.org/issues/176).
> > 
> > So to sum up the questions:
> > * Is there a script for updating fallback mirrors?
> 
> Not an official one that I'm aware of.
What kind of script are you using (if any)?

> 
> >   - Can we modify/improve it?
> 
> Probably. :)
> 
> > * How often are fallback updated?
> 
> Depends on who's maintaining the fallback. I think I have one that I
> keep and then there's the one on dbg, but I don't know how that one is
> fired up.
You maybe know who is maintaining dbg mirror ?

> 
> > * How often are old sourced cleaned? (is it automated?)
> 
> I haven't cleaned out old source since I set up the mirror after the
> last harddrive failure.
> 
> > * Why source from spell live (stable source live.2011.07.21.tar.gz or in
> > test grimoire live.2011.09.02.tar.gz ) ? (Why I am asking about this
> > particular spell? Because upstreams removes old source with every
> > version bump leading to download failures for our spell)
> 
> I don't know? I'm not sure I understand the question.
nvm :)
> 
> > Answers to those would give me basic understanding of how our fallback
> > mirrors work.
> 
> Our fallback mirrors are semi-unofficial. People volunteered to run
> them, since they're supposed to only be used when the primary tanks for
> whatever reason.
> 
> I had been working on a ruby script to do a bit more parallelization of
> the downloads, as well as handling the downloads in a smarter way, for
> failures and verification purposes, but I haven't quite finished it yet.
I hope you know that I have written one script (hashcheck.sh) and commit it 
into guru-tools repository that summons and verifies the sources. It can also 
ignore SCM sources. However it doesn't have parallel download and it is not 
that that smart. Well it is smart as summon. It misses sources that are 
hidden 
behind queries and logic in spells. 

But it would still be better than pure summon for fallback mirrors. 

> 
> David