Skip to Content.
Sympa Menu

sm-discuss - Re: [SM-Discuss] Sorcery and the Broken Cache

sm-discuss AT lists.ibiblio.org

Subject: Public SourceMage Discussion List

List archive

Chronological Thread  
  • From: Seth Alan Woolley <seth AT positivism.org>
  • To: sm-discuss AT lists.ibiblio.org
  • Subject: Re: [SM-Discuss] Sorcery and the Broken Cache
  • Date: Fri, 15 Jul 2005 15:07:33 -0700

On Fri, Jul 15, 2005 at 02:45:29PM -0700, Andrew wrote:
> On Fri, Jul 15, 2005 at 04:03:28PM -0500, Jeremy Blosser (emrys) wrote:
> > On Jul 15, Andrew [afrayedknot AT thefrayedknot.armory.com] wrote:
> > > On Fri, Jul 15, 2005 at 03:29:20PM -0500, Jeremy Blosser (emrys) wrote:
> > > > On Jul 15, Seth Alan Woolley [seth AT positivism.org] wrote:
> > > > > If there are too many broken caches for reliability, we could
> > > > > implement
> > > > > our own caching using a manifest file with a list of md5sums of the
> > > > > big
> > > > > tarballs. This would allow more frequent checks for tarball
> > > > > changes as
> > > > > well, as they wouldn't have to download the whole tarball every
> > > > > time to
> > > > > check for a new update.
> > > >
> > > > Heh, I was just thinking about proposing this today, while trying to
> > > > check
> > > > if some changes had made it to the test tarball yet w/the p4 stunnel
> > > > issues. Having "{scribe,sorcery} update" check an md5 *before*
> > > > starting
> > > > the full download would save a lot of time/bandwidth.
> > >
> > > so, why not rsync?
> >
> > It would be useful for rsync as well. rsync takes a long time to run to
> > even decide it has nothing to update. The md5 comparison would happen
> > within seconds and eliminate the need for rsync to even get called a lot
> > of
> > the time.
>
> scribe with rsync in my project branch doesnt take very long, on the
> order of 30 seconds or less to do an empty update...
>
> Consider that with tarballs + manifest, if just one bit changes in
> the grimoire, the entire tarball has to be downloaded again.
> On the other hand rsync will only download chunks that have changed.
>
> So rsync might lose, barely, if absolutely nothing changed, it might
> also lose, barely, if everything has changed. But that isn't all that
> common unless you update almost constantly. The common case (a few
> changed files) will almost certainly be faster.
>
> Can we try to avoid suggesting unnecessary bloat and complexity to sorcery
> just because it feels good? Whatever happened to simple software.

I'm inclined to agree to a certain extent. I think all we really need
is the cache avoidance hack, as that directly addresses the problem that
we have _actually_ encountered which makes it unusable. If cache
avoidance is anabled, I expect it would take a really really horribly
broken proxy to prevent an if-modified-since sorcery option from being
implemented. wget supports -N and -nc (see the -nc option in the
manual) for timestamping support that could be implemented as well (this
is easier, I think than a manifest, but not as broken-cache robust).

Seth

--
Seth Alan Woolley [seth at positivism.org], SPAM/UCE is unauthorized
Quality Assurance Team Leader & Security Team: Source Mage GNU/linux
Linux so advanced, it may as well be magic http://www.sourcemage.org
Secretary Pacific Green Party of Oregon http://www.pacificgreens.org
Key id 00BA3AF3 = 8BE0 A72E A47E A92A 0737 F2FF 7A3F 6D3C 00BA 3AF3

Attachment: pgpTkgUGgTCkV.pgp
Description: PGP signature




Archive powered by MHonArc 2.6.24.

Top of Page