Public SourceMage Discussion List

[Seth Alan Woolley <seth AT positivism.org>]

On Fri, Jul 15, 2005 at 02:45:29PM -0700, Andrew wrote:
> On Fri, Jul 15, 2005 at 04:03:28PM -0500, Jeremy Blosser (emrys) wrote:
> > On Jul 15, Andrew [afrayedknot AT thefrayedknot.armory.com] wrote:
> > > On Fri, Jul 15, 2005 at 03:29:20PM -0500, Jeremy Blosser (emrys) wrote:
> > > > On Jul 15, Seth Alan Woolley [seth AT positivism.org] wrote:
> > > > > If there are too many broken caches for reliability, we could 
> > > > > implement 
> > > > > our own caching using a manifest file with a list of md5sums of the 
> > > > > big 
> > > > > tarballs.  This would allow more frequent checks for tarball 
> > > > > changes as 
> > > > > well, as they wouldn't have to download the whole tarball every 
> > > > > time to 
> > > > > check for a new update.
> > > > 
> > > > Heh, I was just thinking about proposing this today, while trying to 
> > > > check
> > > > if some changes had made it to the test tarball yet w/the p4 stunnel
> > > > issues.  Having "{scribe,sorcery} update" check an md5 *before* 
> > > > starting
> > > > the full download would save a lot of time/bandwidth.
> > > 
> > > so, why not rsync?
> > 
> > It would be useful for rsync as well.  rsync takes a long time to run to
> > even decide it has nothing to update.  The md5 comparison would happen
> > within seconds and eliminate the need for rsync to even get called a lot 
> > of
> > the time.
> 
> scribe with rsync in my project branch doesnt take very long, on the
> order of 30 seconds or less to do an empty update...
> 
> Consider that with tarballs + manifest, if just one bit changes in
> the grimoire, the entire tarball has to be downloaded again.
> On the other hand rsync will only download chunks that have changed.
> 
> So rsync might lose, barely, if absolutely nothing changed, it might
> also lose, barely, if everything has changed. But that isn't all that
> common unless you update almost constantly. The common case (a few
> changed files) will almost certainly be faster.
> 
> Can we try to avoid suggesting unnecessary bloat and complexity to sorcery
> just because it feels good? Whatever happened to simple software.

I'm inclined to agree to a certain extent.  I think all we really need 
is the cache avoidance hack, as that directly addresses the problem that 
we have _actually_ encountered which makes it unusable.  If cache 
avoidance is anabled, I expect it would take a really really horribly 
broken proxy to prevent an if-modified-since sorcery option from being 
implemented.  wget supports -N and -nc (see the -nc option in the 
manual) for timestamping support that could be implemented as well (this 
is easier, I think than a manifest, but not as broken-cache robust).

Seth

-- 
Seth Alan Woolley [seth at positivism.org], SPAM/UCE is unauthorized
Quality Assurance Team Leader & Security Team: Source Mage GNU/linux
Linux so advanced, it may as well be magic http://www.sourcemage.org
Secretary Pacific Green Party of Oregon http://www.pacificgreens.org
Key id 00BA3AF3 = 8BE0 A72E A47E A92A 0737  F2FF 7A3F 6D3C 00BA 3AF3

Attachment: pgpTkgUGgTCkV.pgp
Description: PGP signature