Public SourceMage Discussion List

["Jeremy Blosser (emrys)" <jblosser-smgl AT firinn.org>]

On May 08, Jeremy Blosser (emrys) [jblosser-smgl AT firinn.org] wrote:
> An update on this, here are svn's numbers for just the current stable
> branch:
> 
> ~/scm/svn# du -chs repo checkout
> 16M     repo
> 532M    checkout
> 548M    total
> 
> That much per branch is several GB total, which seemed completely nuts, so
> I tried again on a 1k block fs instead of the 4k I was on.  That brought
> the checkout down some:
> 
> ~/scm/svn# du -chs repo checkout
> 16M     repo
> 146M    checkout
> 162M    total
> 
> The morale: if we use svn (and svk), you'll want a small block size on your
> scm fs, and still might need around a GB locally.  svn gurus, is there
> something else I might be missing here?

The full size of svn with devel, test, stable-rc, and stable branches (on a
4k block size fs) is:

163M    repo
2.4G    checkout
2.6G    total

Note that the repo only includes the data for the last change to each
spell, not a full history (yet).

> Of course, git on the same 1k block fs is smaller as well:
> 
> ~/scm/git# du -chs grimoire
> 40M     grimoire
> 40M     total
> ~/scm/git# du -chs grimoire/.git
> 9.1M    grimoire/.git
> 9.1M    total
> 
> The data I'm importing:
> ~/scm# du -chs stable
> 31M     stable
> 31M     total
> 
> No history or branches in that one, but one of the git devs has written a
> p4->git importer for us; I'll be trying that soon hopefully and will be
> able to see how much disk our entire commit history requires.

Tony got me a p4 dump to set up locally, and Sean Estabrooks (aka "loops",
the aforementioned git developer) ran his importer against this today for
the devel and test branches (I'm running it on a xen vm on a dev box here,
and it's a very intense process, so we just did a couple branches for now).
You can see the results here:
http://scmweb.sourcemage.org/smgl/grimoire?p=grimoire_p4.git

Note the yellow tag ids after each commit; loops' converter actually adds a
git tag per commit referencing the p4 changeset ID, so you can check out
the git repo at a named changeset if you want.  That bloats the size of the
repo a good bit, so I don't know if we'd distribute that, but it's good to
have the data in at least an archive set.

The full size of the .git repo will all this history data is:
40M     .git
67M     tags
106M    total
(The tags add quite a bit of size, another reason we probably wouldn't
include those in what your checkouts have.)

That includes devel and test, going back to the origin of the
//sgl/grimoires/ structure in p4.  I know we used other paths before that,
so we may actually want to go back farther for some archive set, but I
believe this includes at least the last 1-2 years.  Adding the other
branches probably won't scale linearly even, because git does a lot of
compression and optimization of duplicate references, and the history
entries have a lot of duplication.

In other words, I think it's safe to say without any more hedging that even
if we include all the history, size is not going to be a major concern with
git being distributed to all developers.

Attachment: pgpsgELjsGWfb.pgp
Description: PGP signature