Public SourceMage Discussion List

[Vlad Glagolev <stealth AT tiberian.ru>]

On Sun, 5 Mar 2017 03:37:46 -0300
Ismael Luceno <ismael.luceno AT gmail.com> wrote:

> On 04/Mar/2017 21:47, Vlad Glagolev wrote:
> <...>
> > Tech specs: Python, multi-processing, uses clean YAML config and
> > 'requests' module to perform API calls against GitHub and Bitbucket.
> <...>
> 
> First, let me comment on the software:
> 
> Why not have site-specific rewritting rules for the repo names and 
> make it scan the root directory? that would save quite some config.

Well, that'd be incorrect for 3 reasons:

- repositories need to be composed into groups (projects), tagging them
with "fake" metadata, let's say in .git/info/project, would be more
unclear and lead to metadata duplication
- there are repositories in the root directory that we don't really
want to publish, so scanning might lead to mispredicted results
(excludes?)
- sometimes we want to name them differently than the directory
structure is, e.g. 'sorcery' goes to 'sorcery', but 'misc/castfs'
doesn't go to 'misc-castfs', but to 'castfs', while 'tome/scrolls' goes
to 'tome-scrolls'.

so we're not saving much code/logic here, but we give a clear picture
of what maps to what and have proper grouping instead.

> 
> The project relationship can be stored in the gitdir, and the project
> key for bitbucket can be deduced.

As I explained above, I'd avoid metadata duplication in every
repository, and rather have a single config representing desired
structure of what we have.

> 
> What do you think?
> 
> 
> Now, about the mirrors:
> 
> Public mirrors are useless by themselves. If scm.sourcemage.org dies,
> we stall, no matter how many mirrors.

These are not the benefits that public mirrors give us. Until we
properly build our infrastructure to avoid a single point of failure,
GitHub won't save us from downtime, no doubt.

But theirs purposes are:

- reserved copies in the cloud immune to outages: if the server goes
down forever, those would be the first places to look for the *most*
up-to-date copies of *everything* -- not just what a Sorcery hacker /
Grimoire guru works on on his local box daily
- presence in popular development communities: this affects SEO as well
- simpler contributions from the outside: once the project gets enough
interest from users, we could try the Apache model.

Regarding the last, for now it's clear it's easier to contribute via
GitHub or Bitbucket by submitting patches there. Here's an example of
how it works for them:

- the repo mirror is located here: https://github.com/apache/libcloud
- the repo can be forked and PRs for it are submitted there
- maintainers discuss issues and apply patches to the main repo closing
PRs without merging them into mirrored repo, which gets sync-ed later on

> 
> Our Achilles heel is our reliance on git-receive-pack for
> serialization, until we fix that there's no improvement.
> 
> Since we lack the manpower to do it manually, the only option is
> to develop some lockstepping solution; only then the mirrors become
> interchangeable.
> 
> 
> <...>
> > Mirroring part is quite important. We need to resolve our fallback
> > consistency once and for all.
> 
> Indeed.
> 
> <...>
> > Quick question: do we have any scripts responsible for auto-summon
> > and hash-verification for all spells in the grimoire hanging
> > somewhere? If yes, I'd like to revive them.
> 
> IIRC, no. Thomas had some script, and I prototyped a parsing engine in
> AWK to process our grimoires faster:
> http://git.iodev.co.uk/ismael/smgl/experiments/spell-parser.git/
> 
> While far from perfect, it does a pretty good job already. Could be
> rewritten in C for more speed.
> 
> 
> > The last, but not least: I'd like to introduce a new component for our
> > project. For now I called it 'Wand', but if you want to replace it with
> > something better -- I'd love to hear about your suggestions!
> <...>
> 
> +1.
> 
> Suggestion: recovery system and procedures could be called arround
> the topic of "necromancy" (since "resurrect" is already used), and
> mirroring around "reflection".


--
Vlad Glagolev <stealth AT tiberian.ru>

Attachment: pgpA7ntzTBayS.pgp
Description: PGP signature