[cc-community] Open Content mirrors?

Jonathan Gray jonathan.gray at okfn.org
Wed Apr 9 15:11:35 EDT 2008


Evan Prodromou wrote:
> Evan Prodromou wrote:
>> http://www.ckan.net/
> Oops. Spoke too soon; this is more of a directory than a mirror service.

Just thought I'd forward Rufus' response from okfn-discuss to the thread 
here...

Evan wrote:

<quote>
I'd hoped that this was what CKAN was about, but apparently it's more of 
a directory than an archive network. Is there any service out there that 
mirrors Open Knowledge data sets?
</quote>

Rufus wrote:

<quote>
CKAN is about that but indirectly. If you think about what you need to
data componentization (or federation and replication which is more what
you are interested in) you need first to have a registry of some kind
that stores the basic metadata and acts as a lookup-hub (note that this
service itself can be decentralized and replicated). That is what CKAN
is for.

CKAN, as you have correctly surmized, is not for holding the dumps
themselves but instead provides the 'download_url' field which could be
used either to point to a page with links (perhaps to the various
mirrors), a single downloadable file, a torrent file etc. (What we need
next is code to interact with CKAN in the way that
easy_install/setuptools does with pypi, or perl does with CPAN. This is
currently in progress as datapkg [1]).

[1]: http://knowledgeforge.net/ckan/svn/datapkg/trunk/

However none of these points answers your question which is where can I
dump backup files of the project? Or to put it in a wider perspective
where are the mirror networks for open knowledge that exist for F/OSS?
The answer at present is not that many that I know of (inside academia
there seems to be some degree of work on this in relation to Grid-type
activities). The two immediate options I can think of are:

    * http://www.archive.org/ It appears that the Internet Archive are
currently happy to host datasets and are pretty unlimited in capacity.
    * http://knowledgeforge.net/ This is a complementor project to CKAN
also run by us. It was primarily designed for actual project development
but you could use it simply to store dumps (i.e. uploading via DAV or
via SVN). The storage on this is not unlimited.

Finally, given the lack of anything that exactly fits what is needed
maybe we should think about putting something simple together which can
aggregate storage (everybody gives 10GB ...) and thereby provide a
simple mirror network. There seem plenty of basic off-the-shelf
components that could be used for this (see [2] [3] for summary and links).

~rufus

[2]: http://okfn.org/wiki/DataDistribution
[3]: http://okfn.org/wiki/ToolsWeNeed

</quote>

I think as a long term project, we'd like to get more into mirroring - 
but we do not do this in any serious way yet - partly due to limited 
resources!

Hope this is useful...

Best wishes,

Jonathan


More information about the cc-community mailing list