[BitTorrent] Re: BitTorrent for input file staging on clusters?
antipast0 at yahoo.com
antipast0 at yahoo.com
Thu Dec 2 00:53:02 EST 2004
--- In BitTorrent at yahoogroups.com, "Nathan Edwards" <nje01 at y...>
> Hi, I'm playing with BitTorrent as a method for staging large input
> files to each node of a compute cluster prior to running a
> I was using scp to pull the input file(s) from the job submission
> to each cluster node, but 40 nodes each pulling 1-4GB at the same
> via scp just isn't workable. As I started thinking about how to do
> better (chunks, download from peers, checksums), the solution I came
> up with sounded more and more like BitTorrent.
> I've been hacking at the python BitTorrent tracker server and
> clients to make them more suitable.
> In addition to the patches for track.py from Denis Ahrens, I've had
> add a small number of features to make it suitable for this use
> * Connect only from IP's matching a regular expression
> - in RawServer, for both the tracker and downloader
> * Reparse allowed torrents directory on demand:
> - adding http://tracker/refresh_allowed
> * unescape URL character codes in tracker urls
> - permits http://tracker/file%3Finfo_hash=... for easier scripting
> * downloader exit once file(s) is downloaded
> - terminate once idle (download _and_ upload) for a while
> * numerous parameter tweaks to match this use case
> - e.g. uploader client running on submission server accepts only
> one connection
> Is anyone else working on similar applications of BitTorrent?
> Nathan Edwards, Ph.D.
> Center for Bioinformatics and Computational Biology
> University of Maryland, College Park, MD 20742-3360
> Email: nedwards at umiacs.umd.edu
The merging of the Semantic web stuff (RSS, Syndication, Aggregators)
and BitTorrent is compelling for this reason: automation of
I've been working on designing around this, but in a nutshell, if
you're not familiar with RSS, it is what powers the interconnectivity
and massive syndication of weblogs. It is mainly just XML, and XML
based Remote Procedure calls.
By using that, you can script much easier what you want to do. Have
your BitTorrent client subscribe to a feed of .torrent files, and the
subscribed clients chime in to the swarm of each file as it is
If you need some kind of topology across private or segmented
networks, changing the tracker in the .torrent via a script is easy
enough to do... that directs the swarm to use that sub-network's
You can find use-case diagrams, some C# and Java-based scripts for
doing something like this (as well as integration with blog systems)
at my sourceforge site:
If you'd like to contact me further about this, or figure out how
this can solve your situation, just let me know. This sort of XML
based approach to organizing your automation is about the easiest and
most flexible approach I've seen yet.
------------------------ Yahoo! Groups Sponsor --------------------~-->
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!
Yahoo! Groups Links
<*> To visit your group on the web, go to:
<*> To unsubscribe from this group, send an email to:
BitTorrent-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
More information about the BitTorrent