[BitTorrent] Re: BitTorrent for input file staging on clusters?

antipast0 at yahoo.com antipast0 at yahoo.com
Thu Dec 2 00:53:02 EST 2004

--- In BitTorrent at yahoogroups.com, "Nathan Edwards" <nje01 at y...> 
> Hi, I'm playing with BitTorrent as a method for staging large input
> files to each node of a compute cluster prior to running a 
> job. 
> I was using scp to pull the input file(s) from the job submission 
> to each cluster node, but 40 nodes each pulling 1-4GB at the same 
> via scp just isn't workable. As I started thinking about how to do 
> better (chunks, download from peers, checksums), the solution I came
> up with sounded more and more like BitTorrent.
> I've been hacking at the python BitTorrent tracker server and 
> clients to make them more suitable.
> In addition to the patches for track.py from Denis Ahrens, I've had 
> add a small number of features to make it suitable for this use 
> * Connect only from IP's matching a regular expression
>   - in RawServer, for both the tracker and downloader
> * Reparse allowed torrents directory on demand:
>   - adding http://tracker/refresh_allowed
> * unescape URL character codes in tracker urls
>   - permits http://tracker/file%3Finfo_hash=... for easier scripting
> * downloader exit once file(s) is downloaded
>   - terminate once idle (download _and_ upload) for a while
> * numerous parameter tweaks to match this use case
>   - e.g. uploader client running on submission server accepts only 
>     one connection
> Is anyone else working on similar applications of BitTorrent?
> Thanks,
> nathan
> ---
> Nathan Edwards, Ph.D. 
> Center for Bioinformatics and Computational Biology
> University of Maryland, College Park, MD 20742-3360
> Email: nedwards at umiacs.umd.edu


The merging of the Semantic web stuff (RSS, Syndication, Aggregators) 
and BitTorrent is compelling for this reason: automation of 

I've been working on designing around this, but in a nutshell, if 
you're not familiar with RSS, it is what powers the interconnectivity 
and massive syndication of weblogs. It is mainly just XML, and XML 
based Remote Procedure calls.

By using that, you can script much easier what you want to do. Have 
your BitTorrent client subscribe to a feed of .torrent files, and the 
subscribed clients chime in to the swarm of each file as it is 

If you need some kind of topology across private or segmented 
networks, changing the tracker in the .torrent via a script is easy 
enough to do... that directs the swarm to use that sub-network's 

You can find use-case diagrams, some C# and Java-based scripts for 
doing something like this (as well as integration with blog systems) 
at my sourceforge site:


If you'd like to contact me further about this, or figure out how 
this can solve your situation, just let me know. This sort of XML 
based approach to organizing your automation is about the easiest and 
most flexible approach I've seen yet.


Thomas Winningham

------------------------ Yahoo! Groups Sponsor --------------------~--> 
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!

Yahoo! Groups Links

<*> To visit your group on the web, go to:

<*> To unsubscribe from this group, send an email to:
    BitTorrent-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:

More information about the BitTorrent mailing list