[BitTorrent] BitTorrent for input file staging on clusters?

John Hoffman theshadow at shambala.net
Wed Dec 1 12:47:48 EST 2004

Nathan Edwards wrote:

>Hi, I'm playing with BitTorrent as a method for staging large input
>files to each node of a compute cluster prior to running a distributed
>I was using scp to pull the input file(s) from the job submission node
>to each cluster node, but 40 nodes each pulling 1-4GB at the same time
>via scp just isn't workable. As I started thinking about how to do it
>better (chunks, download from peers, checksums), the solution I came
>up with sounded more and more like BitTorrent.
>I've been hacking at the python BitTorrent tracker server and download
>clients to make them more suitable.
>In addition to the patches for track.py from Denis Ahrens, I've had to
>add a small number of features to make it suitable for this use case. 
>* Connect only from IP's matching a regular expression
>  - in RawServer, for both the tracker and downloader
>* Reparse allowed torrents directory on demand:
>  - adding http://tracker/refresh_allowed
>* unescape URL character codes in tracker urls
>  - permits http://tracker/file%3Finfo_hash=... for easier scripting
>* downloader exit once file(s) is downloaded
>  - terminate once idle (download _and_ upload) for a while
>* numerous parameter tweaks to match this use case
>  - e.g. uploader client running on submission server accepts only 
>    one connection

(1) "Connect only from IP's" isn't worthwhile, IMO; no one can download 
without the torrent, and if you block the tracker's infopage, they can't 
even try unless they guess the hash value.

(2) In future versions of the official release, the torrent directory 
will be reparsed so frequently that forcing a refresh isn't worthwhile.  
(BitTornado does this now.)

(3) Unescaping URLs for the tracker is problematic, since the info hash 
and peer ID may contain those characters.  I recommend coding your 
script properly from the start.

------------------------ Yahoo! Groups Sponsor --------------------~--> 
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!

Yahoo! Groups Links

<*> To visit your group on the web, go to:

<*> To unsubscribe from this group, send an email to:
    BitTorrent-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:

More information about the BitTorrent mailing list