[BitTorrent] Re: BitTorrent for input file staging on clusters?

Nathan Edwards nje01 at yahoo.com
Thu Dec 2 10:12:23 EST 2004



--- In BitTorrent at yahoogroups.com, Jesus Cea <jcea at a...> wrote:
> Nathan Edwards wrote:
> > 
> > Hi, I'm playing with BitTorrent as a method for staging large input
> > files to each node of a compute cluster prior to running a distributed
> > job. 
> 
> If you are in a LAN, I rather suggest using a multicast distribution 
> procedure.

Writing a full on MPI application (or similar) just for file staging
feels like overkill for this problem.

In fact, isn't BT just a dynamic topology multicast? Consider any
chunk of the file. A node only requests it from someone if they don't
have it,  and doesn't forward it until it has it, and only to those
who don't already have it. The path defined by any single chunk must
therefore be a tree. That each chunk's tree is (potentially) different
and that the chunks arrive in an arbitrary order and via different
paths give a measure of fault tolerance and load balancing. 

Am I missing something?

Cheers!

nathan





------------------------ Yahoo! Groups Sponsor --------------------~--> 
Make a clean sweep of pop-up ads. Yahoo! Companion Toolbar.
Now with Pop-Up Blocker. Get it for free!
http://us.click.yahoo.com/L5YrjA/eSIIAA/yQLSAA/dkFolB/TM
--------------------------------------------------------------------~-> 

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/BitTorrent/

<*> To unsubscribe from this group, send an email to:
    BitTorrent-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/
 





More information about the BitTorrent mailing list