[bittorrent] using bittorrent for cluster file distribution

larytet.8708132 at bloglines.com larytet.8708132 at bloglines.com
Thu May 26 20:19:11 EDT 2005


sorry it is 1500Kbytes/s 

see screenshots in the previous post

--- Justin
Cormack <justin at street-vision.com wrote:
On Thu, 2005-05-26 at 20:06, Ryan
Stenson wrote:
> > Hi all,
> >  
> > I'm a consultant at the Cornell Theory
Center (www.tc.cornell.edu
> > <http://www.tc.cornell.edu/> ).  We're considering
using BitTorrent to move
> > large files from a file server/job submit node
to the compute nodes (1:M).
> > Using multicast, we get about 30-40 Mb/s,
however our network (GigE) should
> > be capable of 800+.  Does anyone have
similar experience with this sort of
> > application?
> >  
> > I've setup
Bittorrent on a few servers in an attempt to speed up our current
> > multicast
file distribution system (we do a lot of copying from a few
> > fileservers
to dozens of clients).  
> >  
> > I start the tracker with:           
  bttrack.py --port 6969 --dfile dstate
> > I then create the torrent with:
    btmaketorrent.py http://IP:6969/announce
> > c:\temp\filename
> > and
seed the torrent with:          btdownloadheadless.py
> > \\server\filename.torrent
--save_as c:\temp\filename
> > and download it to the clients with the same
command.
> >  
> > I'm only seeing 20 KB/s transfer speeds over Gigabit
ethernet (these
> > machines are dual xeon's, 4gb ram, etc.).  We see 30-40
Mb/s over multicast,
> > which is extremely slow compared to our theoretical
throughput of 800+ Mb/s.
> > Are there any default settings that are limiting
my throughput?  Any other
> > suggestions?  I realize that using the start
bittorrent code, uncompiled,
> > isn't the optimal solution, but this is
just the first step. 
> 
> Well a few things.
> 
> This has come up a
few times recently, but havent really had much
> feedback from anyone. Have
you got some time to run some tests?
> 
> First, I am writing a BT client
for this type of application. The first
> version (possibly a few residual
bugs, although seems pretty reliable)
> does work. I didnt do any real performance
tests, although on the same
> host with a large file, moving it to the same
hard drive I got around 20
> MB/s, so one should be able to do much better
between machines.
> 
> I am rewriting the new version in my (non-copious)
spare time, and its a
> bit more portable (first version is Linux only),
although I dont have
> any current intentions of porting it to Windows which
you seem to be
> using.... (unless you pay me...)
> 
> However you could
do some helpful tests:
> 
> First check transfer on the same machine (preferable
different disks)
> Then check between 2 machines only, across the network

> Check your network performance (I use ttcp for Unix, should be able to

> compile it for windows)
> 
> Then once you have done that theres more,
depending if there really is a
> point to point bottleneck, or whether it
is a multipoint bottleneck.
> 
> Oh, and what switches are you using and
how many machines are there?
> 
> j
> 
> 
> _______________________________________________

> BitTorrent mailing list
> BitTorrent at lists.ibiblio.org
> http://lists.ibiblio.org/mailman/listinfo/bittorrent

> 



More information about the BitTorrent mailing list