[bittorrent] 'Rarest first' question.

Bruno Hertz brrhtz at yahoo.de
Thu Jun 29 20:36:13 EDT 2006

Arnaud Legout <Arnaud.Legout at sophia.inria.fr> writes:

> Hi,

Hello Arnaud 

>> Well, I'd say too BT suggests rarest first, and I also see
>> why. Problem is there is no written spec apart from 1.0, which
>> suggests random selection. If you know of any other formal, written
>> specification, apart from the mainline source code, please let me
> you are right. I just had a look at 
> http://wiki.theory.org/BitTorrentSpecification
> and found that indeed, they recommend random, and rarest first as a good 
> practice
> I do not remember this paragraph and I wonder whether it was modified.
> This is clearly a bad recommendation. My guess is that they based this 
> specification
> on the earliest versions of BT that implemented random piece selection. 
> This policy changed in 3.x
> if I remember correctly. I suppose that this recommendation on random 
> piece selection was never removed.
> It ought to be removed.

Well, I was thinking of http://www.bittorrent.org/protocol.html
either. An updated version seems to be announced though.

>> I'm aware of that. Currently I'm in a discussion with the libtorrent
>> developer to talk him out of making available a hybrid algorithm,
>> which applies rarest first selection for pieces with availability < 7
>> and strict sequential (!) download for pieces with higher
>> availability. That's what I call hybrid. Link here:
>> http://sourceforge.net/mailarchive/forum.php?thread_id=14663525&forum_id=33471
>> You may browse through the first three or four postings in that thread
>> to get a picture.
> I see and I agrree with the argumentation.
> rarest first is the policy to follow in order to guarantee the best 
> effciency. However,
> under certain circumstances, it makes sense to download pieces 
> sequentially (when the highest
> efficiency is not a requirement). As explained in the thread, rarest 
> first is still the piece selection strategy,
> but for pieces that are really well replicated (the threshold is a rule 
> of thumb, but it seems reasonable)
> sequential piece selection can be used.

Hmm. I'd disagree, but I know people who will likely be pleased to
hear that :)

>> I've been looking into this for about two weeks now and already got
>> the impression developers implement anything but the mainline
>> algorithms (ok, that's an exaggeration, but the tendency is definitely
>> there).
> many people believe they can improve BT. If the target is the highest 
> efficiency, I am still looking for someone
> that do something reasonable, apart from the mainline client. There is a 
> huge difference between minor improvements
> that can improve by 10% the efficiency in 2% of the cases, and real 
> improvements. Finding
> a real improvement is much harder than  many believe. However, trying by 
> implementation is a
> good way to learn.

Well, given that a complete mathematical model is hard to come up
with, empiricial studies sure help either. That's why I found your
papers to be useful.

>> If that isn't feasible though, I'll give you specifics about which
>> results interest me.
> Sure this is feasible, but yet I do not see what it is useful for. By 
> zooming into the PDF we can get a resolution that is enough
> from my point of view.
> Can you point me to the plot for which you need more resolution?

OK, here you go. The plots I find hard to read are those with dotted
and dashed lines:

Fig. 1.  Entropy characterization
Fig. 2.  Evolution of the number of copies of pieces, torrent 8
Fig. 4.  Evolution of the number of copies of pieces, torrent 7
Fig. 7.  CDF of the piece interarrival time for torrent 10
Fig. 8.  CDF of the block interarrival time for torrent 10
Fig. 10. Correlation number of unchokes / interested time.

Many thanks in advance for the effort.

>> Don't feel urged though. If it's a licensing (or competitional) issue
>> I might also pick an open source client for now and modify it myself
> I will post on the mailing list as soon as I can release the 
> instrumentation.

Good to hear, thanks.

Of course, thanks either for the interesting discussion altogether.

Kind regards, Bruno.


Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

More information about the BitTorrent mailing list