[bittorrent] 'Rarest first' question.

Arnaud Legout Arnaud.Legout at sophia.inria.fr
Thu Jun 29 15:59:12 EDT 2006


Bruno Hertz wrote:
> Well, I'd say too BT suggests rarest first, and I also see
> why. Problem is there is no written spec apart from 1.0, which
> suggests random selection. If you know of any other formal, written
> specification, apart from the mainline source code, please let me
you are right. I just had a look at 
and found that indeed, they recommend random, and rarest first as a good 
I do not remember this paragraph and I wonder whether it was modified.
This is clearly a bad recommendation. My guess is that they based this 
on the earliest versions of BT that implemented random piece selection. 
This policy changed in 3.x
if I remember correctly. I suppose that this recommendation on random 
piece selection was never removed.
It ought to be removed.

> I'm aware of that. Currently I'm in a discussion with the libtorrent
> developer to talk him out of making available a hybrid algorithm,
> which applies rarest first selection for pieces with availability < 7
> and strict sequential (!) download for pieces with higher
> availability. That's what I call hybrid. Link here:
> http://sourceforge.net/mailarchive/forum.php?thread_id=14663525&forum_id=33471
> You may browse through the first three or four postings in that thread
> to get a picture.
I see and I agrree with the argumentation.
rarest first is the policy to follow in order to guarantee the best 
effciency. However,
under certain circumstances, it makes sense to download pieces 
sequentially (when the highest
efficiency is not a requirement). As explained in the thread, rarest 
first is still the piece selection strategy,
but for pieces that are really well replicated (the threshold is a rule 
of thumb, but it seems reasonable)
sequential piece selection can be used.

> His motivation btw is better caching, to reduce disk seeks. That's why
> he basically wants to serialize the pieces to some extent, simply by
> downloading them in that order :)
> Developers seem to find all kinds of motivations to deviate from the
> 'standard', I already mentioned another one: watching movies already
> while downloading, again a motivation for sequential piece selection.
> It's either users who come up with all kinds of weird feature
> requests, and developers trying to please them, or the developers
> themselves who think up ways to 'improve' bittorrent, make it faster
> or whatever.
> I've been looking into this for about two weeks now and already got
> the impression developers implement anything but the mainline
> algorithms (ok, that's an exaggeration, but the tendency is definitely
> there).
many people believe they can improve BT. If the target is the highest 
efficiency, I am still looking for someone
that do something reasonable, apart from the mainline client. There is a 
huge difference between minor improvements
that can improve by 10% the efficiency in 2% of the cases, and real 
improvements. Finding
a real improvement is much harder than  many believe. However, trying by 
implementation is a
good way to learn.

> Are they that large? I hoped you could just roll them all into a zip
> or tgz and put them up on your web site. Others might be interested
> too.
> If that isn't feasible though, I'll give you specifics about which
> results interest me.
Sure this is feasible, but yet I do not see what it is useful for. By 
zooming into the PDF we can get a resolution that is enough
from my point of view.
Can you point me to the plot for which you need more resolution?
> Well, I'm not coming from the academic end but rather considering
> client development. Right now I'm checking out projects I might
> contribute to and make use of.
> As mentioned, one issue the libtorrent developer is concerned with is
> caching. Your client might for example help to gather some statistics
> for this, better understand piece request and download behavior and
> maybe discover patterns.
> I myself would like to look into the behavior of different clients on
> live torrents, including deviations from the standard. In particular,
> that's something tracker hosts might be interested in, so again your
> client could be the starting point for something practical.
> Don't feel urged though. If it's a licensing (or competitional) issue
> I might also pick an open source client for now and modify it myself
I will post on the mailing list as soon as I can release the 


More information about the BitTorrent mailing list