[bittorrent] 'Rarest first' question.
Arnaud.Legout at sophia.inria.fr
Thu Jun 29 15:59:12 EDT 2006
Bruno Hertz wrote:
> Well, I'd say too BT suggests rarest first, and I also see
> why. Problem is there is no written spec apart from 1.0, which
> suggests random selection. If you know of any other formal, written
> specification, apart from the mainline source code, please let me
you are right. I just had a look at
and found that indeed, they recommend random, and rarest first as a good
I do not remember this paragraph and I wonder whether it was modified.
This is clearly a bad recommendation. My guess is that they based this
on the earliest versions of BT that implemented random piece selection.
This policy changed in 3.x
if I remember correctly. I suppose that this recommendation on random
piece selection was never removed.
It ought to be removed.
> I'm aware of that. Currently I'm in a discussion with the libtorrent
> developer to talk him out of making available a hybrid algorithm,
> which applies rarest first selection for pieces with availability < 7
> and strict sequential (!) download for pieces with higher
> availability. That's what I call hybrid. Link here:
> You may browse through the first three or four postings in that thread
> to get a picture.
I see and I agrree with the argumentation.
rarest first is the policy to follow in order to guarantee the best
under certain circumstances, it makes sense to download pieces
sequentially (when the highest
efficiency is not a requirement). As explained in the thread, rarest
first is still the piece selection strategy,
but for pieces that are really well replicated (the threshold is a rule
of thumb, but it seems reasonable)
sequential piece selection can be used.
> His motivation btw is better caching, to reduce disk seeks. That's why
> he basically wants to serialize the pieces to some extent, simply by
> downloading them in that order :)
> Developers seem to find all kinds of motivations to deviate from the
> 'standard', I already mentioned another one: watching movies already
> while downloading, again a motivation for sequential piece selection.
> It's either users who come up with all kinds of weird feature
> requests, and developers trying to please them, or the developers
> themselves who think up ways to 'improve' bittorrent, make it faster
> or whatever.
> I've been looking into this for about two weeks now and already got
> the impression developers implement anything but the mainline
> algorithms (ok, that's an exaggeration, but the tendency is definitely
many people believe they can improve BT. If the target is the highest
efficiency, I am still looking for someone
that do something reasonable, apart from the mainline client. There is a
huge difference between minor improvements
that can improve by 10% the efficiency in 2% of the cases, and real
a real improvement is much harder than many believe. However, trying by
implementation is a
good way to learn.
> Are they that large? I hoped you could just roll them all into a zip
> or tgz and put them up on your web site. Others might be interested
> If that isn't feasible though, I'll give you specifics about which
> results interest me.
Sure this is feasible, but yet I do not see what it is useful for. By
zooming into the PDF we can get a resolution that is enough
from my point of view.
Can you point me to the plot for which you need more resolution?
> Well, I'm not coming from the academic end but rather considering
> client development. Right now I'm checking out projects I might
> contribute to and make use of.
> As mentioned, one issue the libtorrent developer is concerned with is
> caching. Your client might for example help to gather some statistics
> for this, better understand piece request and download behavior and
> maybe discover patterns.
> I myself would like to look into the behavior of different clients on
> live torrents, including deviations from the standard. In particular,
> that's something tracker hosts might be interested in, so again your
> client could be the starting point for something practical.
> Don't feel urged though. If it's a licensing (or competitional) issue
> I might also pick an open source client for now and modify it myself
I will post on the mailing list as soon as I can release the
More information about the BitTorrent