[bittorrent] Friend-machines-assisted approach to boosting one's BT download
ehem at m5p.com
Sun Jun 11 19:58:54 EDT 2006
>From: Harold Feit <dwknight at depthstrike.com>
> Elliott Mitchell wrote:
> >> From: Harold Feit <dwknight at depthstrike.com>
> >> Elliott Mitchell wrote:
> >>>> From: Harold Feit <dwknight at depthstrike.com>
> >>>> As on-topic as this suggestion may be, it has not been met with the best
> >>>> of responses in the development community.
> >>>> By having friend systems not duplicate pieces within the friend cluster
> >>>> until after 1.0 distribution within the cluster, the cluster is causing
> >>>> an overall negative skew of the distribution of pieces within the swarm
> >>>> as a whole, causing degradation in swarm-wide performance.
> >>> State the reasoning.
> >>> I don't see how this would happen. If you were calculating your numbers
> >>> incorrectly, I can easily see how you would come to this conclusion, but
> >>> this doesn't match reality. In particular if you count each member of the
> >>> group as a true distinct peer, then yes you reach this conclusion. Since
> >>> there is only one true downloader here though, you must count the entire
> >>> aggregate _group_ as one peer.
> >> No you can not count the entire aggregate group as one peer.
> >> Each member of the aggregate group has different peer connections than
> >> the other. Because of this, each peer that is connected to a different
> >> member of the aggregate group sees a different completion.
> >> With the members of the aggregate group actually following the rules
> >> about reporting the pieces they have, peers that connect to members of
> >> the group can only have a partial view of the pieces that the member
> >> group has.
> >> Even if they have a full view of the pieces that the group has, they can
> >> only ever have access to the pieces of the members they connect to.
> > Yeah, so? Where is the problem?
> > Slaves act as normal peers except they have a list of pieces that they
> > don't have and yet do not want (the pieces that the master or other
> > slaves have obtained). Otherwise their decision making is identical to
> > any other peer. How can this harm the swarm?
> By preventing non-cluster peers that they connect to from downloading
> pieces from outside of the locally-obtained set of a given peer in the
Yes, how does this damage the peer or swarm?
The non-cluster peer can still download those pieces from another
non-cluster peer. Unless a single meta-peer is at least 40 times a swarm
it joins, I don't see it causing any sort of problems. Crucially there is
little point in employing a meta-peer larger than around half the size of
a swarm because at that point, the meta-peer's bandwidth will likely
outperform the swarm's, at which point additional members won't increase
At least with my thoughts, the individual slaves are still act pretty
similarly to other peers. They would still retrieve what appear to be
rarer pieces, since other local peers are likely to want to download
those, in exchange for being able to download other pieces they still
As a /group/ the meta-peer still downloads one copy, and has to upload
roughly one copy. Yes, that copy is broken into distinct sets scattered
across the swarm, but this is nearly an ideal situation for seeding. Any
pieces not held by a local member of the cluster will be obtained via
other peers. The swarm still holds at least two distributed copies, one
on the meta-peer and one across all the peers the meta-peer downloaded it
Where is the damage?
> > Peers *never* have a complete view of the swarm, because in most cases
> > they never connect to every other peer. Yet even with this incomplete
> > view, BT seems to work quite well. The only thing I see would be if a
> > gigantic (say 200+) group joined a very small swarm (say 5 peers),
> > because they would flood the swam.
> > The only harm I see is if a client assumes behavior that is not required
> > by the protocol. Then presuming that a slave will perfectly conform to
> > your model. Such as assuming that a peer will be interested in
> > downloading every piece you have (Azeureus and not telling you about all
> > the pieces it has, on the assumption that it doesn't need to).
> In this proposal, the individual cluster peers won't tell about pieces
> that those peers don't have. If they advertise pieces that other peers
> in the cluster have but they don't, they will be in violation.
Perfectly correct. I think neither CoolByte nor myself were suggesting
such an approach.
What I was pointing out was that one might argue that the master
shouldn't advertise having pieces it got from slaves. The point being
that this might skew the availability of pieces. Pieces the master
downloads directly might be slighty less available, because they get
downloaded once and only appear once from the meta-peer, while pieces the
slaves get would get downloaded once and appear twice. OTOH this might be
compensated for because the master can use nearly all of its upload to
upload its pieces due to not spending upload passing it along. I'm unsure
whether this effect is even worth worrying about given that the swarm
will work as a group to even out that skew.
> > Again, slaves and the master conform to the BitTorrent protocol. If they
> > can harm the swam by conforming to the protocol, then the protocol needs
> > to be fixed. I don't see how they would harm the swarm though.
> If they are selectively not downloading pieces that others in their
> cluster don't have, they are preventing other clients from outside of
> their cluster that connect to them from being able to get the pieces
> from them.
> If all the slaves were operating 100% by the protocol the following
> things would happen:
> 1> They would continue to download pieces already obtained within their
> cluster, either from within their cluster or from outside (I don't care
> which personally).
> 2> They would try to not preference internally to the exclusion of all
> external peers.
This is not required to conform to the BitTorrent protocol. This is
merely conventional client design.
Peers will merely download those pieces from other peers that are not
part of the cluster. The effect you're concerned with isn't serious until
the meta-peer is massively larger than the swarm (until then the peers
are very likely to have connections to non-cluster peers), and it isn't
useful to employ a meta-peer that large because it cannot utilize its
full download nor upload.
> >>> If the central system advertises possession of pieces it got from
> >>> slaves, then you will see all pieces that the helpers download advertised
> >>> from two semi-distinct peers. All pieces the central system downloads
> >>> from the swam will only be advertised once. This though should not have a
> >>> major impact on the balance of the swarm (self-interest on the part of
> >>> normal peers should easily even this out), though it is a decent reason
> >>> to forbid the central system from advertising possession of pieces it
> >>> gets from slaves.
> >> A request to one peer cannot be fulfilled by another peer that has not
> >> received the same request. A request to one peer cannot be fulfilled by
> >> another peer when the second peer isn't connected to the requesting peer.
> > Yes, your point?
> > Perhaps change that sentence, instead of "from two semi-distinct peers,
> > with only one visible download" (the download from slave to master not
> > being visible to the swarm). Odd behavior if your client is observing
> > very carefully and makes assumptions about precisely how a peer operates,
> > but still protocol conformant.
> But part of the original proposal had exactly what I mentioned, having
> one peer in the cluster redirect the traffic for that piece to another
> peer in the cluster.
I didn't read that in what CoolByte was suggesting, nor have I ever
deliberately suggested such a thing. The only traffic redirection
CoolByte appears to of suggested was that the slaves upload pieces to the
master for no compensation (meta-BT, really). Not something a normal peer
would do, but still conformant to the BitTorrent protocol.
> >>>> Additionally, having the friend cluster separate itself from the main
> >>>> swarm once it has reached 1.0 distribution within the cluster causes
> >>>> even MORE of a negative skew of piece distribution of pieces within the
> >>>> swarm.
> >>>> It could potentially make one downloading user faster, at the cost of
> >>>> everyone else in the swarm.
> >>> This is the old "don't disconnect immediately" request. Your slaves
> >>> should only disconnect when the master shuts down.
> >>> And how do you prevent the implementation of this anyway? This is trivial
> >>> compared any of the other attacks on the swarm that come to mind. Come up
> >>> with defenses, don't ignore something that can be done.
> >> Attacks such as?
> > Hordes of zombies can easily cause disruption in a swam, but even without
> > those...
> > One method is to advertise posession of all pieces, but ensure no one
> > ever downloads more than one block from you. By corrupting one block of
> > every piece, the clients will have to discard 224KB of data merely for
> > 32KB of bandwidth on your part. Oh, make that discard 240KB of data for
> > merely 16KB of bandwidth, the smaller block makes this attack a lot more
> > efficient. Magnifying your effecting corrupting bandwidth by a factor of
> > 15 is pretty powerful. The problem here being that you cannot immediately
> > verify every block you download.
> I've seen this attack.
> Most implementations of this attack come from a single /16 or smaller
> subnet that can be trivially blocked with an ip blocker application.
> Additionally, the long-term effectiveness of such an attack against a
> given swarm is almost nonexistent due to the automatic banning
> algorithms already in place in modern mainstream clients.
I'm surprised they're so braindead as to limit themselves to a single
/16. At issue though is if you're talking to an evil peer, how do you
confirm that it is evil? You may need to download the entire piece from
other peers (and how do you know there isn't an evil one there?). True,
the evil peer can be uncovered pretty quickly, but until then it is still
causing 16 times its bandwidth to be wasted (upload the one block, the
piece needs a full download to confirm evilness of peer). That kind of
magnification factor is nasty. Even the more limited factor of 2 waste is
pretty decent. I must consider the inability to verify block a definite
weakness (made worse by smaller block sizes).
> > Simpler, retrieve random single bytes of random pieces that the peer has.
> > Pretty well forces random seeks of your disk subsystem (unless you can
> > store the entire torrent in memory). Kills all caches and just plain
> > murders performance unless your peer charges extra for small requests and
> > every time a new piece is loaded into memory.
> > There is no way to counteract the first without changing the protocol
> > (thankfully the RIAA, MPAA, et al haven't implemented this attack yet).
> See above.
Okay, they're braindead. I suspect they'll learn, eventually (and pigs
might start flying and they might fix their business model).
> > The second can be countered by careful design of the clients, but to my
> > knowledge all current clients will be murdered by this.
> Have you tested to make sure the second attack is actually viable?
So far it is merely a thought experiment.
Though, equally, have you confirmed that a meta-peer will damage a swarm?
(\___(\___(\______ --=> 8-) EHM <=-- ______/)___/)___/)
\BS ( | EHeM at gremlin.m5p.com PGP 8881EF59 | ) /
\_CS\ | _____ -O #include <stddisclaimer.h> O- _____ | / _/
\___\_|_/82 04 A1 3C C7 B1 37 2A*E3 6E 84 DA 97 4C 40 E6\_|_/___/
More information about the BitTorrent