ashwood at msn.com
Wed Jun 22 22:09:27 EDT 2005
Since it has garnered so much media coverage and some very strange debates
in unusual places I figured I'd put some views up here.
I have had the opportunity to consider this problem for quite some time, and
even on this list we have discussed online and tornado codes on a number of
occassions, and Avalanche is really nothing more than this concept given a
So is this concept useful?
That depends entirely on what your swarm looks like. If you have a greedy
swarm (a swarm that disconnects as soon as 100% is reached) then the answer
is yes, but if your swarm is generous it is actually harmful. My firmest
stand though is that the Avalanche team measured the wrong thing, they
measured the network throughput "as the total number of blocks transfered in
a unit of time" (page 6 of the avalanche paper), when the real measure of
performance in this situation is the amount of final file data that is
transferred in a unit of time, this is a very important distinction.
In a greedy swarm the extra pieces create a situation where the rarest
pieces are unnecessary (in bad cases the rarest pieces may disappear forever
leading this absolutely no where), this delivers the extra throughput that
the Avalanche team uncovered and in a greedy swarm this extra throughput
will lead to file saturation faster, and as a result faster delivery of the
final file. This kind of swarm is often seen in the wild and is there
anytime the situation has 1 seed and a million peers, or less difference,
but the idea holds.
The generous swarm is an entirely different concept. A generous swarm is one
where the peers stick around after becoming seeds, these swarms often have
more seeds than peers. The online codes become a penalty in these swarms
because they require the transfer of comparatively more data. In these
situations the swarm provides the redundancy WITHOUT any file based
redundancy. The result is that a generous swarm will saturate any pipe
between a given peer and the rest of the network. This saturation means that
the network coding overhead reduces the file transfer rate, resulting in
lower performance for all peers.
Eventually we end up back at the same decision that has been made a number
of different ways throughout the years, whether the question is whather or
not to use redundancy in RAID, or how many backups to keep, or the new one
in how to encode the data for transfer, this decision has been made over and
over and it is necessary for the developers to realize this so we don't
reinvent the wheel every 3 days.
More information about the BitTorrent