Standards (was [BitTorrent] Back to Merkle Hash Trees...)

Justin Cormack justin at
Mon Feb 7 17:18:17 EST 2005

> > > Is there some URL somewhere that describes the way Merkle hash
> > > trees are supposed to be used in bt2? It seems that a lot of
> > 
> > No (AFAIK), but I've got an experimental implementation described at 
> >
> Olaf,
> This spec is excellent.  It's an outstanding document.  Also, it sounds
> like you've already tested it, so it shouldn't have too many surprises
> for implementers.
> While there are minor points that I would have liked to fine-tune, I
> think having a working, well done spec like this is more important.  If
> I had a vote, I'd vote that we adopt this spec as-is.

Ugh, no certainly not. Its a quick hack to try out some ideas, as a "spec"
it is neither specific enough, nor well thought out.
> This gets back to a fundamental problem I feel this group currently has.
> Without active participation by the author of BT, we'll have trouble
> making ANY advances in the BT protocol, since we don't have any
> mechanism for officially agreeing on anything.  With p2p networking
> advancing so rapidly, I feel the future of the BT protocol is in doubt
> if it cannot evolve.
> So, I'd like to see Olaf's Merkel hash trees adopted.  How can we
> proceed in that direction?
> Perhaps a new unmoderated mailing list be formed, where permission to
> post to the group would be given only to authors of BT clients, or those
> who have at least published patches to existing clients.  Such a group
> might be able to make progress.
> If there's interest in such a group, I could set it up, or perhaps
> someone like Olaf should do it.

Writing good protocols, and writing good specs is a difficult process, 
there are lots of inetrests to be balanced and arguments to have, and it
is not really a job for one person.

As far as I am concerned The most important long term thing is that there
needs to be an approved RFC for a protocol that fulfils the function of
bittorrent. De-facto "standards" (even disregarding the ambiguities and
implementation differences) are not going to legitimize this sort of 
protocol in the way a real standard will.

The initial thought was simply to write up BT1 as it is in the right way
(despite the potential existence of BT2), but the more I have thought about
it the less likely it seems that it would actually be accepted as an RFC,
as there are problems with parts of the protocol.

Things that I think are significantly problematic are
1. torrent files are not ASCII, so they cant be passed by email, irc, whatever.
Hence the almost universal use of html as a transport, although it is
completely unnecessary. As transparent compression is supported in web servers
and many other mechanisms, it should be possible to design a file format that
is as compact but ASCII. The idea of encoding in a URL is related, but also
requires a shorter hash (eg Merkle trees).
2. Related to 1, the info hash depends on the torrent file format not just
on say the hashes of the file (it depends on the filenames too which is not
necessary either). Without this 1. would be fairly irrelevant as you could
make different encodings of the torrent without having to recode to bencoding.
3. Parts of the protocol are not ipv6 compliant (note to implementors: just
say no to compact=1). This is not acceptable in an RFC.
4. The overlapping of multiple subfiles into single pieces is really nasty.
It is not clear to me how multiple files should be handled, as if the
users dont want them all (and their client knows that) then they cease to
be interested in large parts of the torrent, and assumptions about the
peers you are connected to being interested in what you are go away. I
believe that if trackers scaled well you would be better off scrapping all
multifile support.

Things that it would be nice to fix as they are not very orthogonal
1. The statistics gathering part of the tracker protocol is not very well
thought out (eg the start/stop/completed thing doesnt work with UDP tracker,
and doesnt give out much helpful information, and peers can spoof). It would
be best to seperate the function of getting peers from statistics gathering
(and probably junk the stats gathering replacing it with a process that
connects with peers, handshakes and reads the bitmap). This would make it
easier to develop and test different peer distribution protocols, as a
distributed one clearly makes sense.
2. Lack of a dont_have_any_more message limits many types of applications
eg caches that need to drop data later. I cant think of any reason not to
add it, even if most clients wont use it (as it is a rare message you could
just send a new bitmap, although that uses more bandwidth).
3. piece size is all rather odd as the verfification and request units are
mismatched so as far as I could see when writing a client you might as well
request a whole piece from one client at once always. Nothing else made
any sense to me. SO why not make the requests whole piece?

Generally the protocol works well (and the peer protocol needs very little
change, apart from a tightening up of the spec), as demonstrated by experience
however I do think a real standards track process would be beneficial.

Then of course there are Merkle trees...


Yahoo! Groups Links

<*> To visit your group on the web, go to:

<*> To unsubscribe from this group, send an email to:
    BitTorrent-unsubscribe at

<*> Your use of Yahoo! Groups is subject to:

More information about the BitTorrent mailing list