[bittorrent] BitTorrent file system progress?
bill at viasic.com
Wed Jun 29 10:34:49 EDT 2005
On Wed, 2005-06-29 at 12:38 +0100, Justin Cormack wrote:
> On 29 Jun 2005, at 12:28, Bill Cox wrote:
> > Hi.
> > Has anyone made any progress on building a BT based filesystem? If
> > not,
> > I think I'll slap one together.
> Not yet. Still on to do list.
> Would be interested in what kind of design you are going to use.
I'd like to keep it simple. I'd start with btslave code, and get a
ftp-like shell working for downloading files. Then I'd write a fuse
client to allow me to mount the directory just like a read-only disk,
rather than having to use the ftp-like shell. The fuse client is easy.
I've done one before.
If I assume that users have access to the .torrent files, then
everything should be easy. No changes to the protocol are required. I
just have btslave download the files requested, rather than random
blocks. Doing this and writing the fuse wrapper is probably a good
I'd probably also add a simple symbolic link capability that would allow
me to cd into a torrent that I've linked to.
After that, things get harder...
When file systems change often, it becomes tedious and inefficient to
keep downloading new .torrent files. For example, if it was a RPM
package server, updates might happen multiple times per day, and
the .torrent file might be bigger than most of the individual packages
being updated. I like Olaf's spec on Merkle trees. It reduces the
hash values to one per file, and minimizes the torrent file length. If
I fed the tracker a fixed info_hash value, clients could enter the
torrent without having the info_blob, just the fixed info_hash value.
They could download a list of peers, and use a get_info message to
request the torrent data from a peer (which he replies to with an info
message). Btslave already supports the get_info/info messages, but not
Olaf's Merkle trees yet.
Telling the clients that the file system has changed is a bit tricky.
There could be a special trusted seed that tells the tracker when the
torrent data has changed. Clients could be notified on their next
tracker update. In particular, there could be a way to broadcast which
files to delete, and which ones to add. Subversion-style revision
numbers could be used to help the tracker update the client's file list.
There are other changes that might be needed to improve performance.
For example, I'm thinking of hosting Fedora Core 4 RPMs, and not many
users will download every package available. If you wanted
btslave-1.0.0-1.i386.rpm, for example, you'd be in a very small group.
If there are thousands of clients talking to the tracker, you'd be lucky
to find a peer with what you wanted. Thus, there's a need to be able to
find peers with the required data. This could be done in various ways,
some of which I've seen discussed here before.
One way would be to burden the tracker with this task. Each client
would tell the tracker (with a bitfield) what files it has, and which
ones it wants. Then, a simple algorithm could find good peers for you.
To reduce the burden on the tracker, a tree of trackers might be needed.
The main tracker might redirect clients to a child tracker to reduce
This obviously requires a significant extension to the tracker protocol.
As long as I'm doing that, why not just add a couple messages to the
peer protocol to allow clients to be trackers? That way, the trusted
seed would also be the root tracker, and as the load increased, it could
appoint peers to be sub-trackers.
All this is nice, but first I think I'll just get the simple fuse based
thing working with existing torrents. Details of the extensions can be
worked out later.
More information about the BitTorrent