[bittorrent] Re: Intersecting Torrent data ???

David P. Mott dpmott at sep.com
Wed May 11 11:08:50 EDT 2005


On Tue, 10 May 2005, Charles Iliya Krempeaux wrote:


>> I can make hardlinks [SNIP]
>>
>> In this way, you can download from the fastest swarm (you have to choose
>> which one it is), and when you're done, you can seed all three from one
>> set of files (and some cleverly crafted symlinks).
>
> Wouldn't that take alot of intervention from the user? I.e., wouldn't the
> user have to know that the different torrents had the exact same file. If
> BitTorrent "identified" whole files (via sha1, sha256, md5, or something),
> it could be done automatically. (But as I understood it, "pieces" have sha1
> hashes, not entire files. And with the "framing" of the pieces of the files
> being potentially different from torrent to torrent, a list of sha1's
> wouldn't help either. Correct me if I am wrong though.)

Yup, it sure does.  And yes, given that the piece size may be different 
between the two torrents, none of the SHA1 hashes or anything else in the 
metafile will be the same, save for the file size.  If you have a general 
idea about the content, you can make some educated guesses.  If you're 
worried that the file may be different despite the same file size, you can 
download from one torrent, make a backup of the (partial) file, and then 
turn loose the other torrent on the file to see if any of the pieces 
validate.  (Note here that you'd want to start with the torrent that had 
the larger piece size, that way the torrent with the smaller piece size 
would find some complete pieces.  If you do it the other way, the torrent 
with the larger pieces may not find that anything is done).

The BT2 protocol (whenever that comes out, I've long ago stopped holding 
my breath) is supposed to provide an identifier (i.e. SHA1 Merkle root 
hash) for each file.  I don't recall if that is affected by piece size, 
though.  The nice thing about it is that pieces won't overlap files in the 
torrent, so you can download and (perhaps at a later date) verify 
individual files from a larger torrent.

>>
>> You could choose to make a "super torrent" that includes all possible
>> files, and then let peers use a bittorrent client that allows them to set
>> a "do not download" status on some of those files. Then, they can get the
>> files that they want, and none of the ones that they don't want.
>
>
> This is what I was thinking too. Much like you present your files with FTP
> or HTTP (and you pick and choose what you want to download), you could do
> the same with BitTorrent in one "super torrent" as you call it.
>
> It would also seem to allow more chances for swarming. Since a user would
> "contribute" to files they downloaded long ago from a "super torrent".
> Everytime they wanted to "download" something new from the "super torrent".
>


Check out http://wiki.theory.org/BitTorrentSpecification, and read the 
section on "Algorithms/Super Seeding".  A number of bittorrent clients 
support this.

You can also peruse the "Bittorrent Wish List" page here:
   http://wiki.theory.org/BitTorrentWishList

-dpmott



More information about the BitTorrent mailing list