[bittorrent] How should .torrent files with unsorted info dictionary keys be handled

Joris Guisson joris.guisson at gmail.com
Thu Jan 21 09:33:27 EST 2010

On Thu, Jan 21, 2010 at 12:38 PM, Harold Feit <dwknight at depthstrike.com> wrote:
> Dictionaries are for sorted content normally anyway. Lists are for
> unsorted. Having dictionary keys be unsorted, especially now with over 5
> years into the game of having them sorted, is quite troublesome.
> And honestly, out of the top client list, very few are as lax as
> KTorrent about bencoding spec compliance. In fact, it's the only client
> I know of that outright ignores key order. Everybody else is at least
> semi-strict about it.

We are only lax in the data we accept, not what we generate. Given the
current situation, I think this is the best thing you can do.
Rearranging is against the spec at the moment, and rejecting the
torrents would result in user complaints about torrents not working.

> And honestly, do we need another THOUSAND torrents all identifying the
> same file because of unordered dictionaries? There's enough trouble
> dealing with swarm fracturing as it is. This is just going to make
> things a LOT worse.

No we don't. OK, maybe dropping this requirement is not such a good idea.

But I don't like rearranging the keys, that is messy, and it is
basically working around the problem of buggy torrent generators.
If we are going to be stricter, lets be really strict and reject these
torrents. If a number of big name clients do this, then there will be
real pressure to fix this problem in the torrent generators.

Solutions, not workarounds.


> Joris Guisson wrote:
>> KTorrent computes the hash of the substring, ignoring any key order problems.
>> In fact, we ignore the key order everywhere for any bencoded data we
>> come across.
>> So I would be in favor of dropping the key order requirement.
>> Does anybody actually know the reason why the keys should be sorted ?
>> I don't see much reason to do so.
>> Joris,
>> On Wed, Jan 20, 2010 at 9:34 PM,  <arvid at cs.umu.se> wrote:
>>> Quoting Alan McGovern <alan.mcgovern at gmail.com>:
>>>> [...]
>>>> So pros and cons:
>>>> Approach 1:
>>>> Pro: This approach implies that if an invalid bencoded dictionary is found
>>>> it should be converted into a valid representation and used.
>>>> Pro: This should be relatively trivial for most clients to implement.
>>>> Con: The letter of the spec [2] says that we should always use a substring
>>>> of the .torrent metadata. Strictly speaking this approach goes against the
>>>> spec. However, we have to assume that when the spec refers to ".torrent
>>>> metadata' it refers to *specification compliant* .torrent metadata. i.e. the
>>>> keys in the metadata must be sorted for it to be considered valid .torrent
>>>> metadata. Conversely, if the keys are *not* sorted then the data should not
>>>> be considered valid .torrent metadata. If that's the case, then why are you
>>>> generating a valid infohash from invalid metadata?
>>>> Con: You need a non-spec compliant way of decoding bencoded data.
>>>> Approach 2:
>>>> Pro: You follow the letter of the bittorrent specification but still break
>>>> the BEncoding specification.
>>>> Con: I'd argue that this is slightly more complex to implement as you may
>>>> now need to double parse the .torrent file in order to generate the
>>>> infohash. The first time you pass it through your bencoded data decoder to
>>>> generate your in-memory representation. Then you have to parse the file
>>>> manually a second time to find the start and end of the info dictionary and
>>>> extract that substring.
>>>> Con: I'd argue that you're breaking the spirit of the specification again as
>>>> you're now running under the assumption that invalid bencoded data is valid
>>>> .torrent metadata and the spec doesn't explicitly allow this ;)
>>>> Approach 3:
>>>> Pro: Simplest to implement as you just have to make your bencoded data
>>>> decoder spec compliant. Once this happens, the .torrent won't be loadable so
>>>> that question of how to generate the infohash never has to be answered.
>>>> Con: It will render some torrents unloadable but those should be a very
>>>> small percentage.
>>> If you're talking about modifying the spec. Why not add:
>>> Approach 4:
>>> Modify the spec to allow unordered keys in dictionaries.
>>> Con: All torrents will work as intended (presumably the unordered keys are
>>> probably a mistake, and not a clever trick to make clients generate different
>>> info-hashes).
>>> Pro: some clients would have to be updated to be less strict on their input.
>>> --
>>> Arvid Norberg
>>> _______________________________________________
>>> BitTorrent mailing list
>>> BitTorrent at lists.ibiblio.org
>>> http://lists.ibiblio.org/mailman/listinfo/bittorrent
>> _______________________________________________
>> BitTorrent mailing list
>> BitTorrent at lists.ibiblio.org
>> http://lists.ibiblio.org/mailman/listinfo/bittorrent

More information about the BitTorrent mailing list