From stephen@blacksapphire.com Fri May 10 12:19:07 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Fri, 10 May 2002 23:19:07 +1200 Subject: Grapevine Technical Overview Message-ID: <20020510111949.A036732BE9@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 http://www.grapevineproject.org/ The Grapevine Project A decentralized peer-to-peer file storage network Technical Overview by Stephen Blackheath - 8 May 2002 This page aims to be a brief but fairly complete overview of 1. what the Grapevine does, and 2. how it works. A high school level of mathematics and some basic cryptography concepts may be necessary to understand some of the detail. The more widely this information is distributed and understood, the happier I am. Please note that it is protected by the GNU Lesser General Public Licence. This grants you the right to distribute the information freely for non-commercial purposes. I am very interested in any feedback or questions you may have about this information, so please contact me at stephen@blacksapphire.com. At the time of writing, the project is not complete. We need any support you can give us. What does the Grapevine do? The Grapevine is a network composed of individual computers running Grapevine software, which collaborate to store files. (This is quite different to "file sharing" that you may be familiar with.) Even though each machine may be unreliable or even malicious, the network as a whole is realiable. The network also does its best to conceal both the locations where files are stored, and the structure and traffic patterns of the network. The result is reliable, efficient and anonymous publishing and retrieval of information. What technical problems does it claim to solve? Peer-to-peer networking is a difficult problem - a fact that developers of the technology have learned from experience, and some newcomers have learned the hard way. Each project attacks each problem with varying success, so each must be taken on its individual merits. We claim that the Grapevine solves the following specific problems: * Scalability - The network is completely decentralized. Scalability is of the order N1/d where N is the number of nodes and d is around 6. * Traffic efficiency - From the user's point of view it is more efficient than most other peer-to-peer software, but less efficient than web surfing, largely because half the cost of web traffic is paid by the owner of the website. When the technology is mature, we expect the Grapevine will be more efficient than the Web overall because of its intelligent use of resources. * "Karmic debt" - each person gives in proportion to what they take. * Response time - Grapevine should perform more slowly than the Web, but with greater consistency. * Robustness - i.e. tolerance of the unreliability of individual nodes. We do not rely on any centralized infrastructure (except for Internet backbones). * Resistance to the [15]"Slashdot Effect" - i.e. the ability to cope well with files that are enormously commonly requested. * Resistance to Denial-Of-Service attacks - resistance to all attempts to shut the network down, legally sanctioned or otherwise. * Firewalls - not our first priority, but we have a solution to these problems. * Dynamic IP addresses * High throughput - we achieve fast downloads of large files through the very simple mechanism of splitting the files into pieces, and then downloading the pieces in parallel. * Search - one of the biggest problems for these networks, but in fact quite separate to the main problems. We have a solution in development, which we haven't documented yet. * Retrieval of data nearby - Ideally data should be retrieved physically nearby if possible. We have a solution to this, but implementing it will take a low priority. * Mix-netting - the problem of obscuring the workings of the network to anyone who can analyze traffic ("traffic analysis attacks") at the Internet backbone level. We have an almost complete solution to this problem. * Plausible deniability - God forbid that the Thought Police should bash down your door and confiscate your computer, but if this should happen, the files stored by the network are encrypted in such a way that you can plausibly deny any knowledge of what is stored there. The Holy Grail of peer-to-peer is steganography - making the network look as if it is not there at all by making the traffic look like something else. This is just a dream at present. I am not aware of any real work in this area, but there is sure to be some. Co-ordinate space The Grapevine is based upon a multi-dimensional co-ordinate space - and we choose six dimensions. This is difficult to visualize, but not difficult to understand. The argument for this runs as follows: Imagine you have 1 million people standing somewhere, and you want the ability to send a message to any chosen one of them, where each person can only speak to their immediate neighbours on all sides. Messages can be relayed from one person to the next. If you stand them in a straight line, then with each person passing the message on to the next person in line, it will take of the order of 1 million "relays" to get to any one person. If you stand them in a square of 1000 x 1000, then each has approximately 6 neighbours. You can reach any individual person by relaying messages with in the order of 1000 relays. Note that in the 1-dimensional straight-line example, dictating 6 immediate neighbours (i.e. skipping 2 out of 3 people each time) does not achieve this efficiency. We can take it one step further: If you stack the people up in a 100 x 100 x 100 cube, then it takes of the order of 100 relays for a message to reach any person. Since we are not dealing with real space, we can take this to as high a dimensionality as we like. We chose six dimensions, because it fits the size we think the network will get to, and it gives a convenient number of immediate neighbours. Routing Each node has a certain number of immediate neighbours, determined by the number of dimensions. In two dimensional space, it is 6. In three dimensional space, it is 15. In six-dimensional space, it is 118. A neighbour is defined as a node for which we know the IP address, port, and session key which are necessary to talk to it. Nodes and files both have a "location" in our co-ordinate space. Of course files must actually reside on nodes, but their "location" (which is an artificial concept) is between that of the nodes. Files are stored on the nodes "nearest" to it. (I will explain in more detail later.) When a node receives a request for a file, then if it does not have that file, it forwards the request on to its neighbour which is "nearest to" that file. For this calculation, we use Pythagoras's Theorem. When a node does this, it does not give the requesting node any information about where it has routed the request. It only tells it that it has done so. If the node to which we route the request is not responding, then we try the next closest node. Joining the network Nodes are only permitted to learn about their most immediate neighbours. The network protects itself by keeping all nodes ignorant of the IP addresses of any nodes further afield. In order to complete this protection, we also need a way to make sure a node cannot just join the network repeatedly in different locations to eventually find the IP addresses of all nodes. We do this with a strategy called Solve A Hard Problem or SAHP. I will not give the detail here - it is available elsewhere on the website. It has the following properties: 1. Each attempt takes a fixed time, and gives a certain "power". 2. The average time to calculate a solution with a certain power is proportional to that power. 3. One outcome of the solution is a location in co-ordinate space, and this becomes the location of the joining node. 4. Because of the above, it is possible to prove that you spent a certain amount of time calculating the solution. Here is an analogy: Imagine you have a telescope with a very high power, and want to search the sky for the brightest stars. Dim stars are easy to find, because they are common, but it takes a long time to find bright stars, since you have to painstakingly search a large area of sky. If you find a very bright star, then you can convincingly argue that it took you a long time to find it. The simplest approach to protecting the network is just to have a minimum "power" requirement before a new node can join. (We also have more elaborate strategies.) This ensures that a node does not get much control over what location in space it has. Of course any node can attempt to attack its immediate neighbours, but it is computationally difficult for any attacker to get a concentration of "cancer nodes" in any one region of the network. Storage of files Each node stores files in two ways: 1. The "permanent store". If a node is within a certain distance of the location of a file, then it is considered to be inside the "definitive zone" for that file. The node will store any such files it retrieves for a long time. 2. The "cache store". If a node is outside the definitive zone for a file, but it receives a file while processing a request for someone else, then it stores it for a shorter period than it would for the permanent store. These could be implemented as storage areas of a fixed size, where the least recently accessed files are deleted once the size is exceeded. Advertising and Mix-Netting in one Each node advertises its presence periodically to its neighbours. Because of the small number of neighbours, any node can be pretty sure which of its neighbours are up or down at any given moment with a very high efficiency. Neighbours also advertise what files they have in both their permanent and cache stores. These advertisements are used to pad out all traffic other than the contents of files to a fixed size. This helps protect against traffic analysis attacks. To further protect against traffic analysis attacks, we can introduce a delay in the forwarding of requests. At a cost of increasing the response time, this - combined with a judicious method of periodic random advertising - should allow the traffic to look almost completely random to anyone with the power of traffic analysis. Some work will be required to implement this fully. Advertising as a way to improve routing We can use the knowledge of what files our neighbours have as a means of improving routing. If we are asked for a certain file, then we might know that a certain one of our neighbours has that file. The other routing choice is to forward the request in the optimal direction. When we have two choices like this, we can branch the request. The branch that goes to the neighbour known to have the file will only go for one hop. Relaying of file contents Once a file is found on a certain node, the procedure is this: * A notification is sent back to the immediately requesting node. * If the immediately requesting node is within the "definitive zone" for this file, then the contents of the file are also sent. This ensures that commonly requested files are spread widely within the definitive zone. (For less commonly requested files, we should ideally add a mechanism to make sure there is always a minimum number of copies available on the network at any one time.) * The reply hops backwards along the request path. * The first node outside the "definitive zone" that does not already have the file designates itself as the "relay". The relay sends its IP address and port (but not its SAHP credentials, so it cannot be treated as a neighbour) along to the requesting node. As an alternative, it might be better for the requesting node to send its IP address and port, and for the relay to contact it directly. We now have three nodes: 1. The node on which the file was discovered 2. The relay 3. The requestor (the node of the person who wants the file) We have to get the file from the "discovered" node to the requestor via the relay. The reasons for this approach are as follows: 1. The relay acts as a cache for the network. Next time the file is requested, there will be a copy further out from the definitive zone. This means the network does not suffer from the [16]"Slashdot Effect". 2. The relay conceals the source of the file. We could choose multiple relays to increase this, but at the cost of overall network efficiency. One relay is the compromise we have chosen. 3. It allows us to deal with firewalls, as long as either 1. the relay, or 2. both the requestor and "discovered" node, are not firewalled. (We will also need a "buddy" system to allow firewalled nodes to receive incoming connections.) Publishing The routing for file storage (i.e. publishing) is the same as file retrieval. Once we reach the "definitive zone" for the file, we can put several copies of the file on the nodes there. We will use relaying to get the file there as with requesting. The identification of which nodes are involved will be the same, but the relaying will happen in reverse. Key management and plausible deniability First, we split files up into pieces of a standard size. This means that the size will not give away the contents of the file. We use two types of keys to store files: 1. Named keys - Keys calculated from a filename. 2. CHKs - Content Hash Keys, where the key of the file is the hash of the file contents. This allows nodes to easily check the authenticity of a file. Keys are translated into a location in our co-ordinate space. When we store a file, we do this: * We calculate the encryption key from the hash of its filename. * We encrypt the file with the encryption key (in CBC mode with a random initialization vector [IV] - have a look at any cryptography reference). * We calculate the storage key from the hash of the encryption key. * We store a 'file map' under the storage key. This is a list of CHKs, which are nothing more than the hash values of the contents of the blocks of the encrypted file. * We store each block under its CHK. * We pad all files to a standard block size for security reasons, e.g. 32K. In order for someone running a node to know what the contents of a file stored on their system is, they have to guess the filename of the file it belongs to, and then look up the file map and see if that CHK is in the file map. Even if they know the entire contents of the file they suspect, this is not sufficient information, due to the CBC mode initialization vector. Hence plausible deniability. Retrieval of data nearby We have a strategy for this. This involves connecting to several (perhaps three or four) separate network "bands", each of which has a restriction on the maximum response time of nodes. Requests start at the band with the fastest-responding nodes, and jumps to the next band if the request was not successful on that band. Retrieved files can be cached on each band, but this need not happen every time. The detail is at [17]Searching Physically Nearby. Karmic Debt To solve this problem, we introduce the concept of logical network "interfaces" which are analogous to TCP/IP interfaces. Each interface has a separate SAHP solution, a separate location and a separate set of neighbours. Your node "appears" logically in multiple places in the network. Each interface has three states: disconnected, freeloading and participating. When we download data, we adjust the rate of requests so that we receive a fixed bit-rate (including overheads) from each interface, for example 1K bytes/sec. If the user has a 56K bits/sec modem, for example, then this is about 5K bytes/sec, and so we would need to use 5 interfaces to achieve this download rate. (Remember that we download file chunks in parallel.) An interface must either be in the freeloading or participating state when it is being used for downloading. Nodes only send and forward requests and other traffic to nodes that are in the "participating" state. Only the barest minimum demands are made of "freeloading" nodes. A node ensures that it pays its way by ensuring that the total time multiplied by number of interfaces of downloading is ultimately repaid by an equal total time multiplied by number of interfaces spent in the "participating" state. Note that the node is not compelled to pay back its karma. It is quite easy to cheat the system by using modified software. (Though it is still better than most file-sharing software, where repayment of debt is under direct control by the user.) We will do more research in this area. Search We have a solution to the Napster-style keyword search problem, which we have not documented yet. Worms Last, but by no means least, we come to worms. Until peer-to-peer technology is implemented on secure hardware, it is especially vulnerable to worm attacks. This is a very similar problem to email viruses, but worse. Ultimately this technology will be implemented on Internet routers. Until that time, it must never be used for life-critical purposes. The problem is this: Microsoft Windows is not a very secure operating system, as evidenced by the ongoing problems with email worms. No software can protect itself from invasion of the machine on which it resides. Peer-to-peer is especially vulnerable, because it establishes an easily exploitable connectivity from every point in the network to every other. "Freedom of expression - Everyone has the right to freedom of expression, including the freedom to seek, receive, and impart information and opinions of any kind in any form." -- Section 14, New Zealand Bill of Rights Act 1990. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE826yz7I0ehz47OHERAqQSAJ9zlNxkHseSBn7G12gDHuS7TbKAWQCfRdzM u+whaKmU/0WXrJPUcaGvBEI= =SeO2 -----END PGP SIGNATURE----- From ajones@clear.net.nz Fri May 10 18:24:09 2002 From: ajones@clear.net.nz (Anthony Jones) Date: Sat, 11 May 2002 01:24:09 +0800 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020510172411.EC1751C117@escher.takahe.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday 10 May 2002 19:19, you wrote: > http://www.grapevineproject.org/ > > > The Grapevine Project > A decentralized peer-to-peer file storage network > > Technical Overview > > by Stephen Blackheath - 8 May 2002 > > This page aims to be a brief but fairly complete overview of > 1. what the Grapevine does, and > 2. how it works. > > A high school level of mathematics and some basic cryptography > concepts may be necessary to understand some of the detail. Anthony Jones has tried to explain as much as he can in a way that can be understood by everyone. If you can't understand what Steve has written or you just want a quick overview then look at what he's written. > The more widely this information is distributed and understood, the > happier I am. Please note that it is protected by the GNU Lesser > General Public Licence. This grants you the right to distribute the > information freely for non-commercial purposes. > > I am very interested in any feedback or questions you may have about > this information, so please contact me at > stephen@blacksapphire.com. > > At the time of writing, the project is not complete. We need any > support you can give us. > > What does the Grapevine do? > > The Grapevine is a network composed of individual computers running > Grapevine software, which collaborate to store files. (This is quite > different to "file sharing" that you may be familiar with.) Even > though each machine may be unreliable or even malicious, the network > as a whole is realiable. The network also does its best to conceal > both the locations where files are stored, and the structure and > traffic patterns of the network. The result is reliable, efficient and > anonymous publishing and retrieval of information. > > What technical problems does it claim to solve? > > Peer-to-peer networking is a difficult problem - a fact that > developers of the technology have learned from experience, and some > newcomers have learned the hard way. Each project attacks each problem > with varying success, so each must be taken on its individual merits. > We claim that the Grapevine solves the following specific problems: > * Scalability - The network is completely decentralized. Scalability > is of the order N1/d where N is the number of nodes and d is > around 6. This means that if a request takes one second on a network with a thousand nodes it will take two seconds on a million nodes and four seconds on a network billion nodes. > * Traffic efficiency - From the user's point of view it is more > efficient than most other peer-to-peer software, but less > efficient than web surfing, largely because half the cost of web > traffic is paid by the owner of the website. When the technology > is mature, we expect the Grapevine will be more efficient than the > Web overall because of its intelligent use of resources. This means that if you receive 1MB (a one minute song) then other computers on the network will usually have to transfer (send or receive) about 3MB in total. On the world wide web, for every 1MB you download there is a web site which is transferring that 1MB to you. This means that for every 1MB of data you get for yourself you will need to transfer about 3MB for other people. This would be an average figure over a long period of time. > * "Karmic debt" - each person gives in proportion to what they take. > * Response time - Grapevine should perform more slowly than the Web, > but with greater consistency. It might take longer between when you request a document and when it starts transferring compared to the world wide web. However it will usually only make a difference to the total download time of small files. > * Robustness - i.e. tolerance of the unreliability of individual > nodes. We do not rely on any centralized infrastructure (except > for Internet backbones). It shoud be fairly rare that you can't get the file that you want when you want it. [Note - this is a massively better than most file sharing and is expected to be similar to the web in performance] > * Resistance to the [15]"Slashdot Effect" - i.e. the ability to cope > well with files that are enormously commonly requested. Files will always be available no matter how popular they are. [Note - web sites that become suddenly popular often stop functioning] > * Resistance to Denial-Of-Service attacks - resistance to all > attempts to shut the network down, legally sanctioned or > otherwise. Even the people who wrote the software in the first place won't be able to shut it down once it gets established. > * Firewalls - not our first priority, but we have a solution to > these problems. You'll have no technical difficulties running it on any machine which is connected to the Internet. We're getting it working on machines which are directly connected to the Internet first. > * Dynamic IP addresses You'll be able to run it using either dial up or broadband connections. > * High throughput - we achieve fast downloads of large files through > the very simple mechanism of splitting the files into pieces, and > then downloading the pieces in parallel. It allows you to download different parts of the same file off different machines. This means that your download speeds will only be limited by the size of your own connection to the Internet. > * Search - one of the biggest problems for these networks, but in > fact quite separate to the main problems. We have a solution in > development, which we haven't documented yet. You will be able to do keyword searches and get your results within a few seconds. [Note - this will be slower than search engines but faster than most file sharing networks] > * Retrieval of data nearby - Ideally data should be retrieved > physically nearby if possible. We have a solution to this, but > implementing it will take a low priority. If someone in the same country as you already has the data you want then you won't have to download it from the United States or Europe. > * Mix-netting - the problem of obscuring the workings of the network > to anyone who can analyze traffic ("traffic analysis attacks") at > the Internet backbone level. We have an almost complete solution > to this problem. Your own business is your own business. It'll be pretty darn hard for anyone to spy on what you're doing. > * Plausible deniability - God forbid that the Thought Police should > bash down your door and confiscate your computer, but if this > should happen, the files stored by the network are encrypted in > such a way that you can plausibly deny any knowledge of what is > stored there. Files are torn up and scattered across the network. Each machine only has scraps of information - never a whole file. > The Holy Grail of peer-to-peer is steganography - making the network > look as if it is not there at all by making the traffic look like > something else. This is just a dream at present. I am not aware of any > real work in this area, but there is sure to be some. > Co-ordinate space > > The Grapevine is based upon a multi-dimensional co-ordinate space - > and we choose six dimensions. This is difficult to visualize, but not > difficult to understand. > > The argument for this runs as follows: Imagine you have 1 million > people standing somewhere, and you want the ability to send a message > to any chosen one of them, where each person can only speak to their > immediate neighbours on all sides. Messages can be relayed from one > person to the next. If you stand them in a straight line, then with > each person passing the message on to the next person in line, it will > take of the order of 1 million "relays" to get to any one person. If you have a small number of people this line is not a problem but by the time you get to a million people you've got a problem. A line is only one dimension and it only works for small numbers. > If you stand them in a square of 1000 x 1000, then each has > approximately 6 neighbours. You can reach any individual person by > relaying messages with in the order of 1000 relays. Note that in the > 1-dimensional straight-line example, dictating 6 immediate neighbours > (i.e. skipping 2 out of 3 people each time) does not achieve this > efficiency. In a square it's a lot better than before but messages still need to go a long way. A square has two dimensions and requires a lot less message passing. > We can take it one step further: If you stack the people up in a 100 x > 100 x 100 cube, then it takes of the order of 100 relays for a message > to reach any person. Dispite the obvious overcrowding a cube which has three dimensions is even better again. > Since we are not dealing with real space, we can take this to as high > a dimensionality as we like. We chose six dimensions, because it fits > the size we think the network will get to, and it gives a convenient > number of immediate neighbours. As we're using a computer we can do things which are not physically possible (and often hard to even think about). You should be able to figure this mathematical pattern, though: 1,000,000 = 1,000,000,000 a line in one dimension 1,000,000 = 1,000 x 1,000 a square in two dimensions 1,000,000 = 100 x 100 x 100 a cube in three dimensions 1,000,000 = about 32 x 32 x 32 x 32 in four dimensions 1,000,000 = about 16 x 16 x 16 x 16 x 16 in five dimensions 1,000,000 = about 10 x 10 x 10 x 10 x 10 x 10 in six dimensions And then we get go to 8, 6, 5 for 7 8 and 9 dimensions. However we've stopped at six dimensions so that we don't have to worry about having too many neighbours. > Routing > > Each node has a certain number of immediate neighbours, determined by > the number of dimensions. In two dimensional space, it is 6. In three > dimensional space, it is 15. In six-dimensional space, it is 118. A > neighbour is defined as a node for which we know the IP address, port, > and session key which are necessary to talk to it. 118 neighbours is a nice workable number. Too many would mean spending too much time staying in contact with all the nodes. Too few would mean that it may be hard to find a node which is running. > Nodes and files both have a "location" in our co-ordinate space. Of > course files must actually reside on nodes, but their "location" > (which is an artificial concept) is between that of the nodes. Files > are stored on the nodes "nearest" to it. (I will explain in more > detail later.) A node (i.e your computer) doesn't get to choose your location. It also doesn't get to choose the files that are in it's neighbourhood. It just gets the ones which happen to be nearby. > When a node receives a request for a file, then if it does not have > that file, it forwards the request on to its neighbour which is > "nearest to" that file. For this calculation, we use Pythagoras's > Theorem. When a node does this, it does not give the requesting node > any information about where it has routed the request. It only tells > it that it has done so. > > If the node to which we route the request is not responding, then we > try the next closest node. When a node gets asked for a file that it doesn't have, it will find a neighbour that is closer to the file. It will ask that neighbour for the file. > Joining the network > > Nodes are only permitted to learn about their most immediate > neighbours. The network protects itself by keeping all nodes ignorant > of the IP addresses of any nodes further afield. In order to complete > this protection, we also need a way to make sure a node cannot just > join the network repeatedly in different locations to eventually find > the IP addresses of all nodes. Basically all the nodes are on a need-to-know basis. If they don't need to know something then they're not given access to that information. > > We do this with a strategy called Solve A Hard Problem or SAHP. I will > not give the detail here - it is available elsewhere on the website. > It has the following properties: > 1. Each attempt takes a fixed time, and gives a certain "power". > 2. The average time to calculate a solution with a certain power is > proportional to that power. > 3. One outcome of the solution is a location in co-ordinate space, > and this becomes the location of the joining node. > 4. Because of the above, it is possible to prove that you spent a > certain amount of time calculating the solution. The point of this is to make it really hard for a node to choose it's own location. It takes a certain amount of time to choose a random location. If you want to choose a location then you'll have to keep choosing random locations until you get the one you want. This means that it's only hard to get a location if you're fussy about where you want it. > Here is an analogy: Imagine you have a telescope with a very high > power, and want to search the sky for the brightest stars. Dim stars > are easy to find, because they are common, but it takes a long time to > find bright stars, since you have to painstakingly search a large area > of sky. If you find a very bright star, then you can convincingly > argue that it took you a long time to find it. Here's a simpler analogy which neglects the power aspect. Imagine having a bag of marbles numbered from 1 to a million. If you are allowed to pick out one marble every five minute and you want to get near to a specific number then it's going to take you a very long time. If we want to think about power then we can imagine that each of the marbles is a different shade of grey - totally unrelated to the number. Let's decide that the darker the colour of the marble, the more power it represents. Normally you'll try to find the darkest (and most powerful) marbles. After a while you'll have a quite dark marble but you'll have no control over the number. If you're trying to choose your location at the same time then it'll take you a very long time indeed to find a dark marble near a specific number. > The simplest approach to protecting the network is just to have a > minimum "power" requirement before a new node can join. (We also have > more elaborate strategies.) This ensures that a node does not get much > control over what location in space it has. Of course any node can > attempt to attack its immediate neighbours, but it is computationally > difficult for any attacker to get a concentration of "cancer nodes" in > any one region of the network. We could make it so that your computer has to do about fifteen minutes calculation before it's allowed onto the network, however we're planning to make it so that you can get a from a friend so that you can get onto the network instantly. We will do our best to make it as easy to get connected as possible without compromising security too much. > > Storage of files > > Each node stores files in two ways: > 1. The "permanent store". If a node is within a certain distance of > the location of a file, then it is considered to be inside the > "definitive zone" for that file. The node will store any such > files it retrieves for a long time. > 2. The "cache store". If a node is outside the definitive zone for a > file, but it receives a file while processing a request for > someone else, then it stores it for a shorter period than it would > for the permanent store. > > These could be implemented as storage areas of a fixed size, where the > least recently accessed files are deleted once the size is exceeded. Nodes store files which are near them. They also store files that they've fetched on behalf of other nodes but these aren't stored for as long. > Advertising and Mix-Netting in one > > Each node advertises its presence periodically to its neighbours. > Because of the small number of neighbours, any node can be pretty sure > which of its neighbours are up or down at any given moment with a very > high efficiency. Nodes tell their neighbours when they get connected. This allows their neighbours to quickly and easily find them. > Neighbours also advertise what files they have in both their permanent > and cache stores. These advertisements are used to pad out all traffic > other than the contents of files to a fixed size. This helps protect > against traffic analysis attacks. All blocks are the same size so it's hard to work out what's going where. > To further protect against traffic analysis attacks, we can introduce > a delay in the forwarding of requests. At a cost of increasing the > response time, this - combined with a judicious method of periodic > random advertising - should allow the traffic to look almost > completely random to anyone with the power of traffic analysis. Some > work will be required to implement this fully. If we're really keen then we can even make fools of the intelligence angencies trying to watch people's every move. > Advertising as a way to improve routing > > We can use the knowledge of what files our neighbours have as a means > of improving routing. If we are asked for a certain file, then we > might know that a certain one of our neighbours has that file. The > other routing choice is to forward the request in the optimal > direction. When we have two choices like this, we can branch the > request. The branch that goes to the neighbour known to have the file > will only go for one hop. We've got a few tricks to speed things up. > Relaying of file contents > > Once a file is found on a certain node, the procedure is this: > * A notification is sent back to the immediately requesting node. > * If the immediately requesting node is within the "definitive zone" > for this file, then the contents of the file are also sent. This > ensures that commonly requested files are spread widely within the > definitive zone. (For less commonly requested files, we should > ideally add a mechanism to make sure there is always a minimum > number of copies available on the network at any one time.) > * The reply hops backwards along the request path. > * The first node outside the "definitive zone" that does not already > have the file designates itself as the "relay". The relay sends > its IP address and port (but not its SAHP credentials, so it > cannot be treated as a neighbour) along to the requesting node. As > an alternative, it might be better for the requesting node to send > its IP address and port, and for the relay to contact it directly. > > We now have three nodes: > 1. The node on which the file was discovered > 2. The relay > 3. The requestor (the node of the person who wants the file) > > We have to get the file from the "discovered" node to the requestor > via the relay. The reasons for this approach are as follows: > 1. The relay acts as a cache for the network. Next time the file is > requested, there will be a copy further out from the definitive > zone. This means the network does not suffer from the > [16]"Slashdot Effect". > 2. The relay conceals the source of the file. We could choose > multiple relays to increase this, but at the cost of overall > network efficiency. One relay is the compromise we have chosen. > 3. It allows us to deal with firewalls, as long as either 1. the > relay, or 2. both the requestor and "discovered" node, are not > firewalled. (We will also need a "buddy" system to allow > firewalled nodes to receive incoming connections.) There will always be a node in between you and where the file is stored so that you can't work out where the file came from. This is a compromise between performance and security. > Publishing > > The routing for file storage (i.e. publishing) is the same as file > retrieval. Once we reach the "definitive zone" for the file, we can > put several copies of the file on the nodes there. > > We will use relaying to get the file there as with requesting. The > identification of which nodes are involved will be the same, but the > relaying will happen in reverse. Unlike file sharing you insert a file into the network. After this act is performed there is no way to trace the file back to you unless you get "caught in the act" somehow or the contents of the file itself reveal your identity (like if you put your name at the top). > Key management and plausible deniability > > First, we split files up into pieces of a standard size. This means > that the size will not give away the contents of the file. > > We use two types of keys to store files: > 1. Named keys - Keys calculated from a filename. > 2. CHKs - Content Hash Keys, where the key of the file is the hash of > the file contents. This allows nodes to easily check the > authenticity of a file. A named key allows you to associate a file with a text name, kind of like a regular filename or URL string, e.g. "Bordello - Nice bloke.mp3". A CHK gives you a way to identify a file by it's contents. This means that if you know the CHK of a file (which will be very hard to type) then you will be guaranteed of getting the correct file. No one can tamper with it. > Keys are translated into a location in our co-ordinate space. > > When we store a file, we do this: > * We calculate the encryption key from the hash of its filename. > * We encrypt the file with the encryption key (in CBC mode with a > random initialization vector [IV] - have a look at any > cryptography reference). > * We calculate the storage key from the hash of the encryption key. > * We store a 'file map' under the storage key. This is a list of > CHKs, which are nothing more than the hash values of the contents > of the blocks of the encrypted file. > * We store each block under its CHK. > * We pad all files to a standard block size for security reasons, > e.g. 32K. When storing a file using a CHK we tear up the file and scatter it across the network. A given file will always generate the exact same key value. This key value gives us a location in six dimensional space. > In order for someone running a node to know what the contents of a > file stored on their system is, they have to guess the filename of the > file it belongs to, and then look up the file map and see if that CHK > is in the file map. Even if they know the entire contents of the file > they suspect, this is not sufficient information, due to the CBC mode > initialization vector. Hence plausible deniability. It's extremely difficult to work out what's stored on a given computer. Even if you it can be established that a computer has a certain piece of a file, that piece is meaningless without the rest of the pieces of the jigsaw. > Retrieval of data nearby > > We have a strategy for this. This involves connecting to several > (perhaps three or four) separate network "bands", each of which has a > restriction on the maximum response time of nodes. Requests start at > the band with the fastest-responding nodes, and jumps to the next band > if the request was not successful on that band. Retrieved files can be > cached on each band, but this need not happen every time. > > The detail is at [17]Searching Physically Nearby. It's faster to try the nodes in the same country before you start looking on the other side of the world. Even if it takes more hops it will probably be quicker. > Karmic Debt > > To solve this problem, we introduce the concept of logical network > "interfaces" which are analogous to TCP/IP interfaces. Each interface > has a separate SAHP solution, a separate location and a separate set > of neighbours. Your node "appears" logically in multiple places in the > network. > > Each interface has three states: disconnected, freeloading and > participating. > > When we download data, we adjust the rate of requests so that we > receive a fixed bit-rate (including overheads) from each interface, > for example 1K bytes/sec. If the user has a 56K bits/sec modem, for > example, then this is about 5K bytes/sec, and so we would need to use > 5 interfaces to achieve this download rate. (Remember that we download > file chunks in parallel.) > > An interface must either be in the freeloading or participating state > when it is being used for downloading. > > Nodes only send and forward requests and other traffic to nodes that > are in the "participating" state. Only the barest minimum demands are > made of "freeloading" nodes. > > A node ensures that it pays its way by ensuring that the total time > multiplied by number of interfaces of downloading is ultimately repaid > by an equal total time multiplied by number of interfaces spent in the > "participating" state. > > Note that the node is not compelled to pay back its karma. It is quite > easy to cheat the system by using modified software. (Though it is > still better than most file-sharing software, where repayment of debt > is under direct control by the user.) We will do more research in this > area. > > Search > > We have a solution to the Napster-style keyword search problem, which > we have not documented yet. > > Worms > > Last, but by no means least, we come to worms. Until peer-to-peer > technology is implemented on secure hardware, it is especially > vulnerable to worm attacks. This is a very similar problem to email > viruses, but worse. Ultimately this technology will be implemented on > Internet routers. Until that time, it must never be used for > life-critical purposes. Don't run Grapevine on the same machine that operates a crane to shift 10 tonne weights above your head. > The problem is this: Microsoft Windows is not a very secure operating > system, as evidenced by the ongoing problems with email worms. No > software can protect itself from invasion of the machine on which it > resides. Peer-to-peer is especially vulnerable, because it establishes > an easily exploitable connectivity from every point in the network to > every other. If you're using Windows then you obviously don't care about security. Install a virus checker, make regular backups and cross your fingers. You have nothing (more than the usual) to worry about. If you're concerned about security then make sure Grapevine is run as a separate user which has minimal access to system resources. > "Freedom of expression - Everyone has the right to freedom of > expression, including the freedom to seek, receive, and impart > information and opinions of any kind in any form." > -- Section 14, New Zealand Bill of Rights Act 1990. Everyone has the right to freedom of expression in Australia as long as you don't step out of line and say anything contravercial, in which case make sure you've got your air tickets booked. > > --- > You are currently subscribed to bluesky as: ajones@clear.net.nz > For list information visit http://www.transarc.ibm.com/~ota/bluesky/ > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE83AI5hwVaoilFPn0RAjzoAJwIvJc6tAwjlf24rnKX+CHPmb0Y6gCgikT8 HaN6luEDoTC5to6rFn6feT0= =+ul6 -----END PGP SIGNATURE----- From mccoy@io.com Fri May 10 19:50:28 2002 From: mccoy@io.com (Jim McCoy) Date: Fri, 10 May 2002 11:50:28 -0700 Subject: Grapevine Technical Overview In-Reply-To: Message-ID: On 5/10/02 4:19 AM, "Stephen Blackheath" wrote: > What technical problems does it claim to solve? [...] Part of my curiosity about Grapevine is that it seems to be a weird hybrid of MojoNation/Mnet and Freenet. I am still trying to figure out what problems it solves that are not already solved by one or the other system. As each of these two parent systems incorporates more features from the other as their development continues I can't quite understand what niche need this system is supposed to be fulfilling. To make the security/anonymity claims you are making here you need a large pool of hosts running the software, but unless there is a significant win over other existing systems there seems to be little compelling reason for someone to do so. I guess the big question here is this: outside of a cute university research project what is the market for such a system? Who needs this and why? Having gotten that question out of the way I will now pick a bit at some of the claims you are making because my morning mocha was delivered late and cold and so I am a bit cranky (and I am not going to risk disrupting my caffeine supply by bitching at the surly barrista :) > We claim that the Grapevine solves the following specific problems: > * Scalability > * Traffic efficiency > * "Karmic debt" > * Response time > * Robustness > * Resistance to the [15]"Slashdot Effect" > * Resistance to Denial-Of-Service attacks > * Firewalls > * Dynamic IP addresses > * High throughput All of these problems are already solved (and implemented and tested) by features in MojoNation/Mnet. Given that you are walking a path already explored by this system you might really want to take a look at how it worked and ask around to see what problems you are going to run into here... > * Search This is the hard part for decentralized networks and one which I think than Freenet work solved better than others. Information about proposed solutions to the decentralized search problem would be really appreciated. > * Retrieval of data nearby Freenet did this well, but I think that this represents a fundamental conflict with your anonymity claims. If I know what information is held by neighbors then that is the first step in attacking either those sites or the availability of the information itself. It will be hard to re-create the proper balance between privacy/anonymity and pulling from neighbors that Freenet accomplished without just copying the Freenet mechanism. > * Mix-netting What is the cost of your solution here? Any scheme which claims to provide strong anonymity at the packet level and mixnet features will _always_ suffer from latency problems and an increase in bandwidth costs. This is not negotiable. How does this feature not stomp all over your previous claims regarding response time and retrieval of data from neighbors? > * Plausible deniability Plausible deniability claims are not a panacea for legal liability. Specifically you would be wise to look around at concepts like "attractive nuisance" and "vicarious copyright infringement", especially before you plan on any trips to the US or Europe :) Seriously, if you are going to make any claims about the legal protections your system might offer I would suggest that you talk to some good lawyers. If you want I can direct you to a few here in the US who have already gone through the learning curve of understanding P2P systems because we (MojoNation) had to walk them through it ourselves. [...] > > When a node receives a request for a file, then if it does not have > that file, it forwards the request on to its neighbour which is > "nearest to" that file. Ouch. Wave goodbye to that fast response time claim you made earlier. This conflict between forwarded requests and direct requests for data seems to be a fundamental balancing act that P2P data systems have to decide one way or the other. Systems that favor privacy/anonymity like Freenet gain privacy benefits by forwarding requests at the cost of increasing latency. How is it that Grapevine will avoid these costs? > Joining the network > > Nodes are only permitted to learn about their most immediate > neighbours. The network protects itself by keeping all nodes ignorant > of the IP addresses of any nodes further afield. In order to complete > this protection, we also need a way to make sure a node cannot just > join the network repeatedly in different locations to eventually find > the IP addresses of all nodes. > > We do this with a strategy called Solve A Hard Problem [...] How many nodes does the network need have to prevent me from running a batch of hosts in parallel solving the hard problems and then mapping out the network? Unless the network is rather large I think that you are underestimating how hard it will be to map out the connections between nodes. If the hard problem only needs to be solved once by a node then I bet I could outrun your real new node growth rate with a good cluster of PCs doing nothing but creating virtual nodes to map out the mesh. If the hard problem needs to be re-calculated after a disconnect then how much damage could I do to the network by mapping it partially and then taking out those nodes I find which are well-connected? If the cost of re-balancing/re-connecting the network when key nodes disappear is large enough then I can use the SAHP protection mechanism against the network itself (e.g. bounce a few nodes out of the mesh and then use the delay in their return to map the remainder of the network a little bit faster.) > Key management and plausible deniability [...] > In order for someone running a node to know what the contents of a > file stored on their system is, they have to guess the filename of the > file it belongs to, and then look up the file map and see if that CHK > is in the file map. Even if they know the entire contents of the file > they suspect, this is not sufficient information, due to the CBC mode > initialization vector. Hence plausible deniability. Actually you are just shifting the legal burden from the hosts with the data to the person doing the mapping from filename to file map. This was the solution that we used for MojoNation/Mnet and while it solves some legal problems it creates a centralized point for attack (legal and network attacks) which your network is claiming to not have. If the name->map function is distributed then hosts that expected to be able to deny knowledge of what is on their servers could find themselves taking on a hidden legal liability due to the name->map bits that are on their nodes. The system sounds interesting from a theoretical point of view but there seems to be some claims made here that are in conflict with other claims. Clearing some of this up would be greatly appreciated. Jim From ajones@clear.net.nz Sat May 11 07:35:07 2002 From: ajones@clear.net.nz (Anthony Jones) Date: Sat, 11 May 2002 14:35:07 +0800 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020511063511.5F2541C117@escher.takahe.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday 11 May 2002 02:50, Jim: > Part of my curiosity about Grapevine is that it seems to be a weird hybrid > of MojoNation/Mnet and Freenet. I am still trying to figure out what > problems it solves that are not already solved by one or the other system. > As each of these two parent systems incorporates more features from the > other as their development continues I can't quite understand what niche > need this system is supposed to be fulfilling. To make the > security/anonymity claims you are making here you need a large pool of hosts > running the software, but unless there is a significant win over other > existing systems there seems to be little compelling reason for someone to > do so. I guess the big question here is this: outside of a cute university > research project what is the market for such a system? Who needs this and > why? Grapevine is in direct competition with Freenet. We have the same goals as Freenet but we think that we can do it better. Steve and myself have both contributed to Freenet but we decided that Grapevine would be different enough to justify starting again from the beginning. > All of these problems are already solved (and implemented and tested) by > features in MojoNation/Mnet. Given that you are walking a path already > explored by this system you might really want to take a look at how it > worked and ask around to see what problems you are going to run into here... > > > * Search > > This is the hard part for decentralized networks and one which I think than > Freenet work solved better than others. Information about proposed > solutions to the decentralized search problem would be really appreciated. We have an idea which we think will work. I will read up on the Freenet searching - if it looks like they've got a better solution then we will use that. If not then we will try our idea. I hope to have something on searching soon. > > * Retrieval of data nearby This is supposed to be physically near-by not etherally. > Freenet did this well, but I think that this represents a fundamental > conflict with your anonymity claims. If I know what information is held by > neighbors then that is the first step in attacking either those sites or the > availability of the information itself. It will be hard to re-create the > proper balance between privacy/anonymity and pulling from neighbors that > Freenet accomplished without just copying the Freenet mechanism. You're right about keeping information about what your neighbours are storing. It is perhaps a little risky. > > * Mix-netting > > What is the cost of your solution here? Any scheme which claims to provide > strong anonymity at the packet level and mixnet features will _always_ > suffer from latency problems and an increase in bandwidth costs. This is > not negotiable. How does this feature not stomp all over your previous > claims regarding response time and retrieval of data from neighbors? You're right. It's not possible to achieve both. We're aiming for a little mix netting. > > * Plausible deniability > > Plausible deniability claims are not a panacea for legal liability. > Specifically you would be wise to look around at concepts like "attractive > nuisance" and "vicarious copyright infringement", especially before you plan > on any trips to the US or Europe :) Seriously, if you are going to make any > claims about the legal protections your system might offer I would suggest > that you talk to some good lawyers. If you want I can direct you to a few > here in the US who have already gone through the learning curve of > understanding P2P systems because we (MojoNation) had to walk them through > it ourselves. We're just towing the same line as Freenet at the moment. If I think I'm going to get arrested then I will simply stay away from the US or Europe. If you're willing to publish the legal advice that you have recieved then I'm sure that this would be helpful to all peer to peer projects. > > [...] > > > > When a node receives a request for a file, then if it does not have > > that file, it forwards the request on to its neighbour which is > > "nearest to" that file. > > Ouch. Wave goodbye to that fast response time claim you made earlier. This > conflict between forwarded requests and direct requests for data seems to be > a fundamental balancing act that P2P data systems have to decide one way or > the other. Systems that favor privacy/anonymity like Freenet gain privacy > benefits by forwarding requests at the cost of increasing latency. How is > it that Grapevine will avoid these costs? We're expecting to go through about six nodes getting to our destination and only have one or two intermediate nodes on the way back. As we are planning to keep up-to-date lists of which neighbours are up and their latest IP addresses then we expect that we will get significantly better performance than Freenet. > > Joining the network > > > > Nodes are only permitted to learn about their most immediate > > neighbours. The network protects itself by keeping all nodes ignorant > > of the IP addresses of any nodes further afield. In order to complete > > this protection, we also need a way to make sure a node cannot just > > join the network repeatedly in different locations to eventually find > > the IP addresses of all nodes. > > > > We do this with a strategy called Solve A Hard Problem [...] > > How many nodes does the network need have to prevent me from running a batch > of hosts in parallel solving the hard problems and then mapping out the > network? Unless the network is rather large I think that you are > underestimating how hard it will be to map out the connections between > nodes. If the hard problem only needs to be solved once by a node then I > bet I could outrun your real new node growth rate with a good cluster of PCs > doing nothing but creating virtual nodes to map out the mesh. Let's assume that we're going to have a million nodes with a difinitive zone of 100 nodes (it's actually supposed to be 118 but the maths is easier for 100). Let's say we set the minimum SAHP value to be 1 week on an average computer. To get any address it will take a week. To get every group of 100 it will take 10,000 times the computing power (or perhaps more). This means that if you have 1,000 times the computing power of a single computer then you can map the network in about 10 weeks. To improve this we are making it so that it's more difficult to find nodes which have been processing for longer. That is - if you've been processing for a week then you can't see nodes that have been processing for six months. So we can say that it's hard to map but not impossible. > If the hard problem needs to be re-calculated after a disconnect then how > much damage could I do to the network by mapping it partially and then > taking out those nodes I find which are well-connected? If the cost of > re-balancing/re-connecting the network when key nodes disappear is large > enough then I can use the SAHP protection mechanism against the network > itself (e.g. bounce a few nodes out of the mesh and then use the delay in > their return to map the remainder of the network a little bit faster.) Nope - that attack won't work. The SAHP is not recalculated each time. > > Key management and plausible deniability > [...] > > In order for someone running a node to know what the contents of a > > file stored on their system is, they have to guess the filename of the > > file it belongs to, and then look up the file map and see if that CHK > > is in the file map. Even if they know the entire contents of the file > > they suspect, this is not sufficient information, due to the CBC mode > > initialization vector. Hence plausible deniability. > > Actually you are just shifting the legal burden from the hosts with the data > to the person doing the mapping from filename to file map. This was the > solution that we used for MojoNation/Mnet and while it solves some legal > problems it creates a centralized point for attack (legal and network > attacks) which your network is claiming to not have. If the name->map > function is distributed then hosts that expected to be able to deny > knowledge of what is on their servers could find themselves taking on a > hidden legal liability due to the name->map bits that are on their nodes. The name mapping is mangled so you can't know what names you've got - same as Freenet. Freenet has the advantage that the name->CHK mapping floats around the network. It will do this on Grapevine too as new SAHP solutions become available but not quite in the same way. > The system sounds interesting from a theoretical point of view but there > seems to be some claims made here that are in conflict with other claims. > Clearing some of this up would be greatly appreciated. Thanks for your input. It is much appreciated. Regards Anthony Jones -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE83LuehwVaoilFPn0RAut0AJ9r19nmpv4EbRWX2XCbspjDKUScNACgmoJB SjHgv7XjUMcugC2PloN8gL8= =U8Hh -----END PGP SIGNATURE----- From stephen@blacksapphire.com Sun May 12 03:27:59 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Sun, 12 May 2002 14:27:59 +1200 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020512024348.6B8EB3310D@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (Tony - thanks for those extra notes you posted.) Jim, On Sat, 11 May 2002 06:50, Jim McCoy wrote: > Part of my curiosity about Grapevine is that it seems to be a weird hybrid > of MojoNation/Mnet and Freenet. I am pleased that I have aroused your curiosity. The Grapevine is weird (which is hardly surprising given that we are weird people), but I am not so sure about the other accusation. > I am still trying to figure out what > problems it solves that are not already solved by one or the other system. > As each of these two parent systems incorporates more features from the > other as their development continues I can't quite understand what niche > need this system is supposed to be fulfilling. > > To make the > security/anonymity claims you are making here you need a large pool of > hosts running the software, but unless there is a significant win over > other existing systems there seems to be little compelling reason for > someone to do so. I guess the big question here is this: outside of a cute > university research project what is the market for such a system? Who > needs this and why? I can't make you acknowledge our project as an equal, but I can request that you treat us with a little respect. You may want to have a look at our website. I can say that our goals have a lot more in common with Freenet than with MojoNation. > Having gotten that question out of the way I will now pick a bit at some of > the claims you are making because my morning mocha was delivered late and > cold and so I am a bit cranky (and I am not going to risk disrupting my > caffeine supply by bitching at the surly barrista :) Perhaps you want to think about cutting down a little. > > We claim that the Grapevine solves the following specific problems: > > * Scalability > > * Traffic efficiency > > * "Karmic debt" > > * Response time > > * Robustness > > * Resistance to the [15]"Slashdot Effect" > > * Resistance to Denial-Of-Service attacks > > * Firewalls > > * Dynamic IP addresses > > * High throughput > > All of these problems are already solved (and implemented and tested) by > features in MojoNation/Mnet. Given that you are walking a path already > explored by this system you might really want to take a look at how it > worked and ask around to see what problems you are going to run into > here... I looked about seven months ago, but could find very little technical information on the MojoNation website. I just looked now, but the website is down. You can be sure that I will look again. > > * Search > > This is the hard part for decentralized networks and one which I think than > Freenet work solved better than others. Information about proposed > solutions to the decentralized search problem would be really appreciated. We'll do our best to oblige, time-permitting. > > * Retrieval of data nearby > > Freenet did this well, but I think that this represents a fundamental > conflict with your anonymity claims. If I know what information is held by > neighbors then that is the first step in attacking either those sites or > the availability of the information itself. It will be hard to re-create > the proper balance between privacy/anonymity and pulling from neighbors > that Freenet accomplished without just copying the Freenet mechanism. I am not an expert on MojoNation, but I can claim to be at least a partial expert on Freenet. To the best of my knowledge, Freenet does not solve the "retrieving information physically nearby" problem. Have a good look at what we're doing, and I think you'll see that it has its merits over the Freenet mechanism. In our context, I can't see how the knowledge of what files the neighbours have could be an avenue of attack. > > * Mix-netting > > What is the cost of your solution here? Any scheme which claims to provide > strong anonymity at the packet level and mixnet features will _always_ > suffer from latency problems and an increase in bandwidth costs. This is > not negotiable. How does this feature not stomp all over your previous > claims regarding response time and retrieval of data from neighbors? I agree that latency is affected - I see it as a tradeoff. Without going into detail, I am not sure I agree that bandwidth is seriously affected. As I've explained, we sort of kill two birds with one stone. > > * Plausible deniability > > Plausible deniability claims are not a panacea for legal liability. > Specifically you would be wise to look around at concepts like "attractive > nuisance" and "vicarious copyright infringement", especially before you > plan on any trips to the US or Europe :) Seriously, if you are going to > make any claims about the legal protections your system might offer I would > suggest that you talk to some good lawyers. If you want I can direct you > to a few here in the US who have already gone through the learning curve of > understanding P2P systems because we (MojoNation) had to walk them through > it ourselves. My answer may surprise you, and perhaps it gives a clue as to what our "niche" is: I am not making claims about legal protections offered by the Grapevine. I am making claims about plausible deniability. I do not care about the legal issues you mention, with one exception: I am almost completely certain that I am in full compliance with New Zealand law in developing this project. Another way to put this: You can't make an omelette without breaking a few eggs. > [...] > > > When a node receives a request for a file, then if it does not have > > that file, it forwards the request on to its neighbour which is > > "nearest to" that file. > > Ouch. Wave goodbye to that fast response time claim you made earlier. > This conflict between forwarded requests and direct requests for data seems > to be a fundamental balancing act that P2P data systems have to decide one > way or the other. Systems that favor privacy/anonymity like Freenet gain > privacy benefits by forwarding requests at the cost of increasing latency. > How is it that Grapevine will avoid these costs? It's a trade-off of security vs. efficiency. > > Joining the network > > > > Nodes are only permitted to learn about their most immediate > > neighbours. The network protects itself by keeping all nodes ignorant > > of the IP addresses of any nodes further afield. In order to complete > > this protection, we also need a way to make sure a node cannot just > > join the network repeatedly in different locations to eventually find > > the IP addresses of all nodes. > > > > We do this with a strategy called Solve A Hard Problem [...] > > How many nodes does the network need have to prevent me from running a > batch of hosts in parallel solving the hard problems and then mapping out > the network? Unless the network is rather large I think that you are > underestimating how hard it will be to map out the connections between > nodes. If the hard problem only needs to be solved once by a node then I > bet I could outrun your real new node growth rate with a good cluster of > PCs doing nothing but creating virtual nodes to map out the mesh. > > If the hard problem needs to be re-calculated after a disconnect then how > much damage could I do to the network by mapping it partially and then > taking out those nodes I find which are well-connected? If the cost of > re-balancing/re-connecting the network when key nodes disappear is large > enough then I can use the SAHP protection mechanism against the network > itself (e.g. bounce a few nodes out of the mesh and then use the delay in > their return to map the remainder of the network a little bit faster.) You're quite right. As I have described it it only offers a certain level of protection against attack. I never said it was going to be easy. We have some more elaborate strategies in the pipeline that should improve things (see the website). Also see Tony's explanation in a previous email. > > Key management and plausible deniability > > [...] > > > In order for someone running a node to know what the contents of a > > file stored on their system is, they have to guess the filename of the > > file it belongs to, and then look up the file map and see if that CHK > > is in the file map. Even if they know the entire contents of the file > > they suspect, this is not sufficient information, due to the CBC mode > > initialization vector. Hence plausible deniability. > > Actually you are just shifting the legal burden from the hosts with the > data to the person doing the mapping from filename to file map. This was > the solution that we used for MojoNation/Mnet and while it solves some > legal problems it creates a centralized point for attack (legal and network > attacks) which your network is claiming to not have. If the name->map > function is distributed then hosts that expected to be able to deny > knowledge of what is on their servers could find themselves taking on a > hidden legal liability due to the name->map bits that are on their nodes. I hadn't considered that: a CHK is obviously a CHK, while a name->map mapping is obviously not, and this goes against plausible deniability. I'm sure it can't be too hard to think of a solution to this. For your interest, one thing we are considering doing (as I think Tony explained) is having the most trusted nodes both 1. concealed from the rest of the network and 2. performing the name->map mapping. > The system sounds interesting from a theoretical point of view but there > seems to be some claims made here that are in conflict with other claims. > Clearing some of this up would be greatly appreciated. I hope I've made a start on that in this message. Steve -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE83dMv7I0ehz47OHERAmXiAJ9+u73NhI2SvRnlRqwbl2SlbaUatACgxsHu Q3j1PVDr9sdOLpGZeJoFQeI= =IiBd -----END PGP SIGNATURE----- From bram@bitconjurer.org Wed May 22 20:39:37 2002 From: bram@bitconjurer.org (Bram Cohen) Date: Wed, 22 May 2002 12:39:37 -0700 (PDT) Subject: Grapevine Technical Overview Message-ID: Someone sent me this from an anonymous email account, asking me to forward it to the list, since bluesky isn't set up to accept anonymous mail. === Jim McCoy writes: > Part of my curiosity about Grapevine is that it seems to be a weird > hybrid of MojoNation/Mnet and Freenet. I am still trying to figure > out what problems it solves that are not already solved by one or the > other system. Freenet doesn't work. The theory has never been verified and in practice it runs as a full broadcast network. It has long passed the point of scalability and is now completely unusable. Most nodes are instantly rejecting 99%+ of requests because they are overloaded. MojoNation never worked either. Even if Grapevine does no more than what Freenet and MonoJantion promised to do, it will be a huge step forward if it actually succeeds. > > * Search > > This is the hard part for decentralized networks and one which I think > than Freenet work solved better than others. Information about > proposed solutions to the decentralized search problem would be really > appreciated. On the contrary, Freenet has no capabilities for search, and has no design for search that anyone agrees is likely to work. From oskar@freenetproject.org Wed May 22 21:56:33 2002 From: oskar@freenetproject.org (Oskar Sandberg) Date: Wed, 22 May 2002 22:56:33 +0200 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020522205633.GB18057@sporty.spiceworld> On Wed, May 22, 2002 at 12:39:37PM -0700, Bram Cohen wrote: > Someone sent me this from an anonymous email account, asking me to forward > it to the list, since bluesky isn't set up to accept anonymous mail. Ah, of course, your anonymous friend. My anonymous friend has some responses :-). <> > Jim McCoy writes: > > Part of my curiosity about Grapevine is that it seems to be a weird > > hybrid of MojoNation/Mnet and Freenet. I am still trying to figure > > out what problems it solves that are not already solved by one or the > > other system. > > Freenet doesn't work. The theory has never been verified and in practice > it runs as a full broadcast network. It has long passed the point of > scalability and is now completely unusable. Most nodes are instantly > rejecting 99%+ of requests because they are overloaded. The routing lacks a strict mathematical proof (though there has been some rather heuristic hand-waving regarding small world networks), but there is an abundance of experimental evidence that indicates that it does work at least to some extent (I think I know of at least 5 independent groups that have simulated it, and each has come to this conclusion). Whether that can be considered verification is a personal choice of course, I would probably say no, but I have never been able to convince myself that it cannot work either. The current attempt at a real world implementation is indeed at the moment suffering from tremendous overload that is keeping it from functioning at all. However, it would be incorrect to blame this problem on the scaling of the routing - the current implementation is limited in search length, so evidence of a scalability limit would be seen as decreased performance not as increased traffic. The cause of the problem at the moment seems to be a combination of inefficient software, lack of a compelling reason for users to share resources, and/or deliberate flooding. > MojoNation never worked either. > > Even if Grapevine does no more than what Freenet and MonoJantion promised > to do, it will be a huge step forward if it actually succeeds. If Grapevine does it FIRST, then it will be a huge step forward (maybe). We are all vapourware here... < > > > This is the hard part for decentralized networks and one which I think > > than Freenet work solved better than others. Information about > > proposed solutions to the decentralized search problem would be really > > appreciated. > > On the contrary, Freenet has no capabilities for search, and has no > design for search that anyone agrees is likely to work. This is pretty true, though there is some interesting work on the topic published here: http://www.cs.princeton.edu/~akronfol/fasd/ -- Oskar Sandberg oskar@freenetproject.org From stephen@blacksapphire.com Wed May 22 23:12:06 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Thu, 23 May 2002 10:12:06 +1200 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020522221257.8050132BE8@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oskar & everyone, Anonymous writes: > > MojoNation never worked either. > > > > Even if Grapevine does no more than what Freenet and MonoJantion promised > > to do, it will be a huge step forward if it actually succeeds. On Thu, 23 May 2002 08:56, Oskar Sandberg wrote: > If Grapevine does it FIRST, then it will be a huge step forward (maybe). > We are all vapourware here... The Grapevine (the project that Anthony Jones, Ross Smith and I are building) is most definitely vapourware, and the only proof of it is a small experiment I did last year in which I showed to my own satisfaction (and obviously also Ross's and Anthony's) that the basic principle of the routing works - that is, messages routed through the network reach their destination 100% of the time (in my SMALL experiment). I want to make it absolutely clear that I honestly do not care whose system achieves success first, and Anthony definitely feels that way. That is why I have gone to so much trouble to make the technical aspects of Grapevine accessible. I imagine that some time in the future, all our current efforts will be obsolete, and another system composed of the best parts of all of them will triumph. The mass media is happy to declare that peer to peer is dead, because that idea suits the copyright industry that owns it. But of course there are some very successful non-anonymizing systems, such as FastTrak (Morpheus, Kazaa, ...) and E-Donkey. To a large extend what we are trying to achieve now is credibility for the technology itself - "third-generation P2P" if you will. That in turn will encourage more technical innovation, just as I was inspired by the great work of Ian Clarke, Scott Miller, and Oskar Sandberg of the Freenet project. Steve -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE87Be27I0ehz47OHERAg7lAJ9hb2szkn9YhoiE92Lk3muIrT/s3gCeM3fR jqUqHr0COx/RXETOlmPRTfM= =7Fhr -----END PGP SIGNATURE----- From ajones@clear.net.nz Thu May 23 16:40:32 2002 From: ajones@clear.net.nz (Anthony Jones) Date: Thu, 23 May 2002 23:40:32 +0800 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020523154035.28AA41C116@escher.takahe.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday 23 May 2002 04:56, you wrote: > On Wed, May 22, 2002 at 12:39:37PM -0700, Bram Cohen wrote: > > Someone sent me this from an anonymous email account, asking me to forward > > it to the list, since bluesky isn't set up to accept anonymous mail. > > Ah, of course, your anonymous friend. My anonymous friend has some > responses :-). Anonymity is fine by me. That's the whole point of Grapevine. > <> > > Jim McCoy writes: > > > Part of my curiosity about Grapevine is that it seems to be a weird > > > hybrid of MojoNation/Mnet and Freenet. I am still trying to figure > > > out what problems it solves that are not already solved by one or the > > > other system. > > > > Freenet doesn't work. The theory has never been verified and in practice > > it runs as a full broadcast network. It has long passed the point of > > scalability and is now completely unusable. Most nodes are instantly > > rejecting 99%+ of requests because they are overloaded. > > The routing lacks a strict mathematical proof (though there has been > some rather heuristic hand-waving regarding small world networks), but > there is an abundance of experimental evidence that indicates that it > does work at least to some extent (I think I know of at least 5 > independent groups that have simulated it, and each has come to this > conclusion). Whether that can be considered verification is a personal > choice of course, I would probably say no, but I have never been able to > convince myself that it cannot work either. Freenet definitely works very well in simulation, as it would on an ideal network. In the real world it does not work at the moment. This does not mean that it can't be made to work. On paper it is more efficient than Grapevine. Grapevine is based very heavily upon our experiences with Freenet. I feel that we've taken the good ideas from Freenet and that we've addressed some of it's deficiencies. We even discussed leveraging the existing Freenet codebase (or perhaps the C++ port) when we first talked about doing the project. I would like to thank everyone who worked on Freenet for the inspiration it has given our project. I don't expect anyone to take Grapevine all that seriously until we've got something to show for ourselves. I am very pleased with the feedback that we've had from posting to this list - even though a lot of it has been very negative. I have been made aware of many potential problems that may have otherwise gone un-noticed. If we can get to the stage where even one of these project really works then we can all learn a lot by the successes and failures of the project, as we have with Gnutella, Napster and so on. Anthony -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE87Q1whwVaoilFPn0RAgF7AKCjbgXoDuh5NDeImLw5V5PQ0X1uJQCfbDOl z/DpOthN/Gvuq2uLiNOgcRk= =Qyya -----END PGP SIGNATURE----- From oskar@freenetproject.org Thu May 23 17:25:40 2002 From: oskar@freenetproject.org (Oskar Sandberg) Date: Thu, 23 May 2002 18:25:40 +0200 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020523162540.GC18057@sporty.spiceworld> On Thu, May 23, 2002 at 11:40:32PM +0800, Anthony Jones wrote: > On Thursday 23 May 2002 04:56, you wrote: < > > > The routing lacks a strict mathematical proof (though there has been > > some rather heuristic hand-waving regarding small world networks), but > > there is an abundance of experimental evidence that indicates that it > > does work at least to some extent (I think I know of at least 5 > > independent groups that have simulated it, and each has come to this > > conclusion). Whether that can be considered verification is a personal > > choice of course, I would probably say no, but I have never been able to > > convince myself that it cannot work either. > > Freenet definitely works very well in simulation, as it would on an ideal > network. In the real world it does not work at the moment. This does not mean > that it can't be made to work. On paper it is more efficient than Grapevine. The implementation is a lot more than just the routing algorithm. I would argue that the routing is not being tested one way or the other "in the real world" at the moment, because other parts of the implementation have failed to give it a chance. It would be very easy to have a network implementation fall over even if the routing algorithm were deterministic and (in controlled conditions) 100% successful. In fact, the situation may in many ways be worse for such a network, since freenet can route around moderate numbers of nodes that come in and out of eligibility due to overload, whereas a fixed mesh network would need to do an expensive remap every time it happened (in a Plaxton based network I could easily see this leading to an evil circle). > Grapevine is based very heavily upon our experiences with Freenet. I feel > that we've taken the good ideas from Freenet and that we've addressed some of > it's deficiencies. We even discussed leveraging the existing Freenet codebase > (or perhaps the C++ port) when we first talked about doing the project. I > would like to thank everyone who worked on Freenet for the inspiration it has > given our project. I'm not trying to put down grapevine, I don't agree with your technical choices, but your goals are as laudable as any, so I would not particularly mind being wrong. <> -- Oskar Sandberg oskar@freenetproject.org From stephen@blacksapphire.com Thu May 23 17:33:44 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Fri, 24 May 2002 04:33:44 +1200 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020523165037.603B43310D@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oskar, On Fri, 24 May 2002 04:25, you wrote: > It would be very easy to have a network implementation fall over even if > the routing algorithm were deterministic and (in controlled conditions) > 100% successful. In fact, the situation may in many ways be worse for > such a network, since freenet can route around moderate numbers of nodes > that come in and out of eligibility due to overload, whereas a fixed > mesh network would need to do an expensive remap every time it happened > (in a Plaxton based network I could easily see this leading to an evil > circle). Agreed, but Grapevine doesn't work like that. To take one example, if a node disappears from the network, then it is replaced by another node within a short time. A new 118th nearest node comes into its "field of view" and can be picked up with the periodic "advertisement" that takes place. There is no "remap" process. I don't have a full understanding of Plaxton or CAN, but I am pretty sure Grapevine is totally unlike those two in this regard. Steve -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE87Rno7I0ehz47OHERAtmxAKCpH198nF2dylc7Ex6tTJpCHB+DAgCgmtIY E/dmDyTTRm2FV9DEYqW3Pq4= =q1cN -----END PGP SIGNATURE----- From Zooko Thu May 23 17:58:58 2002 From: Zooko (Zooko) Date: Thu, 23 May 2002 09:58:58 -0700 Subject: Grapevine Technical Overview In-Reply-To: Message from Oskar Sandberg of "Thu, 23 May 2002 18:25:40 +0200." References: Message-ID: Oskar Sandberg wrote: > > The implementation is a lot more than just the routing algorithm. I > would argue that the routing is not being tested one way or the other > "in the real world" at the moment, because other parts of the > implementation have failed to give it a chance. I would love to hear more about how Freenet is currently failing. I wrote a document about various ways that Mojo Nation failed: http://zooko.com/IPTPS02.pdf (Note: this focusses on the networking aspect of Mojo Nation and completely ignores Mojo, marketing, economics, user interface, and other stuff. Also note that the non-scalability of Mojo Nation's basic routing algorithm never gets mentioned since it never caused a failure. ;-)) In January of this year I created a project named Mnet (mnet.sf.net). It is the open-source spawn of Mojo Nation. Work on Mnet slowed down (but did not stop) recently when the two most prolific contributors got busy in Real Life. The current state is that Mnet still has the same architectural flaws that are described in that paper (centralized (but redundant) introduction, publishing data onto highly unreliable nodes). Improvements over Mojo Nation that are currently implemented in Mnet include a whole new GUI, a zillion bugfixes and code cleanups, and, uh, that's all that I can think of at the moment. Future directions for Mnet in descending order of probability/priority/schedule include interoperating with other code and other networks (we already have part of the Tristero API implemented), fixing the aforementioned architectural flaws, using uber-sophisticated new algorithms like Chord/Kademlia, exporting a nice programmable API (similar or identical to Tristero), and uh, one zillion other nifty improvements. Regards, Zooko ------- Secure Distributed Systems Consulting -- http://zooko.com/ ------- From bram@bitconjurer.org Thu May 23 19:31:37 2002 From: bram@bitconjurer.org (Bram Cohen) Date: Thu, 23 May 2002 11:31:37 -0700 (PDT) Subject: Grapevine Technical Overview In-Reply-To: Message-ID: Anonymous wrote: > MojoNation never worked either. > I think what this person is referring to is the very low retrieval success rate which Mojo Nation had from the beginning. -Bram From melc@fashionvictims.com Mon May 27 19:11:48 2002 From: melc@fashionvictims.com (melc@fashionvictims.com) Date: Mon, 27 May 2002 20:11:48 +0200 (CEST) Subject: Grapevine Technical Overview In-Reply-To: Message-ID: On Fri, 10 May 2002, Stephen Blackheath wrote: > In order for someone running a node to know what the contents of a > file stored on their system is, they have to guess the filename of the > file it belongs to, and then look up the file map and see if that CHK > is in the file map. Even if they know the entire contents of the file > they suspect, this is not sufficient information, due to the CBC mode > initialization vector. Hence plausible deniability. Hi, I've been thinking about the design of anonymous/censorship-resistant file storage systems (like grapevine) and it seems to me that there is a problem with this kind of plausible deniability. Given a file name or identifier it is possible to find out what blocks the file consists of (it must be, otherwise you can't download all the blocks of the file). Also each block is (I assume) uniquely identified by its CHK. Thus, given the name (or some other identifier) of a forbidden file, it is possible to create a list of CHKs that make up that file. Given that such a list can be made, the "thought police bureau of censorship" simply have to actively discover forbidden files (e.g. using a grapevine search engine), determine the CHKs of the blocks that make up those files, compile and regularly publish a list of forbidden block CHKs and make it illegal to request, host or relay any such blocks. Once they've done that it is simply a question of finding out (e.g., by making grapevine requests for forbidden blocks, or through more physical "break down the door" techniques) who is requesting, hosting or relaying these blocks and cart them off to jail. Is there a way to foil this kind of attack? Ihor. From oskar@freenetproject.org Mon May 27 20:12:45 2002 From: oskar@freenetproject.org (Oskar Sandberg) Date: Mon, 27 May 2002 21:12:45 +0200 Subject: Grapevine Technical Overview In-Reply-To: References: Message-ID: <20020527191245.GM18057@sporty.spiceworld> On Mon, May 27, 2002 at 08:11:48PM +0200, melc@fashionvictims.com wrote: > I've been thinking about the design of anonymous/censorship-resistant file > storage systems (like grapevine) and it seems to me that there is a > problem with this kind of plausible deniability. Yes, you are right. Regimes could easily publish blacklists of key values nodes are not allowed to handle requests for - and implementing the functionality to censor data based on those lists would be left to (read: forced on) the node operator. However, they are really more likely to simply outlaw systems like this all together. The whole plausible deniability thing is a joke, it's like claiming you are innocent because you closed your eyes while firing a machine gun in a crowd (or in this case, more like claiming you are innocent of thought crime because you closed your eyes while speaking banned words...) <> > Is there a way to foil this kind of attack? Keep the very existence of the nodes private. That is easier said then done. -- Oskar Sandberg oskar@freenetproject.org From adam@cypherspace.org Mon May 27 20:17:21 2002 From: adam@cypherspace.org (Adam Back) Date: Mon, 27 May 2002 20:17:21 +0100 Subject: how to do censorship resistance (Re: Grapevine Technical Overview) In-Reply-To: ; from oskar@freenetproject.org on Mon, May 27, 2002 at 09:12:45PM +0200 References: Message-ID: <20020527201721.A367145@exeter.ac.uk> Graydon Hoare's symptomatic file system proposes a solution to the censorship problem: a node is used only once, and is not advertised until a replacement node is in place. In this way, once you've found the content it has moved, so you never learn information allowing you to censor information. The downside is the overhead of the churn this will introduce. Adam On Mon, May 27, 2002 at 09:12:45PM +0200, Oskar Sandberg wrote: > On Mon, May 27, 2002 at 08:11:48PM +0200, melc@fashionvictims.com wrote: > > I've been thinking about the design of anonymous/censorship-resistant file > > storage systems (like grapevine) and it seems to me that there is a > > problem with this kind of plausible deniability. > > Yes, you are right. Regimes could easily publish blacklists of key > values nodes are not allowed to handle requests for - and implementing > the functionality to censor data based on those lists would be left to > (read: forced on) the node operator. > > However, they are really more likely to simply outlaw systems like this > all together. The whole plausible deniability thing is a joke, it's like > claiming you are innocent because you closed your eyes while firing a > machine gun in a crowd (or in this case, more like claiming you are > innocent of thought crime because you closed your eyes while speaking > banned words...) > > <> > > Is there a way to foil this kind of attack? > > Keep the very existence of the nodes private. That is easier said then > done. From lucas@gonze.com Mon May 27 20:13:23 2002 From: lucas@gonze.com (lucas@gonze.com) Date: Mon, 27 May 2002 15:13:23 -0400 (EDT) Subject: Grapevine Technical Overview In-Reply-To: Message-ID: > Once they've done that > it is simply a question of finding out (e.g., by making grapevine requests > for forbidden blocks, or through more physical "break down the door" > techniques) who is requesting, hosting or relaying these blocks and cart > them off to jail. > > Is there a way to foil this kind of attack? > > Ihor. A way to foil this attack, just not in the context of distributed hash tables: Spam out samizdat materials, so that people who don't want them get them, but people who do want them also get them. A way to do it in the context of distributed hash tables: Use a hash algorithm where can you predict some container nodes but not all. For example, publish an item under different names. Then a consumer can find one copy given the name, but the attacker can only take down one node at a time. From stephen@blacksapphire.com Mon May 27 22:30:29 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Tue, 28 May 2002 09:30:29 +1200 Subject: Threats to anonmyzing peer-to-peer networks In-Reply-To: References: Message-ID: <20020527213114.64D4233087@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ihor, > Is there a way to foil this kind of attack? As you pointed out, it's easy to find the retrieval keys for the segments of the "unwelcome" file. To elaborate on what Oskar said, the theory behind Freenet and Grapevine is that it is easy to retrieve a file if you have the key, but it is difficult to map this to a useful IP address. If you do map it to an IP address, it is very unlikely to be the true source of the file. The idea is that while you might be able to establish suspicion, establishing proof is almost impossible. It works similarly for requests. An outside agency will find it difficult to determine what you are requesting (unless they install some kind of "spyware" on your machine such as Windows XP). Or they could just put you in jail without the requirement for proof, as they have done in various tin-pot regimes such as Britain (I am referring to laws regarding disclosure of cryptography keys) and the United States (DMCA, military tribunals, turban-wearing etc...). As Oskar said, it is more likely that they will outlaw these systems altogether. I have been finding out about China recently. If you send an email to China containing the word 'freenet', then you are blocked (by email address, not IP address) from sending any mail to China for a period of about 24 hours. I have also seen this error (I have blanked out the email address): 3.ÄúµÄÐżþ±»¾ÜÊÕÁË,Çë²»Òª*¢ËͲ»ÊÜ»¶Ó­µÄÐżþ. <___@____.com>: host ***.____.com said: 552 Error: content rejected <___@____.com>: host ***.____.com said: This Message is not welcome 4.ÄúµÄÐżþÌ«´óÁË£¬±»¶Ô*½¾ÜÊÕÍË»ØÁË¡£ said: 552 Message size exceeds fixed limit I am also told that my Grapevine Project website is not visible from China, which could be interpreted as a mark of respect. Will they outlaw these systems in the United States? They haven't yet, to my knowledge, but "copyright circumvention devices" (and more seriously the development of them) are banned, and that is only a few pen-strokes away. I can think of two longer-term solutions to this problem: steganography and radio. There is another threat to anonymizing peer-to-peer networks: Internet Service Providers may prevent end-to-end communication. That is, they may forbid all incoming connections apart from ones associated with approved protocols such as ftp. All communication would have to go through some central point, and the Internet would start to resemble television (which, as we know, is an essential component of democracy, guaranteeing everyone the right to live in a cardboard box and eat dirt regardless of gender, race, nationality, sexual-orientation, political views, etc.) One possible solution to this would be to use steganographic email as a transport medium, just like in spy novels. ("Hi, Mum! How is the weather over there? It's 22.47 degrees here, and the tide is full at 7:34 AM. Please find attached a picture of my cat Xyproneroma7E/+giupuss who is orange, brown, and white with purple eyes...") Steve On Tue, 28 May 2002 06:11, melc@fashionvictims.com wrote: > On Fri, 10 May 2002, Stephen Blackheath wrote: > > In order for someone running a node to know what the contents of a > > file stored on their system is, they have to guess the filename of the > > file it belongs to, and then look up the file map and see if that CHK > > is in the file map. Even if they know the entire contents of the file > > they suspect, this is not sufficient information, due to the CBC mode > > initialization vector. Hence plausible deniability. > > Hi, > > I've been thinking about the design of anonymous/censorship-resistant file > storage systems (like grapevine) and it seems to me that there is a > problem with this kind of plausible deniability. > > Given a file name or identifier it is possible to find out what blocks the > file consists of (it must be, otherwise you can't download all the blocks > of the file). Also each block is (I assume) uniquely identified by its > CHK. Thus, given the name (or some other identifier) of a forbidden file, > it is possible to create a list of CHKs that make up that file. Given that > such a list can be made, the "thought police bureau of censorship" simply > have to actively discover forbidden files (e.g. using a grapevine search > engine), determine the CHKs of the blocks that make up those files, > compile and regularly publish a list of forbidden block CHKs and make it > illegal to request, host or relay any such blocks. Once they've done that > it is simply a question of finding out (e.g., by making grapevine requests > for forbidden blocks, or through more physical "break down the door" > techniques) who is requesting, hosting or relaying these blocks and cart > them off to jail. > > Is there a way to foil this kind of attack? > > Ihor. > > > > > > > > --- > You are currently subscribed to bluesky as: stephen@blacksapphire.com > For list information visit http://www.transarc.ibm.com/~ota/bluesky/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE88qV87I0ehz47OHERAmgwAKCv7IhqiIPfnhguOiLE3YLWMLrKHwCfRFSi OK+Frl+dSGtV58Nug37k5Dc= =1auD -----END PGP SIGNATURE----- From stephen@blacksapphire.com Mon May 27 22:53:48 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Tue, 28 May 2002 09:53:48 +1200 Subject: how to do censorship resistance (Re: Grapevine Technical Overview) In-Reply-To: References: Message-ID: <20020527215424.3F1E433087@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Adam, Freenet sort of works this way. Grapevine has more of a "fortress" mentality, the idea being that this is a "cheaper" way of preserving anonymity (i.e. without cost to efficiency). It does mean that Grapevine could suffer from "long-term colonisation" (or "siege"). This does not invalidate the idea - it just changes the design parameters (we have efficiency to trade off while others don't). Of course the point of this exercise is to try different approaches and see how they work. It's too early for every project to choose the "best" strategy. Steve On Tue, 28 May 2002 07:17, Adam Back wrote: > Graydon Hoare's symptomatic file system proposes a solution to the > censorship problem: a node is used only once, and is not advertised > until a replacement node is in place. > > In this way, once you've found the content it has moved, so you never > learn information allowing you to censor information. > > The downside is the overhead of the churn this will introduce. > > Adam > > On Mon, May 27, 2002 at 09:12:45PM +0200, Oskar Sandberg wrote: > > On Mon, May 27, 2002 at 08:11:48PM +0200, melc@fashionvictims.com wrote: > > > I've been thinking about the design of anonymous/censorship-resistant > > > file storage systems (like grapevine) and it seems to me that there is > > > a problem with this kind of plausible deniability. > > > > Yes, you are right. Regimes could easily publish blacklists of key > > values nodes are not allowed to handle requests for - and implementing > > the functionality to censor data based on those lists would be left to > > (read: forced on) the node operator. > > > > However, they are really more likely to simply outlaw systems like this > > all together. The whole plausible deniability thing is a joke, it's like > > claiming you are innocent because you closed your eyes while firing a > > machine gun in a crowd (or in this case, more like claiming you are > > innocent of thought crime because you closed your eyes while speaking > > banned words...) > > > > <> > > > > > Is there a way to foil this kind of attack? > > > > Keep the very existence of the nodes private. That is easier said then > > done. > > --- > You are currently subscribed to bluesky as: stephen@blacksapphire.com > For list information visit http://www.transarc.ibm.com/~ota/bluesky/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE88qrs7I0ehz47OHERAmgJAJ0RaqNRetTekp5BzlCFIAS6JRIsqgCdFGPf nhfmGGBorvBSuVghdLKF338= =u3e2 -----END PGP SIGNATURE----- From bram@bitconjurer.org Tue May 28 01:37:44 2002 From: bram@bitconjurer.org (Bram Cohen) Date: Mon, 27 May 2002 17:37:44 -0700 (PDT) Subject: how to do censorship resistance (Re: Grapevine Technical Overview) In-Reply-To: Message-ID: I should probably mention a technical point about Grapevine's routing - it involves distance in 6-space, with each node getting a random point. This will tend to make some nodes get a lot more traffic than others just because they're in the center of the space, while others are on the edge. You can fix this problem by making the space toroidal, which means the top and bottom are close to each other. I sent in a patch to do this for freenet, even though it's just one-dimensional. I don't know if that patch was ever applied though. -Bram From stephen@blacksapphire.com Tue May 28 02:38:57 2002 From: stephen@blacksapphire.com (Stephen Blackheath) Date: Tue, 28 May 2002 13:38:57 +1200 Subject: how to do censorship resistance (Re: Grapevine Technical Overview) In-Reply-To: References: Message-ID: <20020528013935.564C333087@mail.blacksapphire.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Bram, We've thought of that one. :) Another good thing about toroidal space is that you never have to traverse more than half of space, because if you did, you'd be better off going in the opposite direction. Steve On Tue, 28 May 2002 12:37, Bram Cohen wrote: > I should probably mention a technical point about Grapevine's routing - it > involves distance in 6-space, with each node getting a random point. This > will tend to make some nodes get a lot more traffic than others just > because they're in the center of the space, while others are on the edge. > > You can fix this problem by making the space toroidal, which means the top > and bottom are close to each other. I sent in a patch to do this for > freenet, even though it's just one-dimensional. I don't know if that patch > was ever applied though. > > -Bram -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE88t+x7I0ehz47OHERAvS8AJ4udrExuocY22m//YRX2wLKdVHrgACg13nt pEs/b0x9gvJjMkLfmmngRR4= =nG8Q -----END PGP SIGNATURE----- From bram@bitconjurer.org Tue May 28 03:42:53 2002 From: bram@bitconjurer.org (Bram Cohen) Date: Mon, 27 May 2002 19:42:53 -0700 (PDT) Subject: how to do censorship resistance (Re: Grapevine Technical Overview) In-Reply-To: Message-ID: Stephen Blackheath wrote: > Another good thing about toroidal space is that you never have to > traverse more than half of space, because if you did, you'd be better > off going in the opposite direction. That's cute, but a bit of a non-issue I think, since a factor of 2 doesn't change the asymptotic any. It occurs to me that if you generalize from 6-space to n-space, and simplify so that each peer is at position 0 or 1 instead of a floating point, then you wind up with something very similar to kademlia's xor metric. Each peer has an id which is 160 bits. The distance between peer A and B is the number of 1 bits in the xor of A and B's ids. You can read about kademlia at http://kademlia.scs.cs.nyu.edu/ It's metric differs from the one I just gave in that it's a tree instead of a mesh. You can probably do a good job of pruning peers by having each peer find out about all of it's peers's peers (by querying it's immediate peers) and having a policy that if it's peer A and it has peers B and C and the distance between A and C is greater than both the distance between A and B and B and C (each, not their sum) then it forgets about C. Note that kademlia does this for robustness, rather than anonymity purposes, so the way it uses peers is very different from grapevine. -Bram From aas23@cam.ac.uk Wed May 29 00:16:35 2002 From: aas23@cam.ac.uk (Andrei Serjantov) Date: Tue, 28 May 2002 19:16:35 -0400 Subject: Grapevine Technical Overview Hi, > I've been thinking about the design of anonymous/censorship-resistant file > storage systems (like grapevine) and it seems to me that there is a > problem with this kind of plausible deniability. I am slightly surprised the FreeHaven has not yet been mentioned in this discussion -- it is a good design which tackles some of the problems you have been talking above here. Perhaps everyone is too familiar with it already. However, it moves the shares (parts of the files) around which makes them inefficient to locate. I also recently wrote a paper entitled "Anonymixing Censorship Resistant Systems" which talks about these problems. It is a design of a (surprise!) anonymous censorship resistant system which, again, tackles some of the anonymity and censorship resistance problems you have been discussing here. It uses two techniques: storing pointers to (parts of) files on different machines from the files themselves, stores encrypted (parts of) the files and uses an anonymity system to hide correspondence between the pointers and the files. For those who do not want to read the whole paper the diagrams should give a basic overview of what the solution is. Here is the URL: http://www.cl.cam.ac.uk/~aas23/Anon_p2p2.ps A word of warning: I do not tackle acountability searching and other important problems, but if you want anonymity *and* censorship resistance, I do not know how to do better yet! Thanks, Andrei From TedAnderson@mindspring.com Wed May 29 02:28:10 2002 From: TedAnderson@mindspring.com (Ted Anderson) Date: Tue, 28 May 2002 21:28:10 -0400 Subject: how to do censorship resistance (Re: Grapevine Technical Overview) References: Message-ID: <3CF42EAA.1060207@mindspring.com> Bram Cohen wrote: > That's cute, but a bit of a non-issue I think, since a factor of 2 doesn't > change the asymptotic any. > > It occurs to me that if you generalize from 6-space to n-space, and > simplify so that each peer is at position 0 or 1 instead of a floating > point, then you wind up with something very similar to kademlia's xor > metric. > > Each peer has an id which is 160 bits. The distance between peer A and B > is the number of 1 bits in the xor of A and B's ids. > > You can read about kademlia at http://kademlia.scs.cs.nyu.edu/ It's metric > differs from the one I just gave in that it's a tree instead of a mesh. This Kademlia sounds very much like the boolean n-cube routing scheme I've been touting for a while now. I sent a description of it to one of the Freenet lists around 10/1999, at the paragraph starting "The core of my solution..." in [1]. Kademlia sounds like good stuff and David Mazières has been involved in several interesting projects. Ted Anderson (formerly, ota@transarc.com) [1] http://www.transarc.ibm.com/~ota/datadist-19991026.txt From bram@bitconjurer.org Wed May 29 05:58:19 2002 From: bram@bitconjurer.org (Bram Cohen) Date: Tue, 28 May 2002 21:58:19 -0700 (PDT) Subject: how to do censorship resistance (Re: Grapevine Technical Overview) In-Reply-To: Message-ID: Ted Anderson wrote: > This Kademlia sounds very much like the boolean n-cube routing scheme=20 > I've been touting for a while now. I sent a description of it to one of= =20 > the Freenet lists around 10/1999, at the paragraph starting "The core of= =20 > my solution..." in [1]. Kademlia sounds like good stuff and David=20 > Mazi=E8res has been involved in several interesting projects. Yep, that's the exact same distance function as kademlia uses. That distincae function, by the way, has been reinvented by several people. I find it odd that viewing the xor as a number is intuitive to everyone else while adding up the number of 1s in it is intuitive to me. Maybe everyone else is optimizing for analyzability while I'm optimizing for simplicity and robustness, trying to get away from a heirarchical structure. -Bram From Zooko Wed May 29 13:59:32 2002 From: Zooko (Zooko) Date: Wed, 29 May 2002 05:59:32 -0700 Subject: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview)) In-Reply-To: Message from Bram Cohen of "Tue, 28 May 2002 21:58:19 PDT." References: Message-ID: Bram wrote: > > Ted Anderson wrote: > > > This Kademlia sounds very much like the boolean n-cube routing scheme > > I've been touting for a while now. I sent a description of it to one of > > the Freenet lists around 10/1999, at the paragraph starting "The core of > > my solution..." in [1]. Kademlia sounds like good stuff and David > > Mazi=E8res has been involved in several interesting projects. > > Yep, that's the exact same distance function as kademlia uses. That > distincae function, by the way, has been reinvented by several people. Actually my reading of Ted's letter [1] says that his proposal was, like Bram's, an n-dimensional hypercube (n=160). Did I get that right, Ted, Bram? The Kademlia metric as I understand it from this paper [2] yields a very different structure -- a binary tree. The difference in the way the metric is computed is that Ted's says "I want to know 160 peers, each of which has the same ID as me except for its i'th bit being different.", and Kademlia's says "I want to know up to 160 peers, each of which has the same i-1 most significant bits and a different i'th bit.". I've heard of hypercube routing notion from many people over the years, but the Kademlia notion was novel to me when I saw their paper at the Peer to Peer Workshop. > I find it odd that viewing the xor as a number is intuitive to everyone > else while adding up the number of 1s in it is intuitive to me. Maybe > everyone else is optimizing for analyzability while I'm optimizing for > simplicity and robustness, trying to get away from a heirarchical > structure. Well, the Kademlia structure ("as a number") is much more robust than the hypercube structure ("adding up the number of 1s" == Hamming distance), because it imposes a relatively loose constraint on which nodes a given node can peer with. In the hypercube structure, there is exactly one ID out of all possible IDs that could be your i'th peer. In the binary tree structure, any id which matches your most-significant i-1 bits is a legitimate i'th peer. For example, if there are 2^20 total nodes in a hypercube network, chances are that none of them will satisfy the strict requirement to be your 4th peer (your peer for i=4). Therefore you have to use an approximate match (as Ted's proposal allows) that is "close" (by the same measure). This leaves me wondering if the overall hypercube-like structure is still going to have efficient routing properties when using approximate matches like that, and at the same time whether the approximate match shadowing a more precise match could cause problems. In a binary tree (Kademlia) network of size 2^20, there are 2^15 nodes which are all equally legitimate to be your 4th peer. The claims about routing efficiency and correctness are all justified based on the assumption that you can communicate with *any* one of these 2^15 nodes. Now that's robust! Of course, once you get down to i=12 then this is no longer so easy -- now there are only 2^7 nodes in the entire network who would qualify. I think that this is still more robust than other proposals for two reasons: 1. There are 2^8 - 1 other nodes ("parallel" to you) who are equivalently qualified as you are to find the next hop. The query can try more than one of these, in parallel or in series. (In fact, it is free to pre-emptively choose among these 2^8 nodes on some other basis such as reliability or latency.) 2. The case that the query fails to find the appropriate next-hop will necessarily be handled in the same way as the case that said node has exited the network. That is: one or more of the "current hop" nodes will be responsible for the ID space covered by the absent next-hop nodes. This is an approximation which can cause shadowing (like the use of close matches in the hypercube), but it is purely local -- no query will be effected by this appromixation unless its target ID is in this particular 2^-13 sliver of the ID space. (Purely local approximations which might cause shadowing are necessary in any possible network. Consider the case that there are only 2^12 nodes in the network. Now you are unable to find a node to serve as your i=12 peer, but that's because you are the one responsible for the associated ID space.) Hm. This suggests a twist to Kademlia: If a node doesn't have the data you are looking for, and it also doesn't know any other nodes that are closer to the ID, then call it a "failed leaf node" with respect to this query. When a node responds to a previous-hop node, telling the previous-hop node that the object it sought couldn't be found, it includes a count of the number of failed leaf nodes who have been queried (counting itself, so always at least 1). As long as the count is less than a constant K (which, I guess, should be something like 8), then the previous- node continues the search, by asking other next-nodes. If it has tried all of the appropriate next-nodes that it knows, then it sends a failure message to *its* previous node with the failed-leaf-node count. This is a little mini-flood which cannot impose much bandwidth cost (but can impose much latency!) and greatly reduces the risk of a false negative due to a broken route. You can of course do a similar trick on publication of data to ensure replication onto the K closest reachable nodes. Regards, Zooko ------- Secure Distributed Systems Consulting -- http://zooko.com/ ------- [1] http://www.transarc.ibm.com/~ota/datadist-19991026.txt [2] http://kademlia.scs.cs.nyu.edu/kpos.pdf From Zooko Thu May 30 11:51:03 2002 From: Zooko (Zooko) Date: Thu, 30 May 2002 03:51:03 -0700 Subject: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview)) In-Reply-To: Message from Zooko of "Wed, 29 May 2002 05:59:32 PDT." References: Message-ID: Last night my wife and I drew sketches of Kademlia space. One neat thing that came up is that Kademlia space is almost a hypercube -- it's a hypercuboid! Imagine a hypercube, but without all of its edges being the same length. Instead, each edge which extends into the Nth dimension has length 2^N. That's Kademlia's routing space! It shares with hypercubes the property that you have to traverse at most log2(N) edges to move from any vertex to any other. It is different from a hypercube in that for any vertex A, each other vertex is at a unique distance from A. (In a hypercube, most other vertexes are the same distance from A: log2(N)/2 edges away.) Regards, Zooko ------- Secure Distributed Systems Consulting -- http://zooko.com/ ------- From oskar@freenetproject.org Thu May 30 12:14:38 2002 From: oskar@freenetproject.org (Oskar Sandberg) Date: Thu, 30 May 2002 13:14:38 +0200 Subject: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview)) In-Reply-To: References: Message-ID: <20020530111438.GA2011@sporty.spiceworld> On Wed, May 29, 2002 at 05:59:32AM -0700, Zooko wrote: <> > Actually my reading of Ted's letter [1] says that his proposal was, like > Bram's, an n-dimensional hypercube (n=160). Did I get that right, Ted, Bram? > > The Kademlia metric as I understand it from this paper [2] yields a very > different structure -- a binary tree. > > The difference in the way the metric is computed is that Ted's says "I want to > know 160 peers, each of which has the same ID as me except for its i'th bit > being different.", and Kademlia's says "I want to know up to 160 peers, each > of which has the same i-1 most significant bits and a different i'th bit.". > > I've heard of hypercube routing notion from many people over the years, but > the Kademlia notion was novel to me when I saw their paper at the Peer to > Peer Workshop. The much cited system proposed by Plaxton [1] works this way, and it's derivatives used in systems like Oceanstore and PAST [2][3] as well. By only routing "forward" on the key (only matching the previous levels at each level, rather than all the others), you get more flexibility in choosing your neighbors (which the cited systems use to choose network topologically close neighbors) at no real cost. I was not aware that anyone was proposing a distributed system to route in a strict hypercube which are really more suited connecting processors and other very static nodes. Hypercube routing can be seen as a bunch of interlocked trees as well. They are just different ways of visualizing the topology. Unless Kademlia brings something else to the table, that sounds like old news to me. -- [1] C. Plaxton, R. Rajaraman, A Richa. Accessing Nearby copies of replicated objects in a distributed environment. In Proc. of ACM SPAA, June 1997. [2] J Kubiatowcz, D Bindel, Y Chen, S Czerwinski, P Eaton, D Geels, R Gummandi, S Rhea, H Weatherspoon, W Weirner, C Wells, B Zhao. Oceanstore: An architecture for Global Scale Persistant Storage. In Proc. ASPLOS'2000, November 2000. [3] A Rowstron, P Druschel. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. Microsoft Research, submission to ACM HOTOS VIII, 2001. -- Oskar Sandberg oskar@freenetproject.org From Zooko Thu May 30 13:43:40 2002 From: Zooko (Zooko) Date: Thu, 30 May 2002 05:43:40 -0700 Subject: Kademlia and Pastry (was: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview))) In-Reply-To: Message from Oskar Sandberg of "Thu, 30 May 2002 13:14:38 +0200." References: Message-ID: [I, Zooko, wrote the lines prepended with '> > '. And the ones without any prepended greater-than signs, too.] Oskar wrote: > > > The difference in the way the metric is computed is that Ted's says "I want to > > know 160 peers, each of which has the same ID as me except for its i'th bit > > being different.", and Kademlia's says "I want to know up to 160 peers, each > > of which has the same i-1 most significant bits and a different i'th bit.". > > > > I've heard of hypercube routing notion from many people over the years, but > > the Kademlia notion was novel to me when I saw their paper at the Peer to > > Peer Workshop. > > The much cited system proposed by Plaxton [1] works this way, and it's > derivatives used in systems like Oceanstore and PAST [2][3] as well. By > only routing "forward" on the key (only matching the previous levels at > each level, rather than all the others), you get more flexibility in > choosing your neighbors (which the cited systems use to choose network > topologically close neighbors) at no real cost. [... ellided by Zooko] [...] > Unless Kademlia brings something else to the table, that sounds like old > news to me. Hm. Yes, you are right that Kademlia's routing is really the same as that in e.g. Pastry. To their credit, the authors of Kademlia say as much in their paper. Although I had read the Pastry paper, I didn't understand the advantages of the routing system over e.g. Chord. (In fact, one of the authors of Pastry tried to convince me that Pastry had a qualitative technical advantage over Chord, due to its ability to incorporate network proximity. Unfortunately for him and for my education, I didn't care very much about network proximity. More on that below.) There are several reason why I didn't appreciate the XOR metric even after I read the Pastry paper, and that they are worth looking at. They show, for me at least, some added conceptual value that Kademlia has brought to the table. 1. The Pastry paper describes the technique as successive approximation by appending digits and popping about on a circle. The Kademlia paper describes it (the very same operation) as XOR and traversing a binary tree. I find the latter much more intuitive. 2. The Pastry paper includes a "shift gears" step in which you use a different routing system once you've gotten close enough using the XOR routing system. The Kademlia paper uses only the XOR routing system "all the way down". (By the way, now that I've understood the XOR routing system, I appreciate *why* the Pastry design includes a second technique for the last few hops. Kademlia needs to solve the same problem, and it currently attempts to do so by returning redundant query results...) 3. The Pastry paper never mentions these two interesting properties of XOR routing: that it is both symmetric and unidirectional. The Kademlia paper makes these two points explicitly, and I realized that the routing system offered these properties when reading the Kademlia paper. (By the way, symmetric isn't a big deal with regard to efficiency -- it's just a factor of 2 -- but it might be a bigger deal with regard to bypassing firewalls, which are usually asymmetric in blocking initiation of TCP connections...) 4. The Pastry paper *does* mention the third interesting property of this routing system: next-hop-selection (i.e., that there are a large number of valid next hops, and you may choose among them on some criteria other than routing), but the Pastry paper repeatedly emphasizes that this property is used for network proximity, which I care relatively little about. The Kademlia paper uses the same property for reliability (and, secondarily, for low latency), which I care a lot more about. (By the way, I am now thinking about using that property for the generalization of reliability: trustworthiness...) Regards, Zooko ------- Secure Distributed Systems Consulting -- http://zooko.com/ ------- From oskar@freenetproject.org Thu May 30 16:19:25 2002 From: oskar@freenetproject.org (Oskar Sandberg) Date: Thu, 30 May 2002 17:19:25 +0200 Subject: Kademlia and Pastry (was: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview))) In-Reply-To: References: Message-ID: <20020530151925.GD2011@sporty.spiceworld> On Thu, May 30, 2002 at 05:43:40AM -0700, Zooko wrote: > Oskar wrote: < > > > The much cited system proposed by Plaxton [1] works this way, and it's > > derivatives used in systems like Oceanstore and PAST [2][3] as well. By > > only routing "forward" on the key (only matching the previous levels at > > each level, rather than all the others), you get more flexibility in > > choosing your neighbors (which the cited systems use to choose network > > topologically close neighbors) at no real cost. [... ellided by Zooko] > [...] > > Unless Kademlia brings something else to the table, that sounds like old > > news to me. > > > Hm. Yes, you are right that Kademlia's routing is really the same as that in > e.g. Pastry. To their credit, the authors of Kademlia say as much in their > paper. > > Although I had read the Pastry paper, I didn't understand the advantages of > the routing system over e.g. Chord. (In fact, one of the authors of Pastry > tried to convince me that Pastry had a qualitative technical advantage over > Chord, due to its ability to incorporate network proximity. Unfortunately for > him and for my education, I didn't care very much about network proximity. > More on that below.) I don't consider Pastry to be the best or most elegant of the various proposed systems derived from the Plaxton paper. The system Oceanstore uses (which they call Tapestry [1]) is probably nicer. After having worked on implementing it though, I am not fond of any of these systems. It proved very difficult and costly to make these networks stay consistant under the high node churn scenarios I used for my Freenet simulations. It may have been my fault, but my current hypothesis is that these things may work in a controlled environment such a corporate network, but will be difficult to make a work in the wild where nodes are fickle and unreliable. <> > 1. The Pastry paper describes the technique as successive approximation by > appending digits and popping about on a circle. The Kademlia paper describes > it (the very same operation) as XOR and traversing a binary tree. I find the > latter much more intuitive. That is a matter of your personal taste. I find approaching the problem from a hypercube more intuitive. However, [2] says: "The key observation to make from Figure 3 is that the links form a series of random embedded trees, with each node as the root of one of these trees. As a result, the neighbor links can be used to route from anywhere to a given node, simply by resolving the node's address one link at a time--first a level-one link, then a level-two link, etc." So this not a new way of looking at it either. > 3. The Pastry paper never mentions these two interesting properties of XOR > routing: that it is both symmetric and unidirectional. The Kademlia paper > makes these two points explicitly, and I realized that the routing system > offered these properties when reading the Kademlia paper. I'm not sure what you mean by either. But again, Pastry is not the only paper to have covered this ground previously. > 4. The Pastry paper *does* mention the third interesting property of this > routing system: next-hop-selection (i.e., that there are a large number of > valid next hops, and you may choose among them on some criteria other than > routing), but the Pastry paper repeatedly emphasizes that this property is > used for network proximity, which I care relatively little about. The > Kademlia paper uses the same property for reliability (and, secondarily, for > low latency), which I care a lot more about. The measure for "network promixity" as used by Plaxton [3] was not defined, it can be any metric on the network, including latency. The Oceanstore papers deal with extensively with reliability. -- [1] B Zhao, J Kubiatowcz, A Joseph. Tapestry: An Infrastructure for Fault-tolerant Wide-area Location and Routing. EECS Report No. UCB/CSD-01-1141, Univeristy of California, Berkley, April 2001. [2] J Kubiatowcz, D Bindel, Y Chen, S Czerwinski, P Eaton, D Geels, R Gummandi, S Rhea, H Weatherspoon, W Weirner, C Wells, B Zhao. Oceanstore: An architecture for Global Scale Persistant Storage. In Proc. ASPLOS'2000, November 2000. [3] C. Plaxton, R. Rajaraman, A Richa. Accessing Nearby copies of replicated objects in a distributed environment. In Proc. of ACM SPAA, June 1997. -- Oskar Sandberg oskar@freenetproject.org From Zooko Thu May 30 17:08:39 2002 From: Zooko (Zooko) Date: Thu, 30 May 2002 09:08:39 -0700 Subject: Kademlia and Pastry (was: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview))) In-Reply-To: Message from Oskar Sandberg of "Thu, 30 May 2002 17:19:25 +0200." References: Message-ID: [I, Zooko, wrote the lines prepended with '> > '. And the ones without any prepended greater-than signs, too.] Oskar wrote: > > After having > worked on implementing it though, I am not fond of any of these systems. > It proved very difficult and costly to make these networks stay > consistant under the high node churn scenarios I used for my Freenet > simulations. [... ellided by Zooko] Yep. This is exactly why using the "free-choice" feature for proximity routing is sort of mildly interesting, but using it for reliability is very interesting. My paper at IPTPS02 [1] detailed the kind of churn observed in practice in Mojo Nation. I would love to get some similar stats from you about Freenet. > > 1. The Pastry paper describes the technique as successive approximation by > > appending digits and popping about on a circle. The Kademlia paper describes > > it (the very same operation) as XOR and traversing a binary tree. I find the > > latter much more intuitive. > > That is a matter of your personal taste. I find approaching the problem > from a hypercube more intuitive. [... ellided by Zooko] Indeed. My personal taste is likely shared with many hackers who like me are more familiar with XOR and binary trees than with hyperdimensional cuboids. > > 3. The Pastry paper never mentions these two interesting properties of XOR > > routing: that it is both symmetric and unidirectional. The Kademlia paper > > makes these two points explicitly, and I realized that the routing system > > offered these properties when reading the Kademlia paper. > > I'm not sure what you mean by either. [... ellided by Zooko] Perhaps you ought to read the Kademlia paper. > The measure for "network promixity" as used by Plaxton [3] was not > defined, it can be any metric on the network, including latency. The > Oceanstore papers deal with extensively with reliability. Defining "network proximity" as latency is natural enough, but defining it as reliability seems quite a stretch. ;-) Indeed the Oceanstore papers have dealt extensively with reliability, but they have not suggested the option of using reliability as the selection criterion among valid next-hop nodes. (Instead, like Pastry, they propose that network proximity gets the benefit of that role, and they layer reliability at higher layers.) Regards, Zooko ------- Secure Distributed Systems Consulting -- http://zooko.com/ ------- [1] http://zooko.com/IPTPS02.pdf From TedAnderson@mindspring.com Fri May 31 03:16:24 2002 From: TedAnderson@mindspring.com (Ted Anderson) Date: Thu, 30 May 2002 22:16:24 -0400 Subject: Kademlia Kademlia Kademlia (was: how to do censorship resistance (was: Grapevine Technical Overview)) References: Message-ID: <3CF6DCF8.1030405@mindspring.com> Zooko wrote: > Actually my reading of Ted's letter [1] says that his proposal was, > like Bram's, an n-dimensional hypercube (n=160). Did I get that > right, Ted, Bram? My prose was not too clear. I don't really define "close" on these 160-bit ids, but I had in mind close when treated as a binary integer. The result, I think, is that successive elements of the routing array should match the local node id in one additional bit position. The bits below the complemented one are essentially arbitrary. In fact, it is better to use some other metric than closeness below this complemented bit, such as reliability or latency. > The Kademlia metric as I understand it from this paper [2] yields a > very different structure -- a binary tree. I think the key point here, and in some of the subsequent discussion, is that the hypercube is *vastly* sparse. Linking the extant nodes necessarily requires ignoring most of the bits of the node id. Using an array-like structure, where larger indexes match more bits, essentially maps the existing nodes into a tree. I must say the interlocked trees interpretation of Plaxton tooks me a while to figure out. I have always thought of it as a hypecube, even though its massive sparseness, makes the tree model more physical. I guess the large variety of interpretations of this basic idea makes its acceptance more difficult. > The difference in the way the metric is computed is that Ted's says "I > want to know 160 peers, each of which has the same ID as me except for > its i'th bit being different.", and Kademlia's says "I want to know up > to 160 peers, each of which has the same i-1 most significant bits and > a different i'th bit.". Given the binary integer definition of closeness, I think the two approaches are equivalent. > Well, the Kademlia structure ("as a number") is much more robust than > the hypercube structure ("adding up the number of 1s" == Hamming > distance), because it imposes a relatively loose constraint on which > nodes a given node can peer with. In the hypercube structure, there > is exactly one ID out of all possible IDs that could be your i'th > peer. In the binary tree structure, any id which matches your > most-significant i-1 bits is a legitimate i'th peer. The hamming distance, counting ones in the XOR, gives the distance in a fully populated hypercube. But since it is sparse, a hierarchical approach should be robust. As Oskar and others point out, the fact that (esp. at the top of the tree) there are many routing choices, should allow for efficient, reliable routing as long as the choice of which peers to select for each bucket in the routing table matches reality. In the case of many transient nodes, it may be that reliability, age or longevity are better metrics than network proximity. > (Purely local approximations which might cause shadowing are necessary > in any possible network. Consider the case that there are only 2^12 > nodes in the network. Now you are unable to find a node to serve as > your i=12 peer, but that's because you are the one responsible for the > associated ID space.) Indeed. The need for backtracking to find data should be limited assuming nodes collect matching chunks aggressively enough and there is enough overlap (shadowing) to handle nodes dropping out of the network. Ted