Skip to Content.
Sympa Menu

bluesky - Re: market-based data location (was Scalability)

bluesky AT lists.ibiblio.org

Subject: Global-Scale Distributed Storage Systems

List archive

Chronological Thread  
  • From: Ted Anderson <ota AT transarc.com>
  • To: bluesky AT franklin.oit.unc.edu
  • Subject: Re: market-based data location (was Scalability)
  • Date: Tue, 6 Mar 2001 06:52:17 -0500 (EST)


On 27 Feb 2001 00:12:55 PST <hal AT finney.org> wrote:
> But I was struck by the statement in the introduction to one of their
> papers that their goal is to support a network with 10^10 users (10
> billion users, i.e. the total population of Earth), each storing 10000
> files, for a total of 100 trillion files. That's a lot of files!
> Scaling to that level will be a real challenge.

This figure struck me as well, because it agreed with my thinking about
the appropriate scale of the decentralized file system I envision. This
was crystallized for me when I tried, unsuccessfully, to convince myself
that a 128-bit hash value was sufficiently large for object CHKs. When
I considered how many nodes might participate in the system and how many
objects each might create over a period of years, I could not do it. I
assumed 10^10 nodes (not users, but actively participating computers)
and over its lifetime it seemed plausible that a node might create 10^10
files (about 33/sec for 10 years). This comes to 10^20 (2^66.5) objects
and, since the birthday paradox tells us that collisions will be common
when we reach the square root of the hash size, 128 bits seemed just too
small for comfort. Using the 160 bit SHA-1 output provides a nice
margin of safety.

In a relatively benign environment, I could imagine a system like the
OceanStore data location scheme working at this scale. I have argued in
favor of a similar idea based on boolean hypercubes for several years.
However, I now think the Internet is hostile enough that a more
flexible, adaptable approach based on market incentives will be
necessary. At the beginning of this thread, Wei Dai, very briefly,
described such a system.

The economic model is the only one we know of that can organize
loosely-coupled human systems on this scale. If we generalize the
interpretation of "economic" to include cooperative, mutually beneficial
interactions of all kinds this organizational capability applies to
complex systems of essentially any type. Robert Wright, expounds on
this idea in his book titled "Nonzero: The logic of Human Destiny". He
argues that increasing complexity of all living systems are driven by
nonzero-sum interactions on many levels.

The specific shortcomings of MojoNation-style economics, and with
micro-payments in general, are worth understanding. The conclusion I
think we must draw, however, is not that we should avoid market-based
incentives, but that we need to find a way to incorporate these
mechanisms in a way that will enable mutually beneficial cooperation

between agents while suppressing parasitic interactions.

Ted Anderson




Archive powered by MHonArc 2.6.24.

Top of Page