Scalability

hal@finney.org hal@finney.org
Mon, 26 Feb 2001 23:00:15 -0800


Adam writes:
> - a system to find the URLs you want
> - a system to locate a copy of and download the URL content
>
> Steve Schear and I were discussing the first one and another
> approach is to just let a search engine do it for you.  Either
> by exposing the URLs to the real web through gateways which
> make it appear (to search engines only perhaps) as static content.

Web spidering does work to some extent, but not really that well.
A substantial fraction of web pages aren't indexed.  It should be possible
to do better if the network were designed for it from the beginning.

> Locating a copy of a target URL to download is a different
> problem.  Graydon's log map seems like a reasaonable approach.

What is this?  Is it similar to the OceanStore method of mapping
a tree onto a network?  I'm not familiar with this data structure.

Hal