[cc-community] question about liscensing and netscraping.

Evan Prodromou evan at prodromou.name
Sat Aug 4 20:38:49 EDT 2007


On Sat, 2007-04-08 at 21:24 +0200, Laura Creighton wrote:

> We worry that the scrapers will get higher google rankings than we do,
> on our content, when we think we deserve the top rating.

This is a common concern by Free Content people, and you'll be happy to
know that it really rarely happens. Especially if you set up a Google
sitemap, so that Google's spiders have an idea of the original locus of
the work.

The wiki I started, Wikitravel, is widely mirrored. So is Wikipedia. It
hasn't hurt our traffic much at all. Search engines are pretty darn good
at ignoring duplicate pages. Making mirrors of popular content and
putting automated ads on them just isn't that good a business.

And, really, does it matter that much? Isn't the whole point to get the
content out there? And, aren't you having traffic problems? Are mirrors
so bad?

Maybe the way to solve the problem is to make downloadable bundles of
your entire wiki available. I'm pretty sure you could get the bundles
hosted either on an archiving system like archive.org or on BitTorrent. 

There are other things you can do about Web scrapers: carefully craft
your robots.txt to block compute- or bandwidth-intensive URLs, and add a
Crawl-Delay stanza. (Crawl-Delay is a huge help, by the way, and most
"good" crawlers now honor it.)

And for spiders that ignore robots.txt, you can block them at the IP or
at the HTTP server level. I think returning a 403 error with the link to
your downloadable bundles would be appropriate.

> Is there a CC license that we could license our wiki under that would
> make it possible for us to threaten those who use our wiki in ways we
> do not want with a lawsuit?

Is that _really_ in the spirit of Free Software? Guido doesn't sue
people for using Python in ways he doesn't like. Part of giving things
away is that you don't control how they're used after that. We make Free
things so that people can use them. 

-Evan

-- 
Evan Prodromou - evan at prodromou.name - http://evan.prodromou.name/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 4422 bytes
Desc: not available
Url : http://lists.ibiblio.org/pipermail/cc-community/attachments/20070805/299999e3/attachment.bin 


More information about the cc-community mailing list