Developer discussion for Creative Commons technology and tools

["Victor Stone" <fourstones.net AT gmail.com>]

On Wed, Jul 2, 2008 at 2:10 PM, Nathan Yergler
<nathan AT creativecommons.org> wrote:
> Were having a few issues with aggressive crawlers slowing the machine
> down.

gee, I wonder what that's like ;)

I finally fixed all that will a little snippet of code at the bottom
of every page on ccM with a hidden link marked as 'nofollow' and
explicitly listed in robots.txt (I use javascript to replace the href
with nothing just in case) The link is unique with a rand() parameter
so the bots don't 'learn' to avoid it.

If you follow the link, it leads to a php script that enters your IP
into the deny list in .htaccess. This would stop crawls right as they
started but allows well-behaved crawls from google, yahoo, etc. to
continue. We used to go down once a week, now, pretty much never. We
collect about 50-100 IPs per week. I clean them out regularly because
the evil bots burn the IPs anyway.

VS