Skip to Content.
Sympa Menu

cc-devel - Re: [cc-devel] issues with code server?

cc-devel AT lists.ibiblio.org

Subject: Developer discussion for Creative Commons technology and tools

List archive

Chronological Thread  
  • From: "Victor Stone" <fourstones.net AT gmail.com>
  • To: "Nathan Kinkade" <nkinkade AT creativecommons.org>
  • Cc: creative commons developers <cc-devel AT lists.ibiblio.org>
  • Subject: Re: [cc-devel] issues with code server?
  • Date: Fri, 4 Jul 2008 00:46:14 -0700

we're down again...

VS

On Wed, Jul 2, 2008 at 3:35 PM, Nathan Kinkade
<nkinkade AT creativecommons.org> wrote:
> 2008/7/2 Victor Stone <fourstones.net AT gmail.com>:
>> On Wed, Jul 2, 2008 at 2:10 PM, Nathan Yergler
>> <nathan AT creativecommons.org> wrote:
>>> Were having a few issues with aggressive crawlers slowing the machine
>>> down.
>>
>> gee, I wonder what that's like ;)
>>
>> I finally fixed all that will a little snippet of code at the bottom
>> of every page on ccM with a hidden link marked as 'nofollow' and
>> explicitly listed in robots.txt (I use javascript to replace the href
>> with nothing just in case) The link is unique with a rand() parameter
>> so the bots don't 'learn' to avoid it.
>>
>> If you follow the link, it leads to a php script that enters your IP
>> into the deny list in .htaccess. This would stop crawls right as they
>> started but allows well-behaved crawls from google, yahoo, etc. to
>> continue. We used to go down once a week, now, pretty much never. We
>> collect about 50-100 IPs per week. I clean them out regularly because
>> the evil bots burn the IPs anyway.
>>
>> VS
>
> Very interesting. I was wondering how you had that implemented. I
> think the problem we had had to do with gitweb, which is sort of like
> ViewVC, but for git repositories. Apparently gitweb (or even git
> alone) is processor intensive, and so when the crawl started it just
> ate up all the processor. Asheesh installed a new version of gitweb
> that is written in C instead of some perl CGI script, and which
> caches, so hopefully that will help.
>
> Nathan
>




Archive powered by MHonArc 2.6.24.

Top of Page