Skip to Content.
Sympa Menu

ibiblio-announce - [ibiblio-announce] robots.txt for dev/test sites

ibiblio-announce AT lists.ibiblio.org

Subject: ibiblio services/changes announcement list

List archive

Chronological Thread  
  • From: John Reuning <john AT metalab.unc.edu>
  • To: ibiblio-announce AT lists.ibiblio.org
  • Subject: [ibiblio-announce] robots.txt for dev/test sites
  • Date: Mon, 21 Feb 2005 19:12:07 -0500

To prevent unwanted crawling and indexing of dev and *.test.ibiblio.org
web sites, we have added an alias to robots.txt. This alias references
a common file and enables /robots.txt on all dev & test sites.

If you check this file for your virtual host test or custom dev site,
you should see

User-agent: *
Disallow: /

Please note that this is only for dev/test sites. It does not affect
any production hosted domain web site or any http://www.ibiblio.org URL.
If you have any questions, please email help AT ibiblio.org.

Thank you,

-jrr




  • [ibiblio-announce] robots.txt for dev/test sites, John Reuning, 02/21/2005

Archive powered by MHonArc 2.6.24.

Top of Page