ibiblio-announce AT lists.ibiblio.org
Subject: ibiblio services/changes announcement list
List archive
- From: John Reuning <john AT metalab.unc.edu>
- To: ibiblio-announce AT lists.ibiblio.org
- Subject: [ibiblio-announce] robots.txt for dev/test sites
- Date: Mon, 21 Feb 2005 19:12:07 -0500
To prevent unwanted crawling and indexing of dev and *.test.ibiblio.org
web sites, we have added an alias to robots.txt. This alias references
a common file and enables /robots.txt on all dev & test sites.
If you check this file for your virtual host test or custom dev site,
you should see
User-agent: *
Disallow: /
Please note that this is only for dev/test sites. It does not affect
any production hosted domain web site or any http://www.ibiblio.org URL.
If you have any questions, please email help AT ibiblio.org.
Thank you,
-jrr
- [ibiblio-announce] robots.txt for dev/test sites, John Reuning, 02/21/2005
Archive powered by MHonArc 2.6.24.