app/soc/content/robots.txt
author Lennard de Rijk <ljvderijk@gmail.com>
Sun, 25 Oct 2009 18:43:23 -0700
changeset 3050 5f135cfac194
parent 73 211a3eeacf27
permissions -rw-r--r--
Reworked the redirects to work with the new Request model. Also made use of the collection of Role Views in the Role View in combination with the role property in the Request model, to facilitate the redirects.

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /