app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sun, 01 Mar 2009 20:20:47 +0000
changeset 1584 d8ba8c917f37
parent 73 211a3eeacf27
permissions -rw-r--r--
Make use of decorators.view for all views This gives us a 'view' hook where we can do things that need to be done for all views (such as catching certain exceptions, or checking for maintenance mode). Patch by: Sverre Rabbelier

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /