app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Wed, 04 Mar 2009 23:11:33 +0000
changeset 1672 9b6cc71dd70c
parent 73 211a3eeacf27
permissions -rw-r--r--
Only show key_fields in error message This caused a crash when GET, POST, and context were added to the django_args by the decorators. Patch by: Sverre Rabbelier

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /