app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sun, 15 Feb 2009 22:29:22 +0000
changeset 1360 f62c462037b6
parent 73 211a3eeacf27
permissions -rw-r--r--
Added grouping support to all forms, converted role as example Any field that has the 'group' property set be placed in the specified group. If no such proprty is set, the 'General' group will be used instead. Patch by: Sverre Rabbelier

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /