app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Thu, 09 Oct 2008 23:38:27 +0000
changeset 293 1edd01373e71
parent 73 211a3eeacf27
permissions -rw-r--r--
Add an access control module The module's purpose is to check access requirements, (such as being logged in), and raise an exception containing a authentication request when the requirement is not met. Patch by: Sverre Rabbelier Reviewed by: Pawel Solyga, Augie Fackler, Todd Larsen Reviewed at: http://codereviews.googleopensourceprograms.com/1601 Review id: 1601

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /