app/soc/content/robots.txt
author James Levy <jamesalexanderlevy@gmail.com>
Sun, 28 Jun 2009 13:13:58 +0200
changeset 2429 a0a1dd1cc69e
parent 73 211a3eeacf27
permissions -rw-r--r--
Added SurveyRecordModel. Lennard: On top of the orginial patch I've added several comments to the fields and two TODO's for further subclassing. Patch by: James Levy, Daniel Diniz, Lennard de Rijk Reviewed by: Lennard de Rijk

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /