app/soc/content/robots.txt
author Madhusudan.C.S <madhusudancs@gmail.com>
Tue, 15 Sep 2009 22:01:16 +0200
changeset 2926 effa1cfb3607
parent 73 211a3eeacf27
permissions -rw-r--r--
Added school_type and grade property to Student model. A GAE Task will be provided to update the existing entities in a later commit. Reviewed by: Lennard de Rijk

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /