app/soc/content/robots.txt
author Pawel Solyga <Pawel.Solyga@gmail.com>
Mon, 10 Nov 2008 14:40:16 +0000
changeset 462 1f164cd0529b
parent 73 211a3eeacf27
permissions -rw-r--r--
Adds a check to logic/valididate.py that checks the partial path format using a regexp. This regexp has been added to logic/path_link_name.py, therefore changing PATH_LINKNAME_REGEX to use the PARTIAL_PATH in building this regexp. Plus views/models/docs.py now uses the validation functions in the clean_ methods. Patch by: Lennard de Rijk Review by: Pawel Solyga

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /