app/soc/content/robots.txt
author Pawel Solyga <Pawel.Solyga@gmail.com>
Mon, 03 Nov 2008 15:03:09 +0000
changeset 436 f3c313d54aa4
parent 73 211a3eeacf27
permissions -rw-r--r--
Add missing TODOs to soc.logic.models classes. Fix comments format in base.Logic class. Changes based on comments to r972. http://code.google.com/p/soc/source/detail?r=972 Patch by: Pawel Solyga

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /