app/soc/content/robots.txt
author Lennard de Rijk <ljvderijk@gmail.com>
Sun, 15 Mar 2009 22:32:05 +0000
changeset 1887 8b71f4e58f39
parent 73 211a3eeacf27
permissions -rw-r--r--
Changed notifications helper to properly use the fact that from_user is not a required property. The list template has changed to reflect this. The naming is open for discussion but note that if it is decided to change it that it also needs to be changed in the public.html template. Patch by: Lennard de Rijk Reviewed by: to-be-reviewed

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /