app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sun, 25 Jan 2009 16:28:07 +0000
changeset 987 6fd5c561b446
parent 73 211a3eeacf27
permissions -rw-r--r--
Cache sidebar with a simple caching API We cache the sidebar once for every user, the downside is that once the user meets new access criteria, their sidebar does not get updated. This can be fixed by calling soc.cache.sidebar.flush() when appropriate. Patch by: Sverre Rabbelier

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /