app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sun, 16 Nov 2008 23:04:48 +0000
changeset 495 87afae6e4c51
parent 73 211a3eeacf27
permissions -rw-r--r--
Added basic 'invite' functionality Currently the invite can only be created, existing ones can be listed and edited, or deleted if desired.

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /