app/soc/content/robots.txt
author Pawel Solyga <Pawel.Solyga@gmail.com>
Tue, 16 Sep 2008 15:56:15 +0000
changeset 150 715b07485c48
parent 73 211a3eeacf27
permissions -rw-r--r--
Changed ZIPFILE variable to DJANGO_ZIPFILE in make_release.sh script and added 'cd $APP_FOLDER' before we zip django.

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /