app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Mon, 03 Nov 2008 19:47:02 +0000 (2008-11-03)
changeset 439 0658c3c9a9dc
parent 73 211a3eeacf27
permissions -rw-r--r--
Minor fixes needed for generic key name We no longer try to retreive an entity when there are unset fields. This sort of makes 'getIfFields' obsolete, since we check if fields now anyway. This is needed because getKeyFieldsFromDict expects the fields to be set. Also a minor fix in a Django template so that the generic 'edit' page has a working delete button again.
# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /