app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sat, 06 Dec 2008 11:53:26 +0000
changeset 675 14958011dceb
parent 73 211a3eeacf27
permissions -rw-r--r--
Use key().name() instead of link_id This is now possible because key_name is constructed purely from the key fields of an entity. It is not sufficient to use just link_id, that works only for single-scoped entities (e.g., those that either do not have a scope, or that have a scope which itself does not have a scope). It would break if there was an entity that has a scoped scope (it would only include the scope's link_id in the url, which made it impossible to look up the scope as we missed the link_id of the scope's scope). Patch by: Sverre Rabbelier

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /