app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sat, 31 Jan 2009 15:41:45 +0000
changeset 1115 0a723ff3d27c
parent 73 211a3eeacf27
permissions -rw-r--r--
Cleanups in base.Logic * Use regular kwargs instead of **kwargs * Fold getKeyNameFromFields into _keyName * Fold getIfFields into getFromFieldsOr404 * Rename getKeyValues to getKeyValuesFromEntity * Rename getKeyFieldsFromDict to getKeyFieldsFromFields * Rename getFromFieldsOr404 to getFromKeyFieldsOr404 * Rename getKeyValues to getKeyValuesFromEntity Patch by: Sverre Rabbelier
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
73
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     1
# Directions for web crawlers.
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     2
# See http://www.robotstxt.org/wc/norobots.html.
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     3
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     4
User-agent: HTTrack
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     5
User-agent: puf
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     6
User-agent: MSIECrawler
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     7
User-agent: Nutch
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset
     8
Disallow: /