Mercurial Mercurial > py_tasks_melange / annotate
summary | shortlog | changelog | graph | tags | bookmarks | branches | files | changeset | file | latest | revisions | annotate | diff | comparison | raw | help
Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
app/soc/content/robots.txt
author Sverre Rabbelier <srabbelier@gmail.com>
Sun, 25 Jan 2009 16:25:17 +0000
changeset 986 e9611a2288ca
parent 73 211a3eeacf27
permissions -rw-r--r--
Rename ModelProperties to EntityProperties We use 'model' when referring to the db.Model, the context in which 'ModelProperties' was used we really mean 'entity', which refers to an instantiation of the model. Patch by: Sverre Rabbelier
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
73
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset 
     1
 
# Directions for web crawlers.
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset 
     2
 
# See http://www.robotstxt.org/wc/norobots.html.
211a3eeacf27 Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff changeset 
     3
 












211a3eeacf27
Created robots.txtx and simple Melange favicon.



Pawel Solyga <Pawel.Solyga@gmail.com>


parents: 

diff
changeset




     4


User-agent: HTTrack













211a3eeacf27
Created robots.txtx and simple Melange favicon.



Pawel Solyga <Pawel.Solyga@gmail.com>


parents: 

diff
changeset




     5


User-agent: puf













211a3eeacf27
Created robots.txtx and simple Melange favicon.



Pawel Solyga <Pawel.Solyga@gmail.com>


parents: 

diff
changeset




     6


User-agent: MSIECrawler













211a3eeacf27
Created robots.txtx and simple Melange favicon.



Pawel Solyga <Pawel.Solyga@gmail.com>


parents: 

diff
changeset




     7


User-agent: Nutch













211a3eeacf27
Created robots.txtx and simple Melange favicon.



Pawel Solyga <Pawel.Solyga@gmail.com>


parents: 

diff
changeset




     8


Disallow: /















py_tasks_melange