author | Sverre Rabbelier <srabbelier@gmail.com> |
Thu, 23 Oct 2008 05:21:26 +0000 | |
changeset 409 | 9d24850db88f |
parent 73 | 211a3eeacf27 |
permissions | -rw-r--r-- |
73
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
1 |
# Directions for web crawlers. |
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
2 |
# See http://www.robotstxt.org/wc/norobots.html. |
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
3 |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
4 |
User-agent: HTTrack |
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
5 |
User-agent: puf |
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
6 |
User-agent: MSIECrawler |
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
7 |
User-agent: Nutch |
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
8 |
Disallow: / |