| author | Daniel Hans <Daniel.M.Hans@gmail.com> |
| Fri, 30 Oct 2009 22:24:34 +0100 | |
| changeset 3069 | 1b9c554ca96d |
| parent 73 | 211a3eeacf27 |
| permissions | -rw-r--r-- |
|
73
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
1 |
# Directions for web crawlers. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
2 |
# See http://www.robotstxt.org/wc/norobots.html. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
3 |
|
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
4 |
User-agent: HTTrack |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
5 |
User-agent: puf |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
6 |
User-agent: MSIECrawler |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
7 |
User-agent: Nutch |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
8 |
Disallow: / |