| author | Daniel Hans <Daniel.M.Hans@gmail.com> |
| Sat, 14 Nov 2009 23:58:20 +0100 | |
| changeset 3092 | beeb5d111318 |
| parent 73 | 211a3eeacf27 |
| permissions | -rw-r--r-- |
|
73
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
1 |
# Directions for web crawlers. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
2 |
# See http://www.robotstxt.org/wc/norobots.html. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
3 |
|
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
4 |
User-agent: HTTrack |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
5 |
User-agent: puf |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
6 |
User-agent: MSIECrawler |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
7 |
User-agent: Nutch |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
8 |
Disallow: / |