| author | Lennard de Rijk <ljvderijk@gmail.com> |
| Fri, 03 Jul 2009 18:20:33 +0200 | |
| changeset 2510 | a85484d1f441 |
| parent 73 | 211a3eeacf27 |
| permissions | -rw-r--r-- |
|
73
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
1 |
# Directions for web crawlers. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
2 |
# See http://www.robotstxt.org/wc/norobots.html. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
3 |
|
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
4 |
User-agent: HTTrack |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
5 |
User-agent: puf |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
6 |
User-agent: MSIECrawler |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
7 |
User-agent: Nutch |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
8 |
Disallow: / |