| author | Mario Ferraro <fadinlight@gmail.com> |
| Sat, 09 May 2009 01:14:03 +0200 | |
| changeset 2300 | b9fd1e20d413 |
| parent 73 | 211a3eeacf27 |
| permissions | -rw-r--r-- |
|
73
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
1 |
# Directions for web crawlers. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
2 |
# See http://www.robotstxt.org/wc/norobots.html. |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
3 |
|
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
4 |
User-agent: HTTrack |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
5 |
User-agent: puf |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
6 |
User-agent: MSIECrawler |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
7 |
User-agent: Nutch |
|
211a3eeacf27
Created robots.txtx and simple Melange favicon.
Pawel Solyga <Pawel.Solyga@gmail.com>
parents:
diff
changeset
|
8 |
Disallow: / |