1 # Directions for web crawlers.
2 # See http://www.robotstxt.org/wc/norobots.html.
3
4 User-agent: HTTrack
5 User-agent: puf
6 User-agent: MSIECrawler
7 User-agent: Nutch
8 Disallow: /