app/soc/content/robots.txt
author Pawel Solyga <Pawel.Solyga@gmail.com>
Wed, 20 Aug 2008 21:59:00 +0000
changeset 98 b2b823466a8b
parent 73 211a3eeacf27
permissions -rw-r--r--
User Profile view (without controller yet). Patch by: Pawel Solyga Review by: Todd Larsen Review issue: 386 Review URL: http://codereviews.googleopensourceprograms.com/386

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /