Refactor existing code to use the new access module Instead of ending up with many different ways to do access control, we end up having only one centralized place wher access control is done. Patch by: Sverre Rabbelier Reviewed by: Pawel Solyga, Augie Fackler, Todd Larsen Reviewed at: http://codereviews.googleopensourceprograms.com/1601 Review id: 1601

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /