Fix bug where script fails when it encounters a socket (which is not a regular file). Fix bug where exception is raised if error occurred before parser object was returned from _buildParser(). Add a --notext option, so that just the file paths will be output on matches (both find and find/replace); useful with xargs. Patch by: Todd Larsen

# Directions for web crawlers.
# See http://www.robotstxt.org/wc/norobots.html.

User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
User-agent: Nutch
Disallow: /