# http://www.robotstxt.org/wc/faq.html User-agent: ia_archiver Disallow: / #User-agent: webcrawler #Disallow: #User-Agent: DepSpid #Disallow: / #User-agent: * #Disallow: