robots.txt

Heritrix是一个开源,可扩展的web爬虫项目。Heritrix设计成严格按照robots.txt文件的排除指示和META robots标签。

# This robots.txt is used by the RobotsExclusion heritrix integration self test.
User-agent: heritrix_selftest(+http://localhost.localdomain)
Disallow: /RobotsExclusion/excluded.html
Disallow: /RobotsExclusion/excluded/