byurl.cf

harvest是一个下载html网页得机器人

#
#  byurl.cf - URL naming conventions that yield type recognition
#
#  Format: <Type> <Regular Expression for filename match>
#
#  All regular expressions are in POSIX extended form and must have
#  a ^ first and a $ last.  The types are evaluated in the order 
#  specified.  All regular expressions are CASE SENSITIVE.
#
#  These types must always be evaluated first
#
HomeHTML                ^http:.*/$
HomeHTML                ^http:.*[hH]ome\.html$
HomeHTML                ^http:.*[hH]ome[pP]age\.html$
HomeHTML                ^http:.*[wW]elcome\.html$
HomeHTML                ^http:.*/index\.html$
Unknown			^.*$