byurl.cf

harvest是一个下载html网页得机器人

#
#  byurl.cf - URL naming conventions that yield type recognition
#
#  Format: <Type> <Regular Expression for filename match>
#
#  All regular expressions are in POSIX extended form and must have
#  a ^ first and a $ last.  The types are evaluated in the order 
#  specified.  All regular expressions are CASE SENSITIVE.
#
#  These types must always be evaluated first
#
WUSTL-PCIndex		^ftp.*\.wustl\.edu.*INDEX.*$
HTTP-Query		^http.*cgi.*\?.*$