mydomain-url-filter

harvest是一个下载html网页得机器人

# URL Filter file for 'mydomain'
#
# Here 'URL' really means the pathname part of a URL.  Hosts and ports
# dont belong in this file.
#
# Format is
#
#    Allow	regex
#    Deny	regex
#
# Lines are evaulated in order; the first line to match is applied.
#
Deny	\.gif$				# don't retrieve GIF images
Deny	\.GIF$				# 
Deny	^/internal/			# don't index internal documentso
Allow	.*				# allow everything else.