readme

harvest是一个下载html网页得机器人

liburl is a library that parses and retrieves URLs.  The 'ftp', 'http',
'gopher', and 'file' access methods are supported.  

liburl also supports caching in two different ways:

	1) liburl will store retrieved URLs into a local disk cache.  
	   The local disk cache is stored in the /tmp directory (or 
	   the $TMPDIR environment variable), and may be shared by 
 	   many processes.

	2) liburl will cache control connections to FTP servers, 
	   allowing clients to retrieve multiple files from the 
	   same FTP server with only one login.

liburl was developed at the University of Colorado, Boulder by Jim Guyton,
Darren Hardy, David Merkel, and Mark Peterson as part of the Harvest
system.

Note that as of 10/31 the FTP connection cache is still being tested; 
use at your own risk.