pavuk.1.in

网络爬虫程序

.TP
.I -dumpfd $nr
For scripting is sometimes usable to be able to download document directly to
pipe or variable instead of storing it to regular file. In such case you can use
this option to dump data for example to stdout ($nr = 1).
.TP
.I -dump_after/-nodump_after
While using \fB-dumpfd\fR option in multithreaded pavuk, it is required to dump
document in one moment because documents downloaded in multiple threads can
overlap. This option is also useful when you want to dump document after pavuk
adjusts links inside HTML documents.
.TP
.I -dump_response/-nodump_response
This option have effect only when used with \fB-dumpfd\fR option. It is used to dump HTTP response headers.
.TP
.I -dump_urlfd $nr
When you will use this option, pavuk will output all URLs found in HTML
documents to file descriptor $nr. You can use this option to extract and
convert all URLs to absolute.

.SH Scenario/Task options
.sp
.TP
.I -scenario $str
Name of scenario to load and/or run. Scenarios are files with a structure
similar to the \fB.pavukrc\fR file.
Scenarios contain saved configurations. You can use it for periodical
mirroring. Parameters from scenarios specified at the command line
can be overwritten by command line parameters.
To be able to use this option, you need to specify scenario base
directory with option \fB-scndir\fR.
.TP
.I -dumpscn $filename
Store actual configuration into scenario file with name \fB$filename\fR.
This is useful to quickly create pre-configured scenarios for manual
editing.

.SH Directory options
.sp
.TP
.I -msgcat $dir
Directory which contains the message catalog for pavuk.
If you do not have permission
to store a pavuk message catalog in the system directory, you should simply
create similar structure of directories in your home directory as it is on
your system.
.sp
.B For example:
.sp
Your native language is German, and your home directory is
/home/jano.
.sp
You should at first create the directory
/home/jano/locales/de/LC_MESSAGES/, then put the German pavuk.mo
there and set -msgcat to /home/jano/locales/.  If you have properly
set locale environment values, you will see pavuk speaking German.
This option is available only when you compiled in support for GNU
gettext messages internationalization.
.TP
.I -cdir $dir
Directory where are all retrieved documents are stored. If not
specified, the current directory is used. If the specified directory doesn't
exist, it will be created.
.TP
.I -scndir $dir
Directory in which your scenarios are stored.
You must use this option when you are loading or storing scenario files.

.SH Preserve options
.sp
.TP
.I -preserve_time/-nopreserve_time
Store downloaded document with same modification time as on the remote site. Modification
time will be set only when such information is available (some FTP servers do not
support the \fBMDTM\fR command, and some documents on HTTP servers are created online so pavuk
can't retrieve the modification time of this document).
At default modification time of documents isn't preserved.
.TP
.I -preserve_perm/-nopreserve_perm
Store downloaded document with the same permissions as on the remote site.
This option has effect only when downloading a file through FTP protocol and
assumes that the \fB-ftplist\fR option is used. At default permissions are not
preserved.
.TP
.I -preserve_slinks/-nopreserve_slinks
Set symbolic links to point exactly to same location as on the remote server;
don't do any relocations.
This option has effect only when downloading file through FTP protocol and
assumes that the \fB-ftplist\fR option is used.
Default symbolic links are not preserved, and are retrieved as regular
documents with full contents of linked file.
.sp
For example, assume that on the FTP server ftp.xx.org there is a symbolic link
/pub/pavuk/pavuk-current.tgz, which points to /tmp/pub/pavuk-0.9pl11.tgz.
Pavuk will create symbolic link ftp/ftp.xx.org_21/pub/pavuk/pavuk-current.tgz
.br
if option -preserve_slinks will be used this symbolic link will point to
/tmp/pub/pavuk-0.9pl11.tgz
.br
if option -preserve_slinks want be used, this symbolic link will point to
 ../../tmp/pub/pavuk-0.9pl11.tgz
.TP
.I -retrieve_symlink/-noretrieve_symlink
Retrieve files behind symbolic links instead of replicating symlinks in local tree.

.SH Proxy options
.sp
.TP
.I -http_proxy $site[:$port]
If this parameter is used, then all HTTP requests are going through this proxy
server. This is useful if your site resides behind a firewall, or if you want
to use a HTTP proxy cache server. The default port number is 8080.
Pavuk allows you to specify multiple HTTP proxies
(using multiple -http_proxy options) and it will rotate proxies with roundrobin priority disabling proxies with errors.
.TP
.I -nocache/-cache
Use this option whenever you want to get the document directly from
the site and not from your HTTP proxy cache server. Default pavuk allows
transfer of document copies from cache.
.TP
.I -ftp_proxy $site[:$port]
If this parameter is used, then all FTP requests are going through this proxy
server.
This is useful when your site resides behind a firewall, or if you want to use
FTP proxy cache server.  The default port number is 22.
Pavuk supports three different types of proxies for FTP, see the options
\fB-ftp_httpgw, -ftp_dirtyproxy.\fR
If none of the mentioned options is used, then pavuk assumes a regular
FTP proxy with \fBUSER user@host\fR connecting to remote FTP server.
.TP
.I -ftp_httpgw/-noftp_httpgw
The specified FTP proxy is a HTTP gateway for the FTP protocol. Default FTP proxy is regular FTP proxy.
.TP
.I -ftp_dirtyproxy/-noftp_dirtyproxy
The specified FTP proxy is a HTTP proxy which supports a \fBCONNECT\fR request
(pavuk should use full FTP protocol, except of active data connections).
Default FTP proxy is regular FTP proxy.
If both -ftp_dirtyproxy and -ftp_httpgw are specified, -ftp_dirtyproxy is preferred.
.
.TP
.I -gopher_proxy $site[:$port]
Gopher gateway or proxy/cache server.
.TP
.I -gopher_httpgw/-nogopher_httpgw
The specified Gopher proxy server is a HTTP gateway for Gopher protocol.
When \fB-gopher_proxy\fR is set and this \fB-gopher_httpgw\fR option isn't used,
pavuk is using proxy as HTTP tunnel with \fBCONNECT\fR request to open connections to Gopher servers.
.TP
.I -ssl_proxy $site[:$port]
SSL proxy (tunneling) server [as that in CERN httpd + patch or in
Squid] with enabled \fBCONNECT\fR request (at least on port 443). This
option is available only when compiled with SSL support (you need
the SSleay or OpenSSL libraries with development headers)

.SH Proxy Authentification
.sp
.TP
.I -http_proxy_user $user
Username for HTTP proxy authentification.
.TP
.I -http_proxy_pass $pass
Password for HTTP proxy authentification.
.TP
.I -http_proxy_auth {1/2/3/4/user/Basic/Digest/NTLM}
Authentification scheme for proxy access. Similar meaning as the
\fB-auth_scheme\fR option (see help for this option for more details).
Default is 2 (Basic scheme).
.TP
.I -auth_proxy_ntlm_domain $str
NT or LM domain used for authorization again HTTP proxy server when NTLM authentification scheme is required. This option is available only when compiled with OpenSSL or libdes libraries.
.TP
.I -auth_reuse_proxy_nonce/-noauth_reuse_proxy_nonce
When using HTTP Proxy Digest access authentification scheme use
first received nonce value in multiple following requests.
.TP
.I -ftp_proxy_user $user
Username for FTP proxy authentification.
.TP
.I -ftp_proxy_pass $pass
Password for FTP proxy authentification.

.SH Protocol/Download Options
.sp
.TP
.I -ftp_passive
Uses passive ftp when downloading via ftp.
.TP
.I -ftp_active
Uses active ftp when downloading via ftp.
.TP
.I -active_ftp_port_range $min:$max
This option permits to specify the ports used for
active ftp. This permits easier firewall configuration since the range
of ports can be restricted.
.sp
Pavuk will randomly choose a number from within the specified
range until an open port is found. Should no open ports be found
within the given range, pavuk will default to a normal
kernel-assigned port, and a message (debug level net) is output.
.sp
The port range selected must be in the non-privileged range
(eg. greater than or equal to 1024); it is STRONGLY RECOMMENDED that
the chosen range be large enough to handle many simultaneous
active connections (for example, 49152-65534, the IANA-registered
ephemeral port range).
.TP
.I -always_mdtm/-noalways_mdtm
Force pavuk to always use "MDTM" to determine the file modification
time and never uses cached times determined when listing the remote
files.
.TP
.I -remove_before_store/-noremove_before_store
Force unlink'ing of files before new content is stored to a file. This
is helpful if the local files are hardlinked to some other directory
and after mirroring the hardlinks are checked. All "broken" hardlinks
indicate a file update.
.TP
.I -retry $nr
Set the number of attempts to transfer processed document.
Default set to 1, this mean pavuk will retry once to get documents
which failed on first attempt.
.TP
.I -nregets $nr
Set the number of allowed regets on a single document, after a broken transfer.
Default value for this option is 2.
.TP
.I -nredirs $nr
Set number of allowed HTTP redirects. (use this for prevention of loops)
Default value for this option is 5, and conform to HTTP specification.
.TP
.I -force_reget/-noforce_reget
Force reget'ing of the whole document after a broken transfer when
the server doesn't support retrieving of partial content.
Pavuk default behavior is to stop getting documents which don't allow
restarting of transfer from specified position.
.TP
.I -timeout $nr
Timeout for stalled connections in minutes. This value is also used for
connection timeouts. For sub-minute timeouts you can use floating point numbers.
Default timeout is 0, an that means timeout checking is disabled.
.TP
.I -noRobots/-Robots
This switch suppresses the use of the \fBrobots.txt\fR
standard, which is used to restrict access of Web robots to some
locations on the web server. Default is allowed checking of robots.txt
files on HTTP servers. Enable this option always when you are downloading
huge sets of pages with unpredictable layout.
This prevents you from upsetting server administrators :-).
.TP
.I -noEnc/-Enc
This switch suppresses using of \fBgzip\fR or \fBcompress\fR or \fBdeflate\fR
encoding in transfer. I don't know if some servers are broken or what, but they
are propagating that MIME type application/gzip or application/compress as
encoded. Turn this option off, when you doesn't have libz support compiled in
and also \fBgzip\fR program which is used to decode document encoded this way.
At default is decoding of downloaded document disabled.
.TP
.I -check_size/-nocheck_size
The option -nocheck_size should be used if you are trying to
download pages from a HTTP server which sends a wrong
\fBContent-Length:\fR field in the MIME header of response.
Default pavuk behavior is to check this field and complain
when something is wrong.
.TP
.I -maxrate $nr
If you don't want to give all your transfer bandwidth to pavuk, use
this option to set pavuk's maximum transfer rate. This option
accepts a floating point number to specify the transfer rate in kB/s. If
you want get optimal settings, you also have to play with the size of the read
buffer (option \fB-bufsize\fR) because pavuk is doing flow control only at
application level.
At default pavuk use full bandwidth.
.TP
.I -minrate $nr
If you hate slow transfer rates, this option allows you to break
transfers with slow speed. You can set the minimum transfer rate,
and if the connection gets slower than the given rate, the transfer
will be stopped. The minimum transfer rate is given in kB/s.
At default pavuk doesn't check this limit.
.TP
.I -bufsize $nr
This option is used to specify the size of the read buffer (default
size: 32kB).  If you have a very fast connection, you may increase
the size of the buffer to get a better read performance. If you need
to decrease the transfer rate, you may need to decrease the size of
the buffer and set the maximum transfer rate with the \fB-maxrate\fR
option. This option accepts the size of the buffer in kB.
.TP
.I -fs_quota $nr
If you are running pavuk on a multiuser system, you may need to
avoid filling up your file system. This option lets you specify how
many space must remain free. If pavuk detects an underrun of the
free space, it will stop downloading files. Specify this quota in
kB. Default value is 0, and that mean no checking of this quota.
.TP
.I -file_quota $nr
This option is useful when you want to limit downloading of big files,
but want to download at least $nr kilobytes from big files.
A big file will be transferred, and when it reaches the specified size,
transfer will break. Such document will be processed as properly downloaded,
so be careful when using this option.
At default pavuk is transferring full size of documents.
.TP
.I -trans_quota $nr
If you are aware that your selection should address a big amount of
data, you can use this option to limit the amount of transferred data.
Default is by size unlimited transfer.
.TP
.I -max_time $nr
Set maximum amount of time for program run. After time is exceeded, pavuk
will stop downloading. Time is specified in minutes. Default value is 0, and it means downloading time is not limited.
.TP
.I -url_strategy $strategy
This option allows you to specify a downloading order for URLs in document tree.
This option accepts the following strings as parameters :
.br
.sp
.RS
.B level
- will order URLs as it loads it from HTML files (default)
.br
.B leveli
- as previous, but inline objects URLs come first
.br
.B pre
- will insert URLs from actual HTML document at start, before other
.br
.B prei
- as previous, but inline objects URLs come first
.br
.RE