pavuk.1.in

网络爬虫程序

.I Javascript support
.br
.I Cookie
.br
.I HTML rewriting engine tuning options
.br
.I Filename/URL Conversion Option
.br
.I Other Options
.br

.SH Mode
.sp
.TP
.I -mode {normal, linkupdate, sync, singlepage, singlereget, resumeregets}
Set operation mode.
.br
.B normal
- retrieves recursive documents
.br
.B linkupdate
- update remote URLs in local HTML documents to local URLs if these URLs
exist in the local tree
.br
.B sync
- synchronize remote documents with local tree (if a local copy of
a document is older than remote, the document is retrieved again,
otherwise nothing happens)
.br
.B singlepage
- URL is retrieved as one page with all inline objects (picture, sound ...)
this mode is now obsoleted by \fB-singlepage\fR option.
.br
.B resumeregets
- pavuk scans the local tree for files that were not retrieved fully
and retrieves them again (uses partial get if possible)
.br
.B singlereget
- get URL until it is retrieved in full
.br
.B dontstore
- transfer page from server, but don't store it to the local tree.
This mode is suitable for fetching pages that are held in a local
proxy/cache server.
.br
.B reminder
- used to inform the user about changed documents
.br
.B ftpdir
- used to list of contents of FTP directories
.br
.sp
default operation mode is
.B normal
mode.

.SH Help
.sp
.TP
.I -h
Print long verbose help message
.TP
.I -v
Show version informations and configuration at compilation time.

.SH Indicate/Logging/Interface options
.sp
.TP
.I -quiet
Don't show any messages on the screen.
.TP
.I -verbose
Force to show output messages on the screen (default)
.TP
.I -progress/-noprogress
Show retrieving progress while running in the terminal (default is progress off)
.TP
.I -stime/-nostime
Show start and end time of transfer. (default isn't this information shown)
.TP
.I -xmaxlog $nr
Maximum number of log lines in the Log widget. 0 means unlimited.
This option is available only when compiled with the GTK+ GUI. (default value is 0)
.TP
.I -logfile $file
File where all produced messages are stored.
.TP
.I -unique_log/-nounique_log
When logfile as specified with the option
.B -logfile
is already used by another process, try to generate new unique name
for the log file. (default is this option turned off)
.TP
.I -slogfile $file
File to store short logs in. This file contains one line of
informations per processed document.  This is meant to be used in
connection with any sort of script to produce some statistics, for
validating links on your website, or for generating simple sitemaps.
Multiple pavuk processes can use this file concurrently, without
overwriting each others entries.
Record structure:
.br
.sp
.RS
.nf
- \fBPID\fR of pavuk process
- \fBTIME\fR current time
- \fBCOUNTER\fR in the format current/total number of URLs
- \fBSTATUS\fR contains the type of the error: FATAL, ERR,
  WARN or OK
- \fBERRCODE\fR is the number code of the error
  (see errcode.h in pavuk sources)
- \fBURL\fR of the document
- \fBPARENTURL\fR first parent document of this URL
  (when it doesn't have parent - [none])
- \fBFILENAME\fR is the name of the local file the
  document is saved under
- \fBSIZE\fR size of requested document if known
- \fBDOWNLOAD_TIME\fR time which takes downloading of this
  document in format seconds.mili_seconds
- \fBHTTPRESP\fR contains the first line of the HTTP server
  response
.fi
.RE
.TP
.I -language $str
Native language that pavuk should use for communication with its
user (works only when there is a message catalog for that language)
\fBGNU gettext\fR support (for message internationalization) must also be
compiled in. Default language is taken from your NLS environment variables.
.TP
.I -gui_font $font
Font used in the GUI interface. To list available X fonts use the
.B xlsfonts
command.
This option is available only when compiled with GTK+ GUI support.

.SH Netli options
.SP
.TP
.I -[no]read_css
Enable or disable fetching objects mentioned in style sheets.
.SP
.TP
.I -[no]verify
Enable or disable verifying server CERTS in SSL mode.

.SP
.TP
.I -tlogfile $file
Turn on Netli logging with output to specified file.

.SP
.TP
.I -trelative {object | program}
Make Netli timings relative to the start of the first object or the program.

.SP
.TP
.I -transparent_proxy FQDN[:port]
When processing URL, send the original, but send it to the IP address
at FQDN

.SP
.TP
.I -transparent_ssl_proxy FQDN[:port]
When processing HTTPS URL, send the original, but send it to the IP address
at FQDN

.SP
.TP
.I -sdemo
Output in sdemo compatible format. This is only used by sdemo. (For
now it simply means output '-1' rather than '*'  when measurements are
invalid.)

.SP
.TP
.I -noencode
Do not escape characters that are "unsafe" in URLS.

.SH Special start
.sp
.TP
.I -X
Start program with X Window interface (if compiled with support for GTK+).
Pavuk as default starts without GUI, and behaves as regular commandline tool.
.TP
.I -runX
When used together with the \fB-X\fR option, pavuk starts processing of
URLs immediately after the GUI window is launched. Without the \fB-X\fR
given, this option doesn't have any effect.
Only available when compiled with GTK+ support .
.TP
.I -bg/-nobg
This option allows pavuk to detach from its terminal and run in
background mode.  Pavuk will not output any messages to the terminal
than. If you want to see messages, you have to use the
.B -log_file
option to specify a file where messages will be written.
Default pavuk executes at foreground.
.TP
.I -check_bg/-nocheck_bg
Normally, programs sent into the background after being run in
foreground continue to output messages to the terminal.  If this
option is activated, pavuk checks if it is running as background job
and will not write any messages to the terminal in this case. After
it becomes a foreground job again, it will start writing messages to
terminal in the normal way.  This option is available only when your
system supports retrieving of terminal info via \fBtc*()\fR functions.
.TP
.I -prefs/-noprefs
When you turn this option on, pavuk will preserve all settings when exiting, and
when you run pavuk with GUI interface again, all settings will be restored.
The settings will be stored in the
.B ~./pavuk_prefs
file. Default pavuk want restore its option when started.
This option is available only when compiled with GTK+.
.TP
.I -schedule $time
Execute pavuk at the time specified as parameter. The Format of the
$time parameter is YYYY.MM.DD.hh.mm.  You need a properly configured
scheduling with the \fBat\fR command on your system for using this option.
If default configuration (at -f %f %t %d.%m.%Y) of scheduling command won't work on your system, try to adjust it with \fB-sched_cmd\fR option.
.TP
.I -reschedule $nr
Execute pavuk periodically with $nr hours period.  You need properly
configured scheduling with the \fBat\fR command on your system for using this option.
.TP
.I -sched_cmd $str
Command to use for scheduling. Pavuk explicitly supports scheduling with
\fBat\fR $str should contain regular characters and macros, escaped by \fB%\fR character.
Supported macros are:
.br
.in +3
.B %f
 - for script filename
.br
.B %t
 - for time (in format HH:MM)
.br
 - all macros as supported by the \fBstrftime()\fR function
.in -3
.TP
.I -urls_file $file
If you use this option, pavuk will read URLs from $file before it
starts processing.  In this file, each URL needs to be on a separate
line. After the last URL, a single dot \fB.\fR followed by a LF
(line-feed) character denotes the end.  Pavuk will start processing
right after all URLs have been read.  If \fB$file\fR is given as the \fB-\fR
character, standard input will be read.
.TP
.I -store_info/-nostore_info
This option causes pavuk to store information about each document
into a separate file in the \fB.pavuk_info\fR
directory. This file is used to store the original URL from which
the document was downloaded. For files that are downloaded via
HTTP or HTTPS protocols, the whole HTTP response header is stored
there. I recommend to use this option when you are using options
that change the default layout of the local document tree, because
this info file helps pavuk to map the local filename to the
URL. This option is also very useful when different URLs have the
same filename in the local tree. When this occurs, pavuk detects
this using info files, and it will prefix the local name with
numbers. At default is disabled storing of this extra informations.
.TP
.I -info_dir $dir
You can set with this option location of separate directory for storing
info files created when \fB-store_info\fR option is used. This is useful
when you don't want to mix in destination directory the info files with
regular document files. The structure of the info files is preserved, just
are stored in different directory.
.TP
.I -request $req
With this option you can specify extended informations for starting URLs.
With this option you can specify query data for \fBPOST\fR or \fBGET\fR .
Current syntax of this option is :
\fBURL:["]$url["] [METHOD:["]{GET|POST}["]] [ENCODING:["]{u|m}["]] [FIELD:["]variable=value["]] [FILE:["]variable=filename["] [LNAME:["]local_filename["]]\fR
.sp
.RS
.nf
- \fBURL:\fR specifies request URL
- \fBMETHOD:\fR specifies request method for URL and is
  one of \fIGET\fR or \fIPOST\fR.
- \fBENCODING:\fR specifies encoding for request body data.
    \fBm\fR is for \fImultipart/form-data\fR encoding
    \fBu\fR is for \fIapplication/x-www-form-urlencoded\fR
    encoding
- \fBFIELD:\fR specifies field of request data in format
    \fBvariable=value\fR. For encoding of special characters
    in \fIvariable\fR and \fIvalue\fR you can use same encoding
    as is used in \fBapplication/x-www-form-urlencoded\fR
    encoding.
- \fBFILE:\fR specifies special field of query, which is
    used to specify file for \fBPOST\fR based file upload.
- \fBLNAME:\fR specifies localname for this request
.fi
.RE
When you need to use inside the \fBFIELD:\fR and \fBFILE:\fR fields of
request specification special characters, you should use the
\fBapplication/x-www-form-urlencoded\fR encoding of characters. It means
all nonASCII characters, quote character ("), space character ( ),
ampersand character (&), percent character (%) and equal character (=)
should be encoded in form \fB%xx\fR where \fBxx\fR is hexadecimal
representation of ASCII value of character. So for example \fI%\fR character
should be encoded like \fI%25\fR.
.TP
.I -formdata $req
This option gives you chance to specify contents for HTML forms found during
traversing document tree.
 Current syntax of this option is same as for \fB-request\fR option, but
\fBENCODING:\fR and \fBMETHOD:\fR are meaningless in this option semantics.
 In \fBURL:\fR you have to specify HTML form action URL, which will be matched
against action URLs found in processed HTML documents. If pavuk finds action URL
which matches that supplied in \fB-formdata\fR option, pavuk will construct
\fBGET\fR or \fBPOST\fR request from data supplied in this option and from default
form field values supplied in HTML document. Values supplied on commandline have
precedence before that supplied in HTML file.
.TP
.I -nthreads $nr
By means of this option you can specify how many concurrent threads will
download documents. Default pavuk executes 3 concurrent downloading threads.
This option is available only when pavuk is compiled to support multithreading.
.TP
.I -immesg/-noimmesg
Default pavuks behavior when running multiple downloading threads is to buffer
all output messages in memory buffer and flush that buffered data just when thread
finishes processing of one document. With this option you can change this
behavior to see the messages immediately when it is produced. It is only usable
when you want to debug some specials in multithreading environment.
This option is available only when pavuk is compiled to support multithreading.