wget.1

Wget很好的处理了http和ftp的下载,很值得学习的经典代码

administrator may have chosen to compile Wget without debug support, in
which case \fB\-d\fR will not work.  Please note that compiling with
debug support is always safe\-\-\-Wget compiled with the debug support will
\&\fInot\fR print any debug info unless requested with \fB\-d\fR.
.IP "\fB\-q\fR" 4
.IX Item "-q"
.PD 0
.IP "\fB\-\-quiet\fR" 4
.IX Item "--quiet"
.PD
Turn off Wget's output.
.IP "\fB\-v\fR" 4
.IX Item "-v"
.PD 0
.IP "\fB\-\-verbose\fR" 4
.IX Item "--verbose"
.PD
Turn on verbose output, with all the available data.  The default output
is verbose.
.IP "\fB\-nv\fR" 4
.IX Item "-nv"
.PD 0
.IP "\fB\-\-no\-verbose\fR" 4
.IX Item "--no-verbose"
.PD
Turn off verbose without being completely quiet (use \fB\-q\fR for
that), which means that error messages and basic information still get
printed.
.IP "\fB\-i\fR \fIfile\fR" 4
.IX Item "-i file"
.PD 0
.IP "\fB\-\-input\-file=\fR\fIfile\fR" 4
.IX Item "--input-file=file"
.PD
Read URLs from \fIfile\fR.  If \fB\-\fR is specified as
\&\fIfile\fR, URLs are read from the standard input.  (Use
\&\fB./\-\fR to read from a file literally named \fB\-\fR.)
.Sp
If this function is used, no URLs need be present on the command
line.  If there are URLs both on the command line and in an input
file, those on the command lines will be the first ones to be
retrieved.  The \fIfile\fR need not be an \s-1HTML\s0 document (but no
harm if it is)\-\-\-it is enough if the URLs are just listed
sequentially.
.Sp
However, if you specify \fB\-\-force\-html\fR, the document will be
regarded as \fBhtml\fR.  In that case you may have problems with
relative links, which you can solve either by adding \f(CW\*(C`<base
href="\f(CIurl\f(CW">\*(C'\fR to the documents or by specifying
\&\fB\-\-base=\fR\fIurl\fR on the command line.
.IP "\fB\-F\fR" 4
.IX Item "-F"
.PD 0
.IP "\fB\-\-force\-html\fR" 4
.IX Item "--force-html"
.PD
When input is read from a file, force it to be treated as an \s-1HTML\s0
file.  This enables you to retrieve relative links from existing
\&\s-1HTML\s0 files on your local disk, by adding \f(CW\*(C`<base
href="\f(CIurl\f(CW">\*(C'\fR to \s-1HTML\s0, or using the \fB\-\-base\fR command-line
option.
.IP "\fB\-B\fR \fI\s-1URL\s0\fR" 4
.IX Item "-B URL"
.PD 0
.IP "\fB\-\-base=\fR\fI\s-1URL\s0\fR" 4
.IX Item "--base=URL"
.PD
Prepends \fI\s-1URL\s0\fR to relative links read from the file specified with
the \fB\-i\fR option.
.Sh "Download Options"
.IX Subsection "Download Options"
.IP "\fB\-\-bind\-address=\fR\fI\s-1ADDRESS\s0\fR" 4
.IX Item "--bind-address=ADDRESS"
When making client \s-1TCP/IP\s0 connections, bind to \fI\s-1ADDRESS\s0\fR on
the local machine.  \fI\s-1ADDRESS\s0\fR may be specified as a hostname or \s-1IP\s0
address.  This option can be useful if your machine is bound to multiple
IPs.
.IP "\fB\-t\fR \fInumber\fR" 4
.IX Item "-t number"
.PD 0
.IP "\fB\-\-tries=\fR\fInumber\fR" 4
.IX Item "--tries=number"
.PD
Set number of retries to \fInumber\fR.  Specify 0 or \fBinf\fR for
infinite retrying.  The default is to retry 20 times, with the exception
of fatal errors like \*(L"connection refused\*(R" or \*(L"not found\*(R" (404),
which are not retried.
.IP "\fB\-O\fR \fIfile\fR" 4
.IX Item "-O file"
.PD 0
.IP "\fB\-\-output\-document=\fR\fIfile\fR" 4
.IX Item "--output-document=file"
.PD
The documents will not be written to the appropriate files, but all
will be concatenated together and written to \fIfile\fR.  If \fB\-\fR
is used as \fIfile\fR, documents will be printed to standard output,
disabling link conversion.  (Use \fB./\-\fR to print to a file
literally named \fB\-\fR.)
.Sp
Use of \fB\-O\fR is \fInot\fR intended to mean simply "use the name
\&\fIfile\fR instead of the one in the \s-1URL\s0;" rather, it is
analogous to shell redirection:
\&\fBwget \-O file http://foo\fR is intended to work like
\&\fBwget \-O \- http://foo > file\fR; \fIfile\fR will be truncated
immediately, and \fIall\fR downloaded content will be written there.
.Sp
Note that a combination with \fB\-k\fR is only permitted when
downloading a single document, and combination with any of \fB\-r\fR,
\&\fB\-p\fR, or \fB\-N\fR is not allowed.
.IP "\fB\-nc\fR" 4
.IX Item "-nc"
.PD 0
.IP "\fB\-\-no\-clobber\fR" 4
.IX Item "--no-clobber"
.PD
If a file is downloaded more than once in the same directory, Wget's
behavior depends on a few options, including \fB\-nc\fR.  In certain
cases, the local file will be \fIclobbered\fR, or overwritten, upon
repeated download.  In other cases it will be preserved.
.Sp
When running Wget without \fB\-N\fR, \fB\-nc\fR, \fB\-r\fR, or \fBp\fR,
downloading the same file in the same directory will result in the
original copy of \fIfile\fR being preserved and the second copy being
named \fIfile\fR\fB.1\fR.  If that file is downloaded yet again, the
third copy will be named \fIfile\fR\fB.2\fR, and so on.  When
\&\fB\-nc\fR is specified, this behavior is suppressed, and Wget will
refuse to download newer copies of \fIfile\fR.  Therefore,
"\f(CW\*(C`no\-clobber\*(C'\fR" is actually a misnomer in this mode\-\-\-it's not
clobbering that's prevented (as the numeric suffixes were already
preventing clobbering), but rather the multiple version saving that's
prevented.
.Sp
When running Wget with \fB\-r\fR or \fB\-p\fR, but without \fB\-N\fR
or \fB\-nc\fR, re-downloading a file will result in the new copy
simply overwriting the old.  Adding \fB\-nc\fR will prevent this
behavior, instead causing the original version to be preserved and any
newer copies on the server to be ignored.
.Sp
When running Wget with \fB\-N\fR, with or without \fB\-r\fR or
\&\fB\-p\fR, the decision as to whether or not to download a newer copy
of a file depends on the local and remote timestamp and size of the
file.  \fB\-nc\fR may not be specified at the
same time as \fB\-N\fR.
.Sp
Note that when \fB\-nc\fR is specified, files with the suffixes
\&\fB.html\fR or \fB.htm\fR will be loaded from the local disk and
parsed as if they had been retrieved from the Web.
.IP "\fB\-c\fR" 4
.IX Item "-c"
.PD 0
.IP "\fB\-\-continue\fR" 4
.IX Item "--continue"
.PD
Continue getting a partially-downloaded file.  This is useful when you
want to finish up a download started by a previous instance of Wget, or
by another program.  For instance:
.Sp
.Vb 1
\&        wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
.Ve
.Sp
If there is a file named \fIls\-lR.Z\fR in the current directory, Wget
will assume that it is the first portion of the remote file, and will
ask the server to continue the retrieval from an offset equal to the
length of the local file.
.Sp
Note that you don't need to specify this option if you just want the
current invocation of Wget to retry downloading a file should the
connection be lost midway through.  This is the default behavior.
\&\fB\-c\fR only affects resumption of downloads started \fIprior\fR to
this invocation of Wget, and whose local files are still sitting around.
.Sp
Without \fB\-c\fR, the previous example would just download the remote
file to \fIls\-lR.Z.1\fR, leaving the truncated \fIls\-lR.Z\fR file
alone.
.Sp
Beginning with Wget 1.7, if you use \fB\-c\fR on a non-empty file, and
it turns out that the server does not support continued downloading,
Wget will refuse to start the download from scratch, which would
effectively ruin existing contents.  If you really want the download to
start from scratch, remove the file.
.Sp
Also beginning with Wget 1.7, if you use \fB\-c\fR on a file which is of
equal size as the one on the server, Wget will refuse to download the
file and print an explanatory message.  The same happens when the file
is smaller on the server than locally (presumably because it was changed
on the server since your last download attempt)\-\-\-because \*(L"continuing\*(R"
is not meaningful, no download occurs.
.Sp
On the other side of the coin, while using \fB\-c\fR, any file that's
bigger on the server than locally will be considered an incomplete
download and only \f(CW\*(C`(length(remote) \- length(local))\*(C'\fR bytes will be
downloaded and tacked onto the end of the local file.  This behavior can
be desirable in certain cases\-\-\-for instance, you can use \fBwget \-c\fR
to download just the new portion that's been appended to a data
collection or log file.
.Sp
However, if the file is bigger on the server because it's been
\&\fIchanged\fR, as opposed to just \fIappended\fR to, you'll end up
with a garbled file.  Wget has no way of verifying that the local file
is really a valid prefix of the remote file.  You need to be especially
careful of this when using \fB\-c\fR in conjunction with \fB\-r\fR,
since every file will be considered as an \*(L"incomplete download\*(R" candidate.
.Sp
Another instance where you'll get a garbled file if you try to use
\&\fB\-c\fR is if you have a lame \s-1HTTP\s0 proxy that inserts a
\&\*(L"transfer interrupted\*(R" string into the local file.  In the future a
\&\*(L"rollback\*(R" option may be added to deal with this case.
.Sp
Note that \fB\-c\fR only works with \s-1FTP\s0 servers and with \s-1HTTP\s0
servers that support the \f(CW\*(C`Range\*(C'\fR header.
.IP "\fB\-\-progress=\fR\fItype\fR" 4
.IX Item "--progress=type"
Select the type of the progress indicator you wish to use.  Legal
indicators are \*(L"dot\*(R" and \*(L"bar\*(R".
.Sp
The \*(L"bar\*(R" indicator is used by default.  It draws an \s-1ASCII\s0 progress
bar graphics (a.k.a \*(L"thermometer\*(R" display) indicating the status of
retrieval.  If the output is not a \s-1TTY\s0, the \*(L"dot\*(R" bar will be used by
default.
.Sp
Use \fB\-\-progress=dot\fR to switch to the \*(L"dot\*(R" display.  It traces
the retrieval by printing dots on the screen, each dot representing a
fixed amount of downloaded data.
.Sp
When using the dotted retrieval, you may also set the \fIstyle\fR by
specifying the type as \fBdot:\fR\fIstyle\fR.  Different styles assign
different meaning to one dot.  With the \f(CW\*(C`default\*(C'\fR style each dot
represents 1K, there are ten dots in a cluster and 50 dots in a line.
The \f(CW\*(C`binary\*(C'\fR style has a more \*(L"computer\*(R"\-like orientation\-\-\-8K
dots, 16\-dots clusters and 48 dots per line (which makes for 384K
lines).  The \f(CW\*(C`mega\*(C'\fR style is suitable for downloading very large
files\-\-\-each dot represents 64K retrieved, there are eight dots in a
cluster, and 48 dots on each line (so each line contains 3M).
.Sp
Note that you can set the default style using the \f(CW\*(C`progress\*(C'\fR
command in \fI.wgetrc\fR.  That setting may be overridden from the
command line.  The exception is that, when the output is not a \s-1TTY\s0, the
\&\*(L"dot\*(R" progress will be favored over \*(L"bar\*(R".  To force the bar output,
use \fB\-\-progress=bar:force\fR.
.IP "\fB\-N\fR" 4
.IX Item "-N"
.PD 0
.IP "\fB\-\-timestamping\fR" 4
.IX Item "--timestamping"
.PD
Turn on time\-stamping.  
.IP "\fB\-S\fR" 4
.IX Item "-S"
.PD 0
.IP "\fB\-\-server\-response\fR" 4
.IX Item "--server-response"
.PD
Print the headers sent by \s-1HTTP\s0 servers and responses sent by
\&\s-1FTP\s0 servers.
.IP "\fB\-\-spider\fR" 4
.IX Item "--spider"
When invoked with this option, Wget will behave as a Web \fIspider\fR,
which means that it will not download the pages, just check that they
are there.  For example, you can use Wget to check your bookmarks:
.Sp
.Vb 1
\&        wget --spider --force-html -i bookmarks.html
.Ve
.Sp
This feature needs much more work for Wget to get close to the
functionality of real web spiders.
.IP "\fB\-T seconds\fR" 4
.IX Item "-T seconds"
.PD 0
.IP "\fB\-\-timeout=\fR\fIseconds\fR" 4
.IX Item "--timeout=seconds"
.PD
Set the network timeout to \fIseconds\fR seconds.  This is equivalent
to specifying \fB\-\-dns\-timeout\fR, \fB\-\-connect\-timeout\fR, and
\&\fB\-\-read\-timeout\fR, all at the same time.
.Sp
When interacting with the network, Wget can check for timeout and
abort the operation if it takes too long.  This prevents anomalies
like hanging reads and infinite connects.  The only timeout enabled by
default is a 900\-second read timeout.  Setting a timeout to 0 disables
it altogether.  Unless you know what you are doing, it is best not to
change the default timeout settings.
.Sp
All timeout-related options accept decimal values, as well as
subsecond values.  For example, \fB0.1\fR seconds is a legal (though
unwise) choice of timeout.  Subsecond timeouts are useful for checking
server response times or for testing network latency.
.IP "\fB\-\-dns\-timeout=\fR\fIseconds\fR" 4
.IX Item "--dns-timeout=seconds"
Set the \s-1DNS\s0 lookup timeout to \fIseconds\fR seconds.  \s-1DNS\s0 lookups that
don't complete within the specified time will fail.  By default, there
is no timeout on \s-1DNS\s0 lookups, other than that implemented by system
libraries.
.IP "\fB\-\-connect\-timeout=\fR\fIseconds\fR" 4