wget.1

Wget很好的处理了http和ftp的下载,很值得学习的经典代码

.IX Item "--connect-timeout=seconds"
Set the connect timeout to \fIseconds\fR seconds.  \s-1TCP\s0 connections that
take longer to establish will be aborted.  By default, there is no
connect timeout, other than that implemented by system libraries.
.IP "\fB\-\-read\-timeout=\fR\fIseconds\fR" 4
.IX Item "--read-timeout=seconds"
Set the read (and write) timeout to \fIseconds\fR seconds.  The
\&\*(L"time\*(R" of this timeout refers to \fIidle time\fR: if, at any point in
the download, no data is received for more than the specified number
of seconds, reading fails and the download is restarted.  This option
does not directly affect the duration of the entire download.
.Sp
Of course, the remote server may choose to terminate the connection
sooner than this option requires.  The default read timeout is 900
seconds.
.IP "\fB\-\-limit\-rate=\fR\fIamount\fR" 4
.IX Item "--limit-rate=amount"
Limit the download speed to \fIamount\fR bytes per second.  Amount may
be expressed in bytes, kilobytes with the \fBk\fR suffix, or megabytes
with the \fBm\fR suffix.  For example, \fB\-\-limit\-rate=20k\fR will
limit the retrieval rate to 20KB/s.  This is useful when, for whatever
reason, you don't want Wget to consume the entire available bandwidth.
.Sp
This option allows the use of decimal numbers, usually in conjunction
with power suffixes; for example, \fB\-\-limit\-rate=2.5k\fR is a legal
value.
.Sp
Note that Wget implements the limiting by sleeping the appropriate
amount of time after a network read that took less time than specified
by the rate.  Eventually this strategy causes the \s-1TCP\s0 transfer to slow
down to approximately the specified rate.  However, it may take some
time for this balance to be achieved, so don't be surprised if limiting
the rate doesn't work well with very small files.
.IP "\fB\-w\fR \fIseconds\fR" 4
.IX Item "-w seconds"
.PD 0
.IP "\fB\-\-wait=\fR\fIseconds\fR" 4
.IX Item "--wait=seconds"
.PD
Wait the specified number of seconds between the retrievals.  Use of
this option is recommended, as it lightens the server load by making the
requests less frequent.  Instead of in seconds, the time can be
specified in minutes using the \f(CW\*(C`m\*(C'\fR suffix, in hours using \f(CW\*(C`h\*(C'\fR
suffix, or in days using \f(CW\*(C`d\*(C'\fR suffix.
.Sp
Specifying a large value for this option is useful if the network or the
destination host is down, so that Wget can wait long enough to
reasonably expect the network error to be fixed before the retry.  The
waiting interval specified by this function is influenced by
\&\f(CW\*(C`\-\-random\-wait\*(C'\fR, which see.
.IP "\fB\-\-waitretry=\fR\fIseconds\fR" 4
.IX Item "--waitretry=seconds"
If you don't want Wget to wait between \fIevery\fR retrieval, but only
between retries of failed downloads, you can use this option.  Wget will
use \fIlinear backoff\fR, waiting 1 second after the first failure on a
given file, then waiting 2 seconds after the second failure on that
file, up to the maximum number of \fIseconds\fR you specify.  Therefore,
a value of 10 will actually make Wget wait up to (1 + 2 + ... + 10) = 55
seconds per file.
.Sp
Note that this option is turned on by default in the global
\&\fIwgetrc\fR file.
.IP "\fB\-\-random\-wait\fR" 4
.IX Item "--random-wait"
Some web sites may perform log analysis to identify retrieval programs
such as Wget by looking for statistically significant similarities in
the time between requests. This option causes the time between requests
to vary between 0.5 and 1.5 * \fIwait\fR seconds, where \fIwait\fR was
specified using the \fB\-\-wait\fR option, in order to mask Wget's
presence from such analysis.
.Sp
A 2001 article in a publication devoted to development on a popular
consumer platform provided code to perform this analysis on the fly.
Its author suggested blocking at the class C address level to ensure
automated retrieval programs were blocked despite changing DHCP-supplied
addresses.
.Sp
The \fB\-\-random\-wait\fR option was inspired by this ill-advised
recommendation to block many unrelated users from a web site due to the
actions of one.
.IP "\fB\-\-no\-proxy\fR" 4
.IX Item "--no-proxy"
Don't use proxies, even if the appropriate \f(CW*_proxy\fR environment
variable is defined.
.IP "\fB\-Q\fR \fIquota\fR" 4
.IX Item "-Q quota"
.PD 0
.IP "\fB\-\-quota=\fR\fIquota\fR" 4
.IX Item "--quota=quota"
.PD
Specify download quota for automatic retrievals.  The value can be
specified in bytes (default), kilobytes (with \fBk\fR suffix), or
megabytes (with \fBm\fR suffix).
.Sp
Note that quota will never affect downloading a single file.  So if you
specify \fBwget \-Q10k ftp://wuarchive.wustl.edu/ls\-lR.gz\fR, all of the
\&\fIls\-lR.gz\fR will be downloaded.  The same goes even when several
URLs are specified on the command\-line.  However, quota is
respected when retrieving either recursively, or from an input file.
Thus you may safely type \fBwget \-Q2m \-i sites\fR\-\-\-download will be
aborted when the quota is exceeded.
.Sp
Setting quota to 0 or to \fBinf\fR unlimits the download quota.
.IP "\fB\-\-no\-dns\-cache\fR" 4
.IX Item "--no-dns-cache"
Turn off caching of \s-1DNS\s0 lookups.  Normally, Wget remembers the \s-1IP\s0
addresses it looked up from \s-1DNS\s0 so it doesn't have to repeatedly
contact the \s-1DNS\s0 server for the same (typically small) set of hosts it
retrieves from.  This cache exists in memory only; a new Wget run will
contact \s-1DNS\s0 again.
.Sp
However, it has been reported that in some situations it is not
desirable to cache host names, even for the duration of a
short-running application like Wget.  With this option Wget issues a
new \s-1DNS\s0 lookup (more precisely, a new call to \f(CW\*(C`gethostbyname\*(C'\fR or
\&\f(CW\*(C`getaddrinfo\*(C'\fR) each time it makes a new connection.  Please note
that this option will \fInot\fR affect caching that might be
performed by the resolving library or by an external caching layer,
such as \s-1NSCD\s0.
.Sp
If you don't understand exactly what this option does, you probably
won't need it.
.IP "\fB\-\-restrict\-file\-names=\fR\fImode\fR" 4
.IX Item "--restrict-file-names=mode"
Change which characters found in remote URLs may show up in local file
names generated from those URLs.  Characters that are \fIrestricted\fR
by this option are escaped, i.e. replaced with \fB%HH\fR, where
\&\fB\s-1HH\s0\fR is the hexadecimal number that corresponds to the restricted
character.
.Sp
By default, Wget escapes the characters that are not valid as part of
file names on your operating system, as well as control characters that
are typically unprintable.  This option is useful for changing these
defaults, either because you are downloading to a non-native partition,
or because you want to disable escaping of the control characters.
.Sp
When mode is set to \*(L"unix\*(R", Wget escapes the character \fB/\fR and
the control characters in the ranges 0\-\-31 and 128\-\-159.  This is the
default on Unix-like \s-1OS\s0'es.
.Sp
When mode is set to \*(L"windows\*(R", Wget escapes the characters \fB\e\fR,
\&\fB|\fR, \fB/\fR, \fB:\fR, \fB?\fR, \fB"\fR, \fB*\fR, \fB<\fR,
\&\fB>\fR, and the control characters in the ranges 0\-\-31 and 128\-\-159.
In addition to this, Wget in Windows mode uses \fB+\fR instead of
\&\fB:\fR to separate host and port in local file names, and uses
\&\fB@\fR instead of \fB?\fR to separate the query portion of the file
name from the rest.  Therefore, a \s-1URL\s0 that would be saved as
\&\fBwww.xemacs.org:4300/search.pl?input=blah\fR in Unix mode would be
saved as \fBwww.xemacs.org+4300/search.pl@input=blah\fR in Windows
mode.  This mode is the default on Windows.
.Sp
If you append \fB,nocontrol\fR to the mode, as in
\&\fBunix,nocontrol\fR, escaping of the control characters is also
switched off.  You can use \fB\-\-restrict\-file\-names=nocontrol\fR to
turn off escaping of control characters without affecting the choice of
the \s-1OS\s0 to use as file name restriction mode.
.IP "\fB\-4\fR" 4
.IX Item "-4"
.PD 0
.IP "\fB\-\-inet4\-only\fR" 4
.IX Item "--inet4-only"
.IP "\fB\-6\fR" 4
.IX Item "-6"
.IP "\fB\-\-inet6\-only\fR" 4
.IX Item "--inet6-only"
.PD
Force connecting to IPv4 or IPv6 addresses.  With \fB\-\-inet4\-only\fR
or \fB\-4\fR, Wget will only connect to IPv4 hosts, ignoring \s-1AAAA\s0
records in \s-1DNS\s0, and refusing to connect to IPv6 addresses specified in
URLs.  Conversely, with \fB\-\-inet6\-only\fR or \fB\-6\fR, Wget will
only connect to IPv6 hosts and ignore A records and IPv4 addresses.
.Sp
Neither options should be needed normally.  By default, an IPv6\-aware
Wget will use the address family specified by the host's \s-1DNS\s0 record.
If the \s-1DNS\s0 responds with both IPv4 and IPv6 addresses, Wget will try
them in sequence until it finds one it can connect to.  (Also see
\&\f(CW\*(C`\-\-prefer\-family\*(C'\fR option described below.)
.Sp
These options can be used to deliberately force the use of IPv4 or
IPv6 address families on dual family systems, usually to aid debugging
or to deal with broken network configuration.  Only one of
\&\fB\-\-inet6\-only\fR and \fB\-\-inet4\-only\fR may be specified at the
same time.  Neither option is available in Wget compiled without IPv6
support.
.IP "\fB\-\-prefer\-family=IPv4/IPv6/none\fR" 4
.IX Item "--prefer-family=IPv4/IPv6/none"
When given a choice of several addresses, connect to the addresses
with specified address family first.  IPv4 addresses are preferred by
default.
.Sp
This avoids spurious errors and connect attempts when accessing hosts
that resolve to both IPv6 and IPv4 addresses from IPv4 networks.  For
example, \fBwww.kame.net\fR resolves to
\&\fB2001:200:0:8002:203:47ff:fea5:3085\fR and to
\&\fB203.178.141.194\fR.  When the preferred family is \f(CW\*(C`IPv4\*(C'\fR, the
IPv4 address is used first; when the preferred family is \f(CW\*(C`IPv6\*(C'\fR,
the IPv6 address is used first; if the specified value is \f(CW\*(C`none\*(C'\fR,
the address order returned by \s-1DNS\s0 is used without change.
.Sp
Unlike \fB\-4\fR and \fB\-6\fR, this option doesn't inhibit access to
any address family, it only changes the \fIorder\fR in which the
addresses are accessed.  Also note that the reordering performed by
this option is \fIstable\fR\-\-\-it doesn't affect order of addresses of
the same family.  That is, the relative order of all IPv4 addresses
and of all IPv6 addresses remains intact in all cases.
.IP "\fB\-\-retry\-connrefused\fR" 4
.IX Item "--retry-connrefused"
Consider \*(L"connection refused\*(R" a transient error and try again.
Normally Wget gives up on a \s-1URL\s0 when it is unable to connect to the
site because failure to connect is taken as a sign that the server is
not running at all and that retries would not help.  This option is
for mirroring unreliable sites whose servers tend to disappear for
short periods of time.
.IP "\fB\-\-user=\fR\fIuser\fR" 4
.IX Item "--user=user"
.PD 0
.IP "\fB\-\-password=\fR\fIpassword\fR" 4
.IX Item "--password=password"
.PD
Specify the username \fIuser\fR and password \fIpassword\fR for both
\&\s-1FTP\s0 and \s-1HTTP\s0 file retrieval.  These parameters can be overridden
using the \fB\-\-ftp\-user\fR and \fB\-\-ftp\-password\fR options for 
\&\s-1FTP\s0 connections and the \fB\-\-http\-user\fR and \fB\-\-http\-password\fR 
options for \s-1HTTP\s0 connections.
.Sh "Directory Options"
.IX Subsection "Directory Options"
.IP "\fB\-nd\fR" 4
.IX Item "-nd"
.PD 0
.IP "\fB\-\-no\-directories\fR" 4
.IX Item "--no-directories"
.PD
Do not create a hierarchy of directories when retrieving recursively.
With this option turned on, all files will get saved to the current
directory, without clobbering (if a name shows up more than once, the
filenames will get extensions \fB.n\fR).
.IP "\fB\-x\fR" 4
.IX Item "-x"
.PD 0
.IP "\fB\-\-force\-directories\fR" 4
.IX Item "--force-directories"
.PD
The opposite of \fB\-nd\fR\-\-\-create a hierarchy of directories, even if
one would not have been created otherwise.  E.g. \fBwget \-x
http://fly.srk.fer.hr/robots.txt\fR will save the downloaded file to
\&\fIfly.srk.fer.hr/robots.txt\fR.
.IP "\fB\-nH\fR" 4
.IX Item "-nH"
.PD 0
.IP "\fB\-\-no\-host\-directories\fR" 4
.IX Item "--no-host-directories"
.PD
Disable generation of host-prefixed directories.  By default, invoking
Wget with \fB\-r http://fly.srk.fer.hr/\fR will create a structure of
directories beginning with \fIfly.srk.fer.hr/\fR.  This option disables
such behavior.
.IP "\fB\-\-protocol\-directories\fR" 4
.IX Item "--protocol-directories"
Use the protocol name as a directory component of local file names.  For
example, with this option, \fBwget \-r http://\fR\fIhost\fR will save to
\&\fBhttp/\fR\fIhost\fR\fB/...\fR rather than just to \fIhost\fR\fB/...\fR.
.IP "\fB\-\-cut\-dirs=\fR\fInumber\fR" 4
.IX Item "--cut-dirs=number"
Ignore \fInumber\fR directory components.  This is useful for getting a
fine-grained control over the directory where recursive retrieval will
be saved.
.Sp
Take, for example, the directory at
\&\fBftp://ftp.xemacs.org/pub/xemacs/\fR.  If you retrieve it with
\&\fB\-r\fR, it will be saved locally under
\&\fIftp.xemacs.org/pub/xemacs/\fR.  While the \fB\-nH\fR option can
remove the \fIftp.xemacs.org/\fR part, you are still stuck with
\&\fIpub/xemacs\fR.  This is where \fB\-\-cut\-dirs\fR comes in handy; it
makes Wget not \*(L"see\*(R" \fInumber\fR remote directory components.  Here
are several examples of how \fB\-\-cut\-dirs\fR option works.
.Sp
.Vb 4
\&        No options        -> ftp.xemacs.org/pub/xemacs/
\&        -nH               -> pub/xemacs/
\&        -nH --cut-dirs=1  -> xemacs/
\&        -nH --cut-dirs=2  -> .
.Ve
.Sp
.Vb 2
\&        --cut-dirs=1      -> ftp.xemacs.org/xemacs/
\&        ...
.Ve
.Sp
If you just want to get rid of the directory structure, this option is
similar to a combination of \fB\-nd\fR and \fB\-P\fR.  However, unlike
\&\fB\-nd\fR, \fB\-\-cut\-dirs\fR does not lose with subdirectories\-\-\-for
instance, with \fB\-nH \-\-cut\-dirs=1\fR, a \fIbeta/\fR subdirectory will
be placed to \fIxemacs/beta\fR, as one would expect.
.IP "\fB\-P\fR \fIprefix\fR" 4
.IX Item "-P prefix"