wget.1

Wget很好的处理了http和ftp的下载,很值得学习的经典代码

downloaded file should be.
.IP "\fB\-\-auth\-no\-challenge\fR" 4
.IX Item "--auth-no-challenge"
If this option is given, Wget will send Basic \s-1HTTP\s0 authentication
information (plaintext username and password) for all requests, just
like Wget 1.10.2 and prior did by default.
.Sp
Use of this option is not recommended, and is intended only to support
some few obscure servers, which never send \s-1HTTP\s0 authentication
challenges, but accept unsolicited auth info, say, in addition to
form-based authentication.
.Sh "\s-1HTTPS\s0 (\s-1SSL/TLS\s0) Options"
.IX Subsection "HTTPS (SSL/TLS) Options"
To support encrypted \s-1HTTP\s0 (\s-1HTTPS\s0) downloads, Wget must be compiled
with an external \s-1SSL\s0 library, currently OpenSSL.  If Wget is compiled
without \s-1SSL\s0 support, none of these options are available.
.IP "\fB\-\-secure\-protocol=\fR\fIprotocol\fR" 4
.IX Item "--secure-protocol=protocol"
Choose the secure protocol to be used.  Legal values are \fBauto\fR,
\&\fBSSLv2\fR, \fBSSLv3\fR, and \fBTLSv1\fR.  If \fBauto\fR is used,
the \s-1SSL\s0 library is given the liberty of choosing the appropriate
protocol automatically, which is achieved by sending an SSLv2 greeting
and announcing support for SSLv3 and TLSv1.  This is the default.
.Sp
Specifying \fBSSLv2\fR, \fBSSLv3\fR, or \fBTLSv1\fR forces the use
of the corresponding protocol.  This is useful when talking to old and
buggy \s-1SSL\s0 server implementations that make it hard for OpenSSL to
choose the correct protocol version.  Fortunately, such servers are
quite rare.
.IP "\fB\-\-no\-check\-certificate\fR" 4
.IX Item "--no-check-certificate"
Don't check the server certificate against the available certificate
authorities.  Also don't require the \s-1URL\s0 host name to match the common
name presented by the certificate.
.Sp
As of Wget 1.10, the default is to verify the server's certificate
against the recognized certificate authorities, breaking the \s-1SSL\s0
handshake and aborting the download if the verification fails.
Although this provides more secure downloads, it does break
interoperability with some sites that worked with previous Wget
versions, particularly those using self\-signed, expired, or otherwise
invalid certificates.  This option forces an \*(L"insecure\*(R" mode of
operation that turns the certificate verification errors into warnings
and allows you to proceed.
.Sp
If you encounter \*(L"certificate verification\*(R" errors or ones saying
that \*(L"common name doesn't match requested host name\*(R", you can use
this option to bypass the verification and proceed with the download.
\&\fIOnly use this option if you are otherwise convinced of the
site's authenticity, or if you really don't care about the validity of
its certificate.\fR  It is almost always a bad idea not to check the
certificates when transmitting confidential or important data.
.IP "\fB\-\-certificate=\fR\fIfile\fR" 4
.IX Item "--certificate=file"
Use the client certificate stored in \fIfile\fR.  This is needed for
servers that are configured to require certificates from the clients
that connect to them.  Normally a certificate is not required and this
switch is optional.
.IP "\fB\-\-certificate\-type=\fR\fItype\fR" 4
.IX Item "--certificate-type=type"
Specify the type of the client certificate.  Legal values are
\&\fB\s-1PEM\s0\fR (assumed by default) and \fB\s-1DER\s0\fR, also known as
\&\fB\s-1ASN1\s0\fR.
.IP "\fB\-\-private\-key=\fR\fIfile\fR" 4
.IX Item "--private-key=file"
Read the private key from \fIfile\fR.  This allows you to provide the
private key in a file separate from the certificate.
.IP "\fB\-\-private\-key\-type=\fR\fItype\fR" 4
.IX Item "--private-key-type=type"
Specify the type of the private key.  Accepted values are \fB\s-1PEM\s0\fR
(the default) and \fB\s-1DER\s0\fR.
.IP "\fB\-\-ca\-certificate=\fR\fIfile\fR" 4
.IX Item "--ca-certificate=file"
Use \fIfile\fR as the file with the bundle of certificate authorities
(\*(L"\s-1CA\s0\*(R") to verify the peers.  The certificates must be in \s-1PEM\s0 format.
.Sp
Without this option Wget looks for \s-1CA\s0 certificates at the
system-specified locations, chosen at OpenSSL installation time.
.IP "\fB\-\-ca\-directory=\fR\fIdirectory\fR" 4
.IX Item "--ca-directory=directory"
Specifies directory containing \s-1CA\s0 certificates in \s-1PEM\s0 format.  Each
file contains one \s-1CA\s0 certificate, and the file name is based on a hash
value derived from the certificate.  This is achieved by processing a
certificate directory with the \f(CW\*(C`c_rehash\*(C'\fR utility supplied with
OpenSSL.  Using \fB\-\-ca\-directory\fR is more efficient than
\&\fB\-\-ca\-certificate\fR when many certificates are installed because
it allows Wget to fetch certificates on demand.
.Sp
Without this option Wget looks for \s-1CA\s0 certificates at the
system-specified locations, chosen at OpenSSL installation time.
.IP "\fB\-\-random\-file=\fR\fIfile\fR" 4
.IX Item "--random-file=file"
Use \fIfile\fR as the source of random data for seeding the
pseudo-random number generator on systems without \fI/dev/random\fR.
.Sp
On such systems the \s-1SSL\s0 library needs an external source of randomness
to initialize.  Randomness may be provided by \s-1EGD\s0 (see
\&\fB\-\-egd\-file\fR below) or read from an external source specified by
the user.  If this option is not specified, Wget looks for random data
in \f(CW$RANDFILE\fR or, if that is unset, in \fI$HOME/.rnd\fR.  If
none of those are available, it is likely that \s-1SSL\s0 encryption will not
be usable.
.Sp
If you're getting the \*(L"Could not seed OpenSSL \s-1PRNG\s0; disabling \s-1SSL\s0.\*(R" 
error, you should provide random data using some of the methods
described above.
.IP "\fB\-\-egd\-file=\fR\fIfile\fR" 4
.IX Item "--egd-file=file"
Use \fIfile\fR as the \s-1EGD\s0 socket.  \s-1EGD\s0 stands for \fIEntropy
Gathering Daemon\fR, a user-space program that collects data from
various unpredictable system sources and makes it available to other
programs that might need it.  Encryption software, such as the \s-1SSL\s0
library, needs sources of non-repeating randomness to seed the random
number generator used to produce cryptographically strong keys.
.Sp
OpenSSL allows the user to specify his own source of entropy using the
\&\f(CW\*(C`RAND_FILE\*(C'\fR environment variable.  If this variable is unset, or
if the specified file does not produce enough randomness, OpenSSL will
read random data from \s-1EGD\s0 socket specified using this option.
.Sp
If this option is not specified (and the equivalent startup command is
not used), \s-1EGD\s0 is never contacted.  \s-1EGD\s0 is not needed on modern Unix
systems that support \fI/dev/random\fR.
.Sh "\s-1FTP\s0 Options"
.IX Subsection "FTP Options"
.IP "\fB\-\-ftp\-user=\fR\fIuser\fR" 4
.IX Item "--ftp-user=user"
.PD 0
.IP "\fB\-\-ftp\-password=\fR\fIpassword\fR" 4
.IX Item "--ftp-password=password"
.PD
Specify the username \fIuser\fR and password \fIpassword\fR on an
\&\s-1FTP\s0 server.  Without this, or the corresponding startup option, 
the password defaults to \fB\-wget@\fR, normally used for anonymous 
\&\s-1FTP\s0.
.Sp
Another way to specify username and password is in the \s-1URL\s0 itself.  Either method reveals your password to anyone who
bothers to run \f(CW\*(C`ps\*(C'\fR.  To prevent the passwords from being seen,
store them in \fI.wgetrc\fR or \fI.netrc\fR, and make sure to protect
those files from other users with \f(CW\*(C`chmod\*(C'\fR.  If the passwords are
really important, do not leave them lying in those files either\-\-\-edit
the files and delete them after Wget has started the download.
.IP "\fB\-\-no\-remove\-listing\fR" 4
.IX Item "--no-remove-listing"
Don't remove the temporary \fI.listing\fR files generated by \s-1FTP\s0
retrievals.  Normally, these files contain the raw directory listings
received from \s-1FTP\s0 servers.  Not removing them can be useful for
debugging purposes, or when you want to be able to easily check on the
contents of remote server directories (e.g. to verify that a mirror
you're running is complete).
.Sp
Note that even though Wget writes to a known filename for this file,
this is not a security hole in the scenario of a user making
\&\fI.listing\fR a symbolic link to \fI/etc/passwd\fR or something and
asking \f(CW\*(C`root\*(C'\fR to run Wget in his or her directory.  Depending on
the options used, either Wget will refuse to write to \fI.listing\fR,
making the globbing/recursion/time\-stamping operation fail, or the
symbolic link will be deleted and replaced with the actual
\&\fI.listing\fR file, or the listing will be written to a
\&\fI.listing.\fInumber\fI\fR file.
.Sp
Even though this situation isn't a problem, though, \f(CW\*(C`root\*(C'\fR should
never run Wget in a non-trusted user's directory.  A user could do
something as simple as linking \fIindex.html\fR to \fI/etc/passwd\fR
and asking \f(CW\*(C`root\*(C'\fR to run Wget with \fB\-N\fR or \fB\-r\fR so the file
will be overwritten.
.IP "\fB\-\-no\-glob\fR" 4
.IX Item "--no-glob"
Turn off \s-1FTP\s0 globbing.  Globbing refers to the use of shell-like
special characters (\fIwildcards\fR), like \fB*\fR, \fB?\fR, \fB[\fR
and \fB]\fR to retrieve more than one file from the same directory at
once, like:
.Sp
.Vb 1
\&        wget ftp://gnjilux.srk.fer.hr/*.msg
.Ve
.Sp
By default, globbing will be turned on if the \s-1URL\s0 contains a
globbing character.  This option may be used to turn globbing on or off
permanently.
.Sp
You may have to quote the \s-1URL\s0 to protect it from being expanded by
your shell.  Globbing makes Wget look for a directory listing, which is
system\-specific.  This is why it currently works only with Unix \s-1FTP\s0
servers (and the ones emulating Unix \f(CW\*(C`ls\*(C'\fR output).
.IP "\fB\-\-no\-passive\-ftp\fR" 4
.IX Item "--no-passive-ftp"
Disable the use of the \fIpassive\fR \s-1FTP\s0 transfer mode.  Passive \s-1FTP\s0
mandates that the client connect to the server to establish the data
connection rather than the other way around.
.Sp
If the machine is connected to the Internet directly, both passive and
active \s-1FTP\s0 should work equally well.  Behind most firewall and \s-1NAT\s0
configurations passive \s-1FTP\s0 has a better chance of working.  However,
in some rare firewall configurations, active \s-1FTP\s0 actually works when
passive \s-1FTP\s0 doesn't.  If you suspect this to be the case, use this
option, or set \f(CW\*(C`passive_ftp=off\*(C'\fR in your init file.
.IP "\fB\-\-retr\-symlinks\fR" 4
.IX Item "--retr-symlinks"
Usually, when retrieving \s-1FTP\s0 directories recursively and a symbolic
link is encountered, the linked-to file is not downloaded.  Instead, a
matching symbolic link is created on the local filesystem.  The
pointed-to file will not be downloaded unless this recursive retrieval
would have encountered it separately and downloaded it anyway.
.Sp
When \fB\-\-retr\-symlinks\fR is specified, however, symbolic links are
traversed and the pointed-to files are retrieved.  At this time, this
option does not cause Wget to traverse symlinks to directories and
recurse through them, but in the future it should be enhanced to do
this.
.Sp
Note that when retrieving a file (not a directory) because it was
specified on the command\-line, rather than because it was recursed to,
this option has no effect.  Symbolic links are always traversed in this
case.
.IP "\fB\-\-no\-http\-keep\-alive\fR" 4
.IX Item "--no-http-keep-alive"
Turn off the \*(L"keep\-alive\*(R" feature for \s-1HTTP\s0 downloads.  Normally, Wget
asks the server to keep the connection open so that, when you download
more than one document from the same server, they get transferred over
the same \s-1TCP\s0 connection.  This saves time and at the same time reduces
the load on the server.
.Sp
This option is useful when, for some reason, persistent (keep\-alive)
connections don't work for you, for example due to a server bug or due
to the inability of server-side scripts to cope with the connections.
.Sh "Recursive Retrieval Options"
.IX Subsection "Recursive Retrieval Options"
.IP "\fB\-r\fR" 4
.IX Item "-r"
.PD 0
.IP "\fB\-\-recursive\fR" 4
.IX Item "--recursive"
.PD
Turn on recursive retrieving.  
.IP "\fB\-l\fR \fIdepth\fR" 4
.IX Item "-l depth"
.PD 0
.IP "\fB\-\-level=\fR\fIdepth\fR" 4
.IX Item "--level=depth"
.PD
Specify recursion maximum depth level \fIdepth\fR.  The default maximum depth is 5.
.IP "\fB\-\-delete\-after\fR" 4
.IX Item "--delete-after"
This option tells Wget to delete every single file it downloads,
\&\fIafter\fR having done so.  It is useful for pre-fetching popular
pages through a proxy, e.g.:
.Sp
.Vb 1
\&        wget -r -nd --delete-after http://whatever.com/~popular/page/
.Ve
.Sp
The \fB\-r\fR option is to retrieve recursively, and \fB\-nd\fR to not
create directories.  
.Sp
Note that \fB\-\-delete\-after\fR deletes files on the local machine.  It
does not issue the \fB\s-1DELE\s0\fR command to remote \s-1FTP\s0 sites, for
instance.  Also note that when \fB\-\-delete\-after\fR is specified,
\&\fB\-\-convert\-links\fR is ignored, so \fB.orig\fR files are simply not
created in the first place.
.IP "\fB\-k\fR" 4
.IX Item "-k"
.PD 0
.IP "\fB\-\-convert\-links\fR" 4
.IX Item "--convert-links"
.PD
After the download is complete, convert the links in the document to
make them suitable for local viewing.  This affects not only the visible
hyperlinks, but any part of the document that links to external content,
such as embedded images, links to style sheets, hyperlinks to non-HTML
content, etc.
.Sp
Each link will be changed in one of the two ways:
.RS 4
.IP "*" 4
The links to files that have been downloaded by Wget will be changed to
refer to the file they point to as a relative link.
.Sp
Example: if the downloaded file \fI/foo/doc.html\fR links to
\&\fI/bar/img.gif\fR, also downloaded, then the link in \fIdoc.html\fR
will be modified to point to \fB../bar/img.gif\fR.  This kind of
transformation works reliably for arbitrary combinations of directories.
.IP "*" 4
The links to files that have not been downloaded by Wget will be changed
to include host name and absolute path of the location they point to.
.Sp
Example: if the downloaded file \fI/foo/doc.html\fR links to
\&\fI/bar/img.gif\fR (or to \fI../bar/img.gif\fR), then the link in
\&\fIdoc.html\fR will be modified to point to
\&\fIhttp://\fIhostname\fI/bar/img.gif\fR.