curl.1

harvest是一个下载html网页得机器人

.\" You can view this file with:
.\" nroff -man curl.1
.\" Written by Daniel Stenberg
.\"
.TH curl 1 "22 Oct 2003" "Curl 7.10.8" "Curl Manual"
.SH NAME
curl \- transfer a URL
.SH SYNOPSIS
.B curl [options]
.I [URL...]
.SH DESCRIPTION
.B curl
is a tool to transfer data from or to a server, using one of the supported
protocols (HTTP, HTTPS, FTP, FTPS, GOPHER, DICT, TELNET, LDAP or FILE). The
command is designed to work without user interaction.

curl offers a busload of useful tricks like proxy support, user
authentication, ftp upload, HTTP post, SSL (https:) connections, cookies, file
transfer resume and more. As you will see below, the amount of features will
make your head spin!

curl is powered by libcurl for all transfer-related features. See
.BR libcurl (3)
for details.
.SH URL
The URL syntax is protocol dependent. You'll find a detailed description in
RFC 2396.

You can specify multiple URLs or parts of URLs by writing part sets within
braces as in:

 http://site.{one,two,three}.com

or you can get sequences of alphanumeric series by using [] as in:

 ftp://ftp.numericals.com/file[1-100].txt
 ftp://ftp.numericals.com/file[001-100].txt    (with leading zeros)
 ftp://ftp.letters.com/file[a-z].txt

No nesting of the sequences is supported at the moment:

 http://www.any.org/archive[1996-1999]/volume[1-4]part{a,b,c,index}.html

You can specify any amount of URLs on the command line. They will be fetched
in a sequential manner in the specified order.

Curl will attempt to re-use connections for multiple file transfers, so that
getting many files from the same server will not do multiple connects /
handshakes. This improves speed. Of course this is only done on files
specified on a single command line and cannot be used between separate curl
invokes.
.SH OPTIONS
.IP "-a/--append"
(FTP) When used in an FTP upload, this will tell curl to append to the target
file instead of overwriting it. If the file doesn't exist, it will be created.

If this option is used twice, the second one will disable append mode again.
.IP "-A/--user-agent <agent string>"
(HTTP)
Specify the User-Agent string to send to the HTTP server. Some badly done CGIs
fail if its not set to "Mozilla/4.0".  To encode blanks in the string,
surround the string with single quote marks.  This can also be set with the
-H/--header flag of course.

If this option is set more than once, the last one will be the one that's
used.
.IP "--anyauth"
(HTTP) Tells curl to figure out authentication method by itself, and use the
most secure one the remote site claims it supports. This is done by first
doing a request and checking the response-headers, thus inducing an extra
network round-trip. This is used instead of setting a specific authentication
method, which you can do with \fI--digest\fP, \fI--ntlm\fP, and
\fI--negotiate\fP. (Added in 7.10.6)

If this option is used several times, the following occurrences make no
difference.
.IP "-b/--cookie <name=data>"
(HTTP)
Pass the data to the HTTP server as a cookie. It is supposedly the
data previously received from the server in a "Set-Cookie:" line.
The data should be in the format "NAME1=VALUE1; NAME2=VALUE2".

If no '=' letter is used in the line, it is treated as a filename to use to
read previously stored cookie lines from, which should be used in this session
if they match. Using this method also activates the "cookie parser" which will
make curl record incoming cookies too, which may be handy if you're using this
in combination with the -L/--location option. The file format of the file to
read cookies from should be plain HTTP headers or the Netscape/Mozilla cookie
file format.

.B NOTE
that the file specified with -b/--cookie is only used as input. No cookies
will be stored in the file. To store cookies, use the -c/--cookie-jar option
or you could even save the HTTP headers to a file using -D/--dump-header!

If this option is set more than once, the last one will be the one that's
used.
.IP "-B/--use-ascii"
Use ASCII transfer when getting an FTP file or LDAP info. For FTP, this can
also be enforced by using an URL that ends with ";type=A". This option causes
data sent to stdout to be in text mode for win32 systems.

If this option is used twice, the second one will disable ASCII usage.
.IP "--basic"
(HTTP) Tells curl to use HTTP Basic authentication. This is the default and
this option is usually pointless, unless you use it to override a previously
set option that sets a different authentication method (such as \fI--ntlm\fP,
\fI--digest\fP and \fI--negotiate\fP). (Added in 7.10.6)

If this option is used several times, the following occurrences make no
difference.
.IP "--ciphers <list of ciphers>"
(SSL) Specifies which ciphers to use in the connection. The list of ciphers
must be using valid ciphers. Read up on SSL cipher list details on this URL:
.I http://www.openssl.org/docs/apps/ciphers.html

If this option is used several times, the last one will override the others.
.IP "--compressed"
(HTTP) Request a compressed response using one of the algorithms libcurl
supports, and return the uncompressed document.  If this option is used and
the server sends an unsupported encoding, Curl will report an error.

If this option is used several times, each occurrence will toggle it on/off.
.IP "--connect-timeout <seconds>"
Maximum time in seconds that you allow the connection to the server to take.
This only limits the connection phase, once curl has connected this option is
of no more use. See also the \fI--max-time\fP option.

If this option is used several times, the last one will be used.
.IP "-c/--cookie-jar <file name>"
Specify to which file you want curl to write all cookies after a completed
operation. Curl writes all cookies previously read from a specified file as
well as all cookies received from remote server(s). If no cookies are known,
no file will be written. The file will be written using the Netscape cookie
file format. If you set the file name to a single dash, "-", the cookies will
be written to stdout.

.B NOTE
If the cookie jar can't be created or written to, the whole curl operation
won't fail or even report an error clearly. Using -v will get a warning
displayed, but that is the only visible feedback you get about this possibly
lethal situation.

If this option is used several times, the last specfied file name will be
used.
.IP "-C/--continue-at <offset>"
Continue/Resume a previous file transfer at the given offset. The given offset
is the exact number of bytes that will be skipped counted from the beginning
of the source file before it is transfered to the destination.  If used with
uploads, the ftp server command SIZE will not be used by curl.

Use "-C -" to tell curl to automatically find out where/how to resume the
transfer. It then uses the given output/input files to figure that out.

If this option is used several times, the last one will be used.
.IP "--create-dirs"
When used in conjunction with the -o option, curl will create the necessary 
local directory hierarchy as needed.
.IP "--crlf"
(FTP) Convert LF to CRLF in upload. Useful for MVS (OS/390).

If this option is used twice, the second will again disable crlf converting.
.IP "-d/--data <data>"
(HTTP) Sends the specified data in a POST request to the HTTP server, in a way
that can emulate as if a user has filled in a HTML form and pressed the submit
button. Note that the data is sent exactly as specified with no extra
processing (with all newlines cut off).  The data is expected to be
\&"url-encoded". This will cause curl to pass the data to the server using the
content-type application/x-www-form-urlencoded. Compare to -F. If more than
one -d/--data option is used on the same command line, the data pieces
specified will be merged together with a separating &-letter. Thus, using '-d
name=daniel -d skill=lousy' would generate a post chunk that looks like
\&'name=daniel&skill=lousy'.

If you start the data with the letter @, the rest should be a file name to
read the data from, or - if you want curl to read the data from stdin.  The
contents of the file must already be url-encoded. Multiple files can also be
specified. Posting data from a file named 'foobar' would thus be done with
\&"--data @foobar".

To post data purely binary, you should instead use the --data-binary option.

-d/--data is the same as --data-ascii.

If this option is used several times, the ones following the first will
append data.
.IP "--data-ascii <data>"
(HTTP) This is an alias for the -d/--data option.

If this option is used several times, the ones following the first will
append data.
.IP "--data-binary <data>"
(HTTP) This posts data in a similar manner as --data-ascii does, although when
using this option the entire context of the posted data is kept as-is. If you
want to post a binary file without the strip-newlines feature of the
--data-ascii option, this is for you.

If this option is used several times, the ones following the first will
append data.
.IP "--digest"
(HTTP) Enables HTTP Digest authentication. This is a authentication that
prevents the password from being sent over the wire in clear text. Use this in
combination with the normal -u/--user option to set user name and
password. See also \fI--ntlm\fP, \fI--negotiate\fP and \fI--anyauth\fP for
related options. (Added in curl 7.10.6)

If this option is used several times, the following occurrences make no
difference.
.IP "--disable-eprt"
(FTP) Tell curl to disable the use of the EPRT and LPRT commands when doing
active FTP transfers. Curl will normally always first attempt to use EPRT,
then LPRT before using PORT, but with this option, it will use PORT right
away. EPRT and LPRT are extensions to the original FTP protocol, may not work
on all servers but enable more functionality in a better way than the
traditional PORT command. (Aded in 7.10.5)

If this option is used several times, each occurrence will toggle this on/off.
.IP "--disable-epsv"
(FTP) Tell curl to disable the use of the EPSV command when doing passive FTP
transfers. Curl will normally always first attempt to use EPSV before PASV,
but with this option, it will not try using EPSV.

If this option is used several times, each occurrence will toggle this on/off.
.IP "-D/--dump-header <file>"
Write the protocol headers to the specified file.

This option is handy to use when you want to store the headers that a HTTP
site sends to you. Cookies from the headers could then be read in a second
curl invoke by using the -b/--cookie option! The -c/--cookie-jar option is
however a better way to store cookies.

When used on FTP, the ftp server response lines are considered being "headers"
and thus are saved there.

If this option is used several times, the last one will be used.
.IP "-e/--referer <URL>"
(HTTP) Sends the "Referer Page" information to the HTTP server. This can also
be set with the -H/--header flag of course.  When used with
.I -L/--location 
you can append ";auto" to the referer URL to make curl automatically set the
previous URL when it follows a Location: header. The ";auto" string can be
used alone, even if you don't set an initial referer.

If this option is used several times, the last one will be used.
.IP "--environment"
(RISC OS ONLY) Sets a range of environment variables, using the names the -w
option supports, to easier allow extraction of useful information after having
run curl.

If this option is used several times, each occurrence will toggle this on/off.
.IP "--egd-file <file>"
(HTTPS) Specify the path name to the Entropy Gathering Daemon socket. The
socket is used to seed the random engine for SSL connections. See also the
.I "--random-file"
option.
.IP "-E/--cert <certificate[:password]>"
(HTTPS)
Tells curl to use the specified certificate file when getting a file
with HTTPS. The certificate must be in PEM format.
If the optional password isn't specified, it will be queried for on
the terminal. Note that this certificate is the private key and the private
certificate concatenated!

If this option is used several times, the last one will be used.
.IP "--cacert <CA certificate>"
(HTTPS) Tells curl to use the specified certificate file to verify the
peer. The file may contain multiple CA certificates. The certificate(s) must
be in PEM format.

curl recognizes the environment variable named 'CURL_CA_BUNDLE' if that is
set, and uses the given path as a path to a CA cert bundle. This option
overrides that variable.

The windows version of curl will automatically look for a CA certs file named
\'curl-ca-bundle.crt\', either in the same directory as curl.exe, or in the
Current Working Directory, or in any folder along your PATH.

If this option is used several times, the last one will be used.
.IP "--capath <CA certificate directory>"
(HTTPS) Tells curl to use the specified certificate directory to verify the
peer. The certificates must be in PEM format, and the directory must have been
processed using the c_rehash utility supplied with openssl. Using --capath can
allow curl to make https connections much more efficiently than using --cacert
if the --cacert file contains many CA certificates.

If this option is used several times, the last one will be used.
.IP "-f/--fail"
(HTTP) Fail silently (no output at all) on server errors. This is mostly done
like this to better enable scripts etc to better deal with failed attempts. In
normal cases when a HTTP server fails to deliver a document, it returns a HTML
document stating so (which often also describes why and more). This flag will
prevent curl from outputting that and fail silently instead.

If this option is used twice, the second will again disable silent failure.
.IP "--ftp-create-dirs"
(FTP) When an FTP URL/operation uses a path that doesn't currently exist on
the server, the standard behaviour of curl is to fail. Using this option, curl
will instead attempt to create missing directories. (Added in 7.10.7)

If this option is used twice, the second will again disable silent failure.
.IP "-F/--form <name=content>"
(HTTP) This lets curl emulate a filled in form in which a user has pressed the
submit button. This causes curl to POST data using the content-type
multipart/form-data according to RFC1867. This enables uploading of binary
files etc. To force the 'content' part to be be a file, prefix the file name
with an @ sign. To just get the content part from a file, prefix the file name
with the letter <. The difference between @ and < is then that @ makes a file
get attached in the post as a file upload, while the < makes a text field and
just get the contents for that text field from a file.

Example, to send your password file to the server, where
\&'password' is the name of the form-field to which /etc/passwd will be the
input:

\fBcurl\fP -F password=@/etc/passwd www.mypasswords.com

To read the file's content from stdin insted of a file, use - where the file
name should've been. This goes for both @ and < constructs.

You can also tell curl what Content-Type to use for the file upload part, by
using 'type=', in a manner similar to:

\fBcurl\fP -F "web=@index.html;type=text/html" url.com

See further examples and details in the MANUAL.

This option can be used multiple times.
.IP "-g/--globoff"
This option switches off the "URL globbing parser". When you set this option,
you can specify URLs that contain the letters {}[] without having them being
interpreted by curl itself. Note that these letters are not normal legal URL
contents but they should be encoded according to the URI standard.
.IP "-G/--get"
When used, this option will make all data specified with -d/--data or
--data-binary to be used in a HTTP GET request instead of the POST request
that otherwise would be used. The data will be appended to the URL with a '?'
separator.

If used in combination with -I, the POST data will instead be appended to the
URL with a HEAD request.

If used multiple times, nothing special happens.
.IP "-h/--help"
Usage help.
.IP "-H/--header <header>"
(HTTP) Extra header to use when getting a web page. You may specify any number
of extra headers. Note that if you should add a custom header that has the
same name as one of the internal ones curl would use, your externally set
header will be used instead of the internal one. This allows you to make even
trickier stuff than curl would normally do. You should not replace internally
set headers without knowing perfectly well what you're doing. Replacing an
internal header with one without content on the right side of the colon will
prevent that header from appearing.

This option can be used multiple times to add/replace/remove multiple headers.
.IP "-i/--include"
(HTTP)
Include the HTTP-header in the output. The HTTP-header includes things
like server-name, date of the document, HTTP-version and more...

If this option is used twice, the second will again disable header include.
.IP "--interface <name>"
Perform an operation using a specified interface. You can enter interface
name, IP address or host name. An example could look like: