curl.1

harvest是一个下载html网页得机器人

URL(s) in a config file.

This option may be used any number of times. To control where this URL is written, use the
.I -o
or the
.I -O
options.
.IP "-v/--verbose"
Makes the fetching more verbose/talkative. Mostly usable for debugging. Lines
starting with '>' means data sent by curl, '<' means data received by curl
that is hidden in normal cases and lines starting with '*' means additional
info provided by curl.

Note that if you want to see HTTP headers in the output, \fI-i/--include\fP
might be option you're looking for.

If you think this option still doesn't give you enough details, consider using
\fI--trace\fP or \fI--trace-ascii\fP instead.

If this option is used twice, the second will again disable verbose.
.IP "-V/--version"
Displays information about curl and the libcurl version it uses.

The first line includes the full version of curl, libcurl and other 3rd party
libraries linked with the executable.

The second line (starts with "Protocols:") shows all protocols that libcurl
reports to support.

The third line (starts with "Features:") shows specific features libcurl
reports to offer.

.IP "-w/--write-out <format>"
Defines what to display after a completed and successful operation. The format
is a string that may contain plain text mixed with any number of variables. The
string can be specified as "string", to get read from a particular file you
specify it "@filename" and to tell curl to read the format from stdin you
write "@-".

The variables present in the output format will be substituted by the value or
text that curl thinks fit, as described below. All variables are specified
like %{variable_name} and to output a normal % you just write them like
%%. You can output a newline by using \\n, a carriage return with \\r and a tab
space with \\t.

.B NOTE:
The %-letter is a special letter in the win32-environment, where all
occurrences of % must be doubled when using this option.

Available variables are at this point:
.RS
.TP 15
.B url_effective
The URL that was fetched last. This is mostly meaningful if you've told curl
to follow location: headers.
.TP
.B http_code
The numerical code that was found in the last retrieved HTTP(S) page.
.TP
.B time_total
The total time, in seconds, that the full operation lasted. The time will be
displayed with millisecond resolution.
.TP
.B time_namelookup
The time, in seconds, it took from the start until the name resolving was
completed.
.TP
.B time_connect
The time, in seconds, it took from the start until the connect to the remote
host (or proxy) was completed.
.TP
.B time_pretransfer
The time, in seconds, it took from the start until the file transfer is just
about to begin. This includes all pre-transfer commands and negotiations that
are specific to the particular protocol(s) involved.
.TP
.B time_starttransfer
The time, in seconds, it took from the start until the first byte is just about
to be transfered. This includes time_pretransfer and also the time the
server needs to calculate the result.
.TP
.B size_download
The total amount of bytes that were downloaded.
.TP
.B size_upload
The total amount of bytes that were uploaded.
.TP
.B size_header
The total amount of bytes of the downloaded headers.
.TP
.B size_request
The total amount of bytes that were sent in the HTTP request.
.TP
.B speed_download
The average download speed that curl measured for the complete download.
.TP
.B speed_upload
The average upload speed that curl measured for the complete upload.
.TP
.B content_type
The Content-Type of the requested document, if there was any. (Added in 7.9.5)
.RE

If this option is used several times, the last one will be used.
.IP "-x/--proxy <proxyhost[:port]>"
Use specified HTTP proxy. If the port number is not specified, it is assumed
at port 1080.

This option overrides existing environment variables that sets proxy to
use. If there's an environment variable setting a proxy, you can set proxy to
\&"" to override it.

\fBNote\fP that all operations that are performed over a HTTP proxy will
transparantly be converted to HTTP. It means that certain protocol specific
operations might not be available. This is not the case if you can tunnel
through the proxy, as done with the \fI-p/--proxytunnel\fP option.

If this option is used several times, the last one will be used.
.IP "-X/--request <command>"
(HTTP)
Specifies a custom request to use when communicating with the HTTP server.
The specified request will be used instead of the standard GET. Read the
HTTP 1.1 specification for details and explanations.

(FTP)
Specifies a custom FTP command to use instead of LIST when doing file lists
with ftp.

If this option is used several times, the last one will be used.
.IP "-y/--speed-time <time>"
If a download is slower than speed-limit bytes per second during a speed-time
period, the download gets aborted. If speed-time is used, the default
speed-limit will be 1 unless set with -y.

This option controls transfers and thus will not affect slow connects etc. If
this is a concern for you, try the \fI--connect-timeout\fP option.

If this option is used several times, the last one will be used.
.IP "-Y/--speed-limit <speed>"
If a download is slower than this given speed, in bytes per second, for
speed-time seconds it gets aborted. speed-time is set with -Y and is 30 if
not set.

If this option is used several times, the last one will be used.
.IP "-z/--time-cond <date expression>"
(HTTP)
Request to get a file that has been modified later than the given time and
date, or one that has been modified before that time. The date expression can
be all sorts of date strings or if it doesn't match any internal ones, it
tries to get the time from a given file name instead! See the
.BR "GNU date(1)"
or
.BR "curl_getdate(3)"
man pages for date expression details.

Start the date expression with a dash (-) to make it request for a document
that is older than the given date/time, default is a document that is newer
than the specified date/time.

If this option is used several times, the last one will be used.
.IP "-Z/--max-redirs <num>"
Set maximum number of redirection-followings allowed. If -L/--location is
used, this option can be used to prevent curl from following redirections "in
absurdum".

If this option is used several times, the last one will be used.
.IP "-0/--http1.0"
(HTTP) Forces curl to issue its requests using HTTP 1.0 instead of using its
internally preferred: HTTP 1.1.
.IP "-1/--tlsv1"
(HTTPS)
Forces curl to use TSL version 1 when negotiating with a remote TLS server.
.IP "-2/--sslv2"
(HTTPS)
Forces curl to use SSL version 2 when negotiating with a remote SSL server.
.IP "-3/--sslv3"
(HTTPS)
Forces curl to use SSL version 3 when negotiating with a remote SSL server.
.IP "-4/--ipv4"
If libcurl is capable of resolving an address to multiple IP versions (which
it is if it is ipv6-capable), this option tells libcurl to resolve names to
IPv4 addresses only. (Added in 7.10.8)
.IP "-6/--ipv6"
If libcurl is capable of resolving an address to multiple IP versions (which
it is if it is ipv6-capable), this option tells libcurl to resolve names to
IPv6 addresses only. (Added in 7.10.8)
.IP "-#/--progress-bar"
Make curl display progress information as a progress bar instead of the
default statistics.

If this option is used twice, the second will again disable the progress bar.
.SH FILES
.I ~/.curlrc
.RS
Default config file.

.SH ENVIRONMENT
.IP "http_proxy [protocol://]<host>[:port]"
Sets proxy server to use for HTTP.
.IP "HTTPS_PROXY [protocol://]<host>[:port]"
Sets proxy server to use for HTTPS.
.IP "FTP_PROXY [protocol://]<host>[:port]"
Sets proxy server to use for FTP.
.IP "GOPHER_PROXY [protocol://]<host>[:port]"
Sets proxy server to use for GOPHER.
.IP "ALL_PROXY [protocol://]<host>[:port]"
Sets proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
'*' only, it matches all hosts.
.SH EXIT CODES
There exists a bunch of different error codes and their corresponding error
messages that may appear during bad conditions. At the time of this writing,
the exit codes are:
.IP 1
Unsupported protocol. This build of curl has no support for this protocol.
.IP 2
Failed to initialize.
.IP 3
URL malformat. The syntax was not correct.
.IP 4
URL user malformatted. The user-part of the URL syntax was not correct.
.IP 5
Couldn't resolve proxy. The given proxy host could not be resolved.
.IP 6
Couldn't resolve host. The given remote host was not resolved.
.IP 7
Failed to connect to host.
.IP 8
FTP weird server reply. The server sent data curl couldn't parse.
.IP 9
FTP access denied. The server denied login.
.IP 10
FTP user/password incorrect. Either one or both were not accepted by the
server.
.IP 11
FTP weird PASS reply. Curl couldn't parse the reply sent to the PASS request.
.IP 12
FTP weird USER reply. Curl couldn't parse the reply sent to the USER request.
.IP 13
FTP weird PASV reply, Curl couldn't parse the reply sent to the PASV request.
.IP 14
FTP weird 227 format. Curl couldn't parse the 227-line the server sent.
.IP 15
FTP can't get host. Couldn't resolve the host IP we got in the 227-line.
.IP 16
FTP can't reconnect. Couldn't connect to the host we got in the 227-line.
.IP 17
FTP couldn't set binary. Couldn't change transfer method to binary.
.IP 18
Partial file. Only a part of the file was transfered.
.IP 19
FTP couldn't download/access the given file, the RETR (or similar) command
failed.
.IP 20
FTP write error. The transfer was reported bad by the server.
.IP 21
FTP quote error. A quote command returned error from the server.
.IP 22
HTTP page not retrieved. The requested url was not found or returned another
error with the HTTP error code being 400 or above. This return code only
appears if --fail is used.
.IP 23
Write error. Curl couldn't write data to a local filesystem or similar.
.IP 24
Malformed user. User name badly specified.
.IP 25
FTP couldn't STOR file. The server denied the STOR operation, used for FTP
uploading.
.IP 26
Read error. Various reading problems.
.IP 27
Out of memory. A memory allocation request failed.
.IP 28
Operation timeout. The specified time-out period was reached according to the
conditions.
.IP 29
FTP couldn't set ASCII. The server returned an unknown reply.
.IP 30
FTP PORT failed. The PORT command failed. Not all FTP servers support the PORT
command, try doing a transfer using PASV instead!
.IP 31
FTP couldn't use REST. The REST command failed. This command is used for
resumed FTP transfers.
.IP 32
FTP couldn't use SIZE. The SIZE command failed. The command is an extension
to the original FTP spec RFC 959.
.IP 33
HTTP range error. The range "command" didn't work.
.IP 34
HTTP post error. Internal post-request generation error.
.IP 35
SSL connect error. The SSL handshaking failed.
.IP 36
FTP bad download resume. Couldn't continue an earlier aborted download.
.IP 37
FILE couldn't read file. Failed to open the file. Permissions?
.IP 38
LDAP cannot bind. LDAP bind operation failed.
.IP 39
LDAP search failed.
.IP 40
Library not found. The LDAP library was not found.
.IP 41
Function not found. A required LDAP function was not found.
.IP 42
Aborted by callback. An application told curl to abort the operation.
.IP 43
Internal error. A function was called with a bad parameter.
.IP 44
Internal error. A function was called in a bad order.
.IP 45
Interface error. A specified outgoing interface could not be used.
.IP 46
Bad password entered. An error was signaled when the password was entered.
.IP 47
Too many redirects. When following redirects, curl hit the maximum amount.
.IP 48
Unknown TELNET option specified.
.IP 49
Malformed telnet option.
.IP 51
The remote peer's SSL certificate wasn't ok
.IP 52
The server didn't reply anything, which here is considered an error.
.IP 53
SSL crypto engine not found
.IP 54
Cannot set SSL crypto engine as default
.IP 55
Failed sending network data
.IP 56
Failure in receiving network data
.IP 57
Share is in use (internal error)
.IP 58
Problem with the local certificate
.IP 59
Couldn't use specified SSL cipher
.IP 60
Problem with the CA cert (path? permission?)
.IP 61
Unrecognized transfer encoding
.IP 62
Invalid LDAP URL
.IP 63
Maximum file size exceeded
.IP XX
There will appear more error codes here in future releases. The existing ones
are meant to never change.
.SH AUTHORS / CONTRIBUTORS
Daniel Stenberg is the main author, but the whole list of contributors is
found in the separate THANKS file.
.SH WWW
http://curl.haxx.se
.SH FTP
ftp://ftp.sunet.se/pub/www/utilities/curl/
.SH "SEE ALSO"
.BR ftp (1),
.BR wget (1),
.BR snarf (1)