manual

harvest是一个下载html网页得机器人

LATEST VERSION

  You always find news about what's going on as well as the latest versions
  from the curl web pages, located at:

        http://curl.haxx.se

SIMPLE USAGE

  Get the main page from netscape's web-server:

        curl http://www.netscape.com/

  Get the README file the user's home directory at funet's ftp-server:

        curl ftp://ftp.funet.fi/README

  Get a web page from a server using port 8000:

        curl http://www.weirdserver.com:8000/

  Get a list of a directory of an FTP site:

        curl ftp://cool.haxx.se/

  Get a gopher document from funet's gopher server:

        curl gopher://gopher.funet.fi

  Get the definition of curl from a dictionary:

        curl dict://dict.org/m:curl

  Fetch two documents at once:

        curl ftp://cool.haxx.se/ http://www.weirdserver.com:8000/

DOWNLOAD TO A FILE

  Get a web page and store in a local file:

        curl -o thatpage.html http://www.netscape.com/

  Get a web page and store in a local file, make the local file get the name
  of the remote document (if no file name part is specified in the URL, this
  will fail):

        curl -O http://www.netscape.com/index.html

  Fetch two files and store them with their remote names:

        curl -O www.haxx.se/index.html -O curl.haxx.se/download.html

USING PASSWORDS

 FTP

   To ftp files using name+passwd, include them in the URL like:

        curl ftp://name:[e-mail]:port/full/path/to/file

   or specify them with the -u flag like

        curl -u name:passwd ftp://machine.domain:port/full/path/to/file

 HTTP

   The HTTP URL doesn't support user and password in the URL string. Curl
   does support that anyway to provide a ftp-style interface and thus you can
   pick a file like:

        curl http://name:[e-mail]/full/path/to/file

   or specify user and password separately like in

        curl -u name:passwd http://machine.domain/full/path/to/file

   NOTE! Since HTTP URLs don't support user and password, you can't use that
   style when using Curl via a proxy. You _must_ use the -u style fetch
   during such circumstances.

 HTTPS

   Probably most commonly used with private certificates, as explained below.

 GOPHER

   Curl features no password support for gopher.

PROXY

 Get an ftp file using a proxy named my-proxy that uses port 888:

        curl -x my-proxy:888 ftp://ftp.leachsite.com/README

 Get a file from a HTTP server that requires user and password, using the
 same proxy as above:

        curl -u user:passwd -x my-proxy:888 http://www.get.this/

 Some proxies require special authentication. Specify by using -U as above:

        curl -U user:passwd -x my-proxy:888 http://www.get.this/

 See also the environment variables Curl support that offer further proxy
 control.

RANGES

  With HTTP 1.1 byte-ranges were introduced. Using this, a client can request
  to get only one or more subparts of a specified document. Curl supports
  this with the -r flag.

  Get the first 100 bytes of a document:

        curl -r 0-99 http://www.get.this/

  Get the last 500 bytes of a document:

        curl -r -500 http://www.get.this/

  Curl also supports simple ranges for FTP files as well. Then you can only
  specify start and stop position.

  Get the first 100 bytes of a document using FTP:

        curl -r 0-99 ftp://www.get.this/README  

UPLOADING

 FTP

  Upload all data on stdin to a specified ftp site:

        curl -T - ftp://ftp.upload.com/myfile

  Upload data from a specified file, login with user and password:

        curl -T uploadfile -u user:passwd ftp://ftp.upload.com/myfile

  Upload a local file to the remote site, and use the local file name remote
  too:
 
        curl -T uploadfile -u user:passwd ftp://ftp.upload.com/

  Upload a local file to get appended to the remote file using ftp:

        curl -T localfile -a ftp://ftp.upload.com/remotefile

  Curl also supports ftp upload through a proxy, but only if the proxy is
  configured to allow that kind of tunneling. If it does, you can run curl in
  a fashion similar to:

        curl --proxytunnel -x proxy:port -T localfile ftp.upload.com

 HTTP

  Upload all data on stdin to a specified http site:

        curl -T - http://www.upload.com/myfile

  Note that the http server must've been configured to accept PUT before this
  can be done successfully.

  For other ways to do http data upload, see the POST section below.

VERBOSE / DEBUG

  If curl fails where it isn't supposed to, if the servers don't let you in,
  if you can't understand the responses: use the -v flag to get verbose
  fetching. Curl will output lots of info and what it sends and receives in
  order to let the user see all client-server interaction (but it won't show
  you the actual data).

        curl -v ftp://ftp.upload.com/

  To get even more details and information on what curl does, try using the
  --trace or --trace-ascii options with a given file name to log to, like
  this:

        curl --trace trace.txt www.haxx.se
 

DETAILED INFORMATION

  Different protocols provide different ways of getting detailed information
  about specific files/documents. To get curl to show detailed information
  about a single file, you should use -I/--head option. It displays all
  available info on a single file for HTTP and FTP. The HTTP information is a
  lot more extensive.

  For HTTP, you can get the header information (the same as -I would show)
  shown before the data by using -i/--include. Curl understands the
  -D/--dump-header option when getting files from both FTP and HTTP, and it
  will then store the headers in the specified file.

  Store the HTTP headers in a separate file (headers.txt in the example):

        curl --dump-header headers.txt curl.haxx.se

  Note that headers stored in a separate file can be very useful at a later
  time if you want curl to use cookies sent by the server. More about that in
  the cookies section.

POST (HTTP)

  It's easy to post data using curl. This is done using the -d <data>
  option.  The post data must be urlencoded.

  Post a simple "name" and "phone" guestbook.

        curl -d "name=Rafael%20Sagula&phone=3320780" \
                http://www.where.com/guest.cgi

  How to post a form with curl, lesson #1:

  Dig out all the <input> tags in the form that you want to fill in. (There's
  a perl program called formfind.pl on the curl site that helps with this).

  If there's a "normal" post, you use -d to post. -d takes a full "post
  string", which is in the format

        <variable1>=<data1>&<variable2>=<data2>&...

  The 'variable' names are the names set with "name=" in the <input> tags, and
  the data is the contents you want to fill in for the inputs. The data *must*
  be properly URL encoded. That means you replace space with + and that you
  write weird letters with %XX where XX is the hexadecimal representation of
  the letter's ASCII code.

  Example:

  (page located at http://www.formpost.com/getthis/

        <form action="post.cgi" method="post">
        <input name=user size=10>
        <input name=pass type=password size=10>
        <input name=id type=hidden value="blablabla">
        <input name=ding value="submit">
        </form>

  We want to enter user 'foobar' with password '12345'.

  To post to this, you enter a curl command line like:

        curl -d "user=foobar&pass=12345&id=blablabla&ding=submit"  (continues)
          http://www.formpost.com/getthis/post.cgi


  While -d uses the application/x-www-form-urlencoded mime-type, generally
  understood by CGI's and similar, curl also supports the more capable
  multipart/form-data type. This latter type supports things like file upload.

  -F accepts parameters like -F "name=contents". If you want the contents to
  be read from a file, use <@filename> as contents. When specifying a file,
  you can also specify the file content type by appending ';type=<mime type>'
  to the file name. You can also post the contents of several files in one
  field.  For example, the field name 'coolfiles' is used to send three files,
  with different content types using the following syntax:

        curl -F "coolfiles=@fil1.gif;type=image/gif,fil2.txt,fil3.html" \
        http://www.post.com/postit.cgi

  If the content-type is not specified, curl will try to guess from the file
  extension (it only knows a few), or use the previously specified type (from
  an earlier file if several files are specified in a list) or else it will
  using the default type 'text/plain'.

  Emulate a fill-in form with -F. Let's say you fill in three fields in a
  form. One field is a file name which to post, one field is your name and one
  field is a file description. We want to post the file we have written named
  "cooltext.txt". To let curl do the posting of this data instead of your
  favourite browser, you have to read the HTML source of the form page and
  find the names of the input fields. In our example, the input field names
  are 'file', 'yourname' and 'filedescription'.

        curl -F "file=@cooltext.txt" -F "yourname=Daniel" \
             -F "filedescription=Cool text file with cool text inside" \
             http://www.post.com/postit.cgi

  To send two files in one post you can do it in two ways:

  1. Send multiple files in a single "field" with a single field name:
 
        curl -F "pictures=@dog.gif,cat.gif" 
 
  2. Send two fields with two field names: 

        curl -F "docpicture=@dog.gif" -F "catpicture=@cat.gif" 

REFERRER

  A HTTP request has the option to include information about which address
  that referred to actual page.  Curl allows you to specify the