faq

harvest是一个下载html网页得机器人

Updated: June 17, 2003 (http://curl.haxx.se/docs/faq.html)
                                  _   _ ____  _     
                              ___| | | |  _ \| |    
                             / __| | | | |_) | |    
                            | (__| |_| |  _ <| |___ 
                             \___|\___/|_| \_\_____|

FAQ

 1. Philosophy
  1.1 What is cURL?
  1.2 What is libcurl?
  1.3 What is cURL not?
  1.4 When will you make curl do XXXX ?
  1.5 Who makes cURL?
  1.6 What do you get for making cURL?
  1.7 What about CURL from curl.com?
  1.8 I have a problem who do I mail?

 2. Install Related Problems
  2.1 configure doesn't find OpenSSL even when it is installed
   2.1.1. native linker doesn't find openssl
   2.1.2. only the libssl lib is missing
  2.2 Does curl work/build with other SSL libraries?
  2.3 Where can I find a copy of LIBEAY32.DLL?
  2.4 Does cURL support Socks (RFC 1928) ?

 3. Usage Problems
  3.1 curl: (1) SSL is disabled, https: not supported
  3.2 How do I tell curl to resume a transfer?
  3.3 Why doesn't my posting using -F work?
  3.4 How do I tell curl to run custom FTP commands?
  3.5 How can I disable the Pragma: nocache header?
  3.6 Does curl support ASP, XML, XHTML or HTML version Y?
  3.7 Can I use curl to delete/rename a file through FTP?
  3.8 How do I tell curl to follow HTTP redirects?
  3.9 How do I use curl in my favorite programming language?
  3.10 What about SOAP, WebDAV, XML-RPC or similar protocols over HTTP?
  3.11 How do I POST with a different Content-Type?
  3.12 Why do FTP specific features over HTTP proxy fail?
  3.13 Why does my single/double quotes fail?
  3.14 Does curl support javascript or pac (automated proxy config)?

 4. Running Problems
  4.1 Problems connecting to SSL servers.
  4.2 Why do I get problems when I use & or % in the URL?
  4.3 How can I use {, }, [ or ] to specify multiple URLs?
  4.4 Why do I get downloaded data even though the web page doesn't exist?
  4.5 Why do I get return code XXX from a HTTP server?
   4.5.1 "400 Bad Request"
   4.5.2 "401 Unauthorized"
   4.5.3 "403 Forbidden"
   4.5.4 "404 Not Found"
   4.5.5 "405 Method Not Allowed"
   4.5.6 "301 Moved Permanently"
  4.6 Can you tell me what error code 142 means?
  4.7 How do I keep user names and passwords secret in Curl command lines?
  4.8 I found a bug!
  4.9 Curl can't authenticate to the server that requires NTLM?
  4.10 My HTTP request using HEAD, PUT or DELETE doesn't work!
  4.11 Why does my HTTP range requests return the full document?
  4.12 Why do I get "certificate verify failed" ?

 5. libcurl Issues
  5.1 Is libcurl thread-safe?
  5.2 How can I receive all data into a large memory chunk?
  5.3 How do I fetch multiple files with libcurl?
  5.4 Does libcurl do Winsock initing on win32 systems?
  5.5 Does CURLOPT_FILE and CURLOPT_INFILE work on win32 ?
  5.6 What about Keep-Alive or persistent connections?
  5.7 Link errors when building libcurl on Windows!

 6. License Issues
  6.1 I have a GPL program, can I use the libcurl library?
  6.2 I have a closed-source program, can I use the libcurl library?
  6.3 I have a BSD licensed program, can I use the libcurl library?
  6.4 I have a program that uses LGPL libraries, can I use libcurl?
  6.5 Can I modify curl/libcurl for my program and keep the changes secret?
  6.6 Can you please change the curl/libcurl license to XXXX?

==============================================================================

1. Philosophy

  1.1 What is cURL?

  cURL (or simply just 'curl') is a command line tool for getting or sending
  files using URL syntax. The name is a play on 'Client for URLs', originally
  with URL spelled in uppercase to make it obvious it deals with URLs. The
  fact it can also be pronounced 'see URL' also helped, it works as an
  abbrivation for "Client URL Request Library" or why not the recursive
  version: "Curl is a URL Request Library".

  Curl supports a range of common Internet protocols, currently including
  HTTP, HTTPS, FTP, FTPS, GOPHER, LDAP, DICT, TELNET and FILE.

  We spell it cURL or just curl. We pronounce it with an initial k sound:
  [kurl].

  NOTE: there are numerous sub-projects and related projects that also use the
  word curl in the project names in various combinations, but you should take
  notice that this FAQ is directed at the command-line tool named curl (and
  libcurl the library), and may therefore not be valid for other curl
  projects.

  1.2 What is libcurl?

  libcurl is a reliable and portable library which provides you with an easy
  interface to a range of common Internet protocols.

  You can use libcurl for free in your application even if it is commercial
  or closed-source.

  1.3 What is cURL not?
  
  Curl is *not* a wget clone even though that is a very common misconception.
  Never, during curl's development, have we intended curl to replace wget or
  compete on its market. Curl is targeted at single-shot file transfers.

  Curl is not a web site mirroring program. If you wanna use curl to mirror
  something: fine, go ahead and write a script that wraps around curl to make
  it reality (like curlmirror.pl does).

  Curl is not an FTP site mirroring program. Sure, get and send FTP with curl
  but if you want systematic and sequential behavior you should write a
  script (or write a new program that interfaces libcurl) and do it.

  Curl is not a PHP tool, even though it works perfectly well when used from
  or with PHP.

  Curl is not a single-OS program. Curl exists, compiles, builds and runs
  under a wide range of operating systems, including all modern Unixes (and a
  bunch of older ones too), Windows, Amiga, BeOS, OS/2, OS X, QNX etc.

  1.4 When will you make curl do XXXX ?

  We love suggestions of what to change in order to make curl and libcurl
  better. We do however believe in a few rules when it comes to the future of
  curl:

  * Curl -- the command line tool -- is to remain a non-graphical command line
    tool. If you want GUIs or fancy scripting capabilities, you should look
    for another tool that uses libcurl.

  * We do not add things to curl that other small and available tools already
    do very fine at the side. Curl's output is fine to pipe into another
    program or redirect to another file for the next program to interpret.

  * We focus on protocol related issues and improvements. If you wanna do more
    magic with the supported protocols than curl currently does, chances are
    big we will agree. If you wanna add more protocols, we may very well
    agree.

  * If you want someone else to make all the work while you wait for us to
    implement it for you, that is not a very friendly attitude. We spend a
    considerable time already on maintaining and developing curl. In order to
    get more out of us, you should consider trading in some of your time and
    efforts in return.

  * If you write the code, chances are bigger that it will get into curl
    faster.

  1.5 Who makes cURL?

  cURL and libcurl are not made by any single individual. Sure, Daniel
  Stenberg writes the major parts, but other persons' submissions are
  important and crucial. Anyone can contribute and post their changes and
  improvements and have them inserted in the main sources (of course on the
  condition that developers agree on that the fixes are good).

  The list of contributors in the docs/THANKS file is only a small part of all
  the people that every day provide us with bug reports, suggestions, ideas
  and source code.

  curl is developed by a community, with Daniel at the wheel.

  1.6 What do you get for making cURL?

  Project cURL is entirely free and open. No person gets paid for developing
  curl. We do this voluntarily on our spare time.

  We get some help from companies. Contactor Data hosts the curl web site,
  Haxx owns the curl web site's domain and sourceforge.net hosts several
  project services we take advantage from, like the bug tracker, mailing lists
  and more.

  If you want to support our project with a donation or similar, one way of
  doing that would be to buy "gift certificates" at useful online shopping
  sites, such as amazon.com or thinkgeek.com. Another way would be to sponsor
  us through a banner-program or even better: by helping us coding,
  documenting, testing etc. You're welcome to send us a buck using paypal, as
  described here: http://curl.haxx.se/donation.html

  1.7 What about CURL from curl.com?

  During the summer 2001, curl.com was busy advertising their client-side
  programming language for the web, named CURL.

  We are in no way associated with curl.com or their CURL programming
  language. 

  Our project name curl has been in effective use since 1998. We were not the
  first computer related project to use the name "curl" and do not claim any
  first-hand rights to the name.

  We recognize that we will be living in parallel with curl.com and wish them
  every success.

  1.8 I have a problem who do I mail?

  Please do not mail any single individual unless you really need to. Keep
  curl-related questions on a suitable mailing list. All available mailing
  lists are listed in the MANUAL document and online at
  http://curl.haxx.se/mail/

  Keeping curl-related questions and discussions on mailing lists allows
  others to join in and help, to share their ideas, contribute their
  suggestions and spread their wisdom. Keeping discussions on public mailing
  lists also allows for others to learn from this (both current and future
  users thanks to the web based archives of the mailing lists), thus saving us
  from having to repeat ourselves even more. Thanks for respecting this.


2. Install Related Problems

  2.1. configure doesn't find OpenSSL even when it is installed

  This may be because of several reasons.

    2.1.1. native linker doesn't find openssl

    Affected platforms:
      Solaris (native cc compiler)
      HPUX (native cc compiler)
      SGI IRIX (native cc compiler)
      SCO UNIX (native cc compiler)

    When configuring curl, I specify --with-ssl. OpenSSL is installed in
    /usr/local/ssl Configure reports SSL in /usr/local/ssl, but fails to find
    CRYPTO_lock in -lcrypto

    Cause: The cc for this test places the -L/usr/local/ssl/lib AFTER
    -lcrypto, so ld can't find the library. This is due to a bug in the GNU
    autoconf tool.

    Workaround: Specifying "LDFLAGS=-L/usr/local/ssl/lib" in front of
    ./configure places the -L/usr/local/ssl/lib early enough in the command
    line to make things work

    Solution submitted by: Bob Allison <[e-mail]>

    2.1.2. only the libssl lib is missing

    If all include files and the libcrypto lib is present, with only the
    libssl being missing according to configure, this is mostly likely because
    a few functions are left out from the libssl.

    If the function names missing include RSA or RSAREF you can be certain
    that this is because libssl requires the RSA and RSAREF libs to build.

    See the INSTALL file section that explains how to add those libs to