faq

功能最强大的网络爬虫,希望大家好好学习啊,好好研究啊

Updated: November 2, 2005 (http://curl.haxx.se/docs/faq.html)
                                  _   _ ____  _
                              ___| | | |  _ \| |
                             / __| | | | |_) | |
                            | (__| |_| |  _ <| |___
                             \___|\___/|_| \_\_____|

FAQ

 1. Philosophy
  1.1 What is cURL?
  1.2 What is libcurl?
  1.3 What is curl not?
  1.4 When will you make curl do XXXX ?
  1.5 Who makes curl?
  1.6 What do you get for making curl?
  1.7 What about CURL from curl.com?
  1.8 I have a problem who do I mail?
  1.9 Where do I buy commercial support for curl?
  1.10 How many are using curl?

 2. Install Related Problems
  2.1 configure doesn't find OpenSSL even when it is installed
   2.1.1 native linker doesn't find OpenSSL
   2.1.2 only the libssl lib is missing
  2.2 Does curl work/build with other SSL libraries?
  2.3 Where can I find a copy of LIBEAY32.DLL?
  2.4 Does curl support Socks (RFC 1928) ?

 3. Usage Problems
  3.1 curl: (1) SSL is disabled, https: not supported
  3.2 How do I tell curl to resume a transfer?
  3.3 Why doesn't my posting using -F work?
  3.4 How do I tell curl to run custom FTP commands?
  3.5 How can I disable the Pragma: nocache header?
  3.6 Does curl support ASP, XML, XHTML or HTML version Y?
  3.7 Can I use curl to delete/rename a file through FTP?
  3.8 How do I tell curl to follow HTTP redirects?
  3.9 How do I use curl in my favorite programming language?
  3.10 What about SOAP, WebDAV, XML-RPC or similar protocols over HTTP?
  3.11 How do I POST with a different Content-Type?
  3.12 Why do FTP specific features over HTTP proxy fail?
  3.13 Why does my single/double quotes fail?
  3.14 Does curl support javascript or pac (automated proxy config)?
  3.15 Can I do recursive fetches with curl?
  3.16 What certificates do I need when I use SSL?
  3.17 How do I list the root dir of an FTP server?
  3.18 Can I use curl to send a POST/PUT and not wait for a response?

 4. Running Problems
  4.1 Problems connecting to SSL servers.
  4.2 Why do I get problems when I use & or % in the URL?
  4.3 How can I use {, }, [ or ] to specify multiple URLs?
  4.4 Why do I get downloaded data even though the web page doesn't exist?
  4.5 Why do I get return code XXX from a HTTP server?
   4.5.1 "400 Bad Request"
   4.5.2 "401 Unauthorized"
   4.5.3 "403 Forbidden"
   4.5.4 "404 Not Found"
   4.5.5 "405 Method Not Allowed"
   4.5.6 "301 Moved Permanently"
  4.6 Can you tell me what error code 142 means?
  4.7 How do I keep user names and passwords secret in Curl command lines?
  4.8 I found a bug!
  4.9 Curl can't authenticate to the server that requires NTLM?
  4.10 My HTTP request using HEAD, PUT or DELETE doesn't work!
  4.11 Why does my HTTP range requests return the full document?
  4.12 Why do I get "certificate verify failed" ?
  4.13 Why is curl -R on Windows one hour off?
  4.14 Redirects work in browser but not with curl!

 5. libcurl Issues
  5.1 Is libcurl thread-safe?
  5.2 How can I receive all data into a large memory chunk?
  5.3 How do I fetch multiple files with libcurl?
  5.4 Does libcurl do Winsock initing on win32 systems?
  5.5 Does CURLOPT_WRITEDATA and CURLOPT_READDATA work on win32 ?
  5.6 What about Keep-Alive or persistent connections?
  5.7 Link errors when building libcurl on Windows!
  5.8 libcurl.so.3: open failed: No such file or directory
  5.9 How does libcurl resolve host names?
  5.10 How do I prevent libcurl from writing the response to stdout?
  5.11 How do I make libcurl not receive the whole HTTP response?
  5.12 Can I make libcurl fake or hide my real IP address?

 6. License Issues
  6.1 I have a GPL program, can I use the libcurl library?
  6.2 I have a closed-source program, can I use the libcurl library?
  6.3 I have a BSD licensed program, can I use the libcurl library?
  6.4 I have a program that uses LGPL libraries, can I use libcurl?
  6.5 Can I modify curl/libcurl for my program and keep the changes secret?
  6.6 Can you please change the curl/libcurl license to XXXX?
  6.7 What are my obligations when using libcurl in my commerical apps?

 7. PHP/CURL Issues
  7.1 What is PHP/CURL?
  7.2 Who write PHP/CURL?
  7.3 Can I perform multiple requests using the same handle?

==============================================================================

1. Philosophy

  1.1 What is cURL?

  cURL is the name of the project. The name is a play on 'Client for URLs',
  originally with URL spelled in uppercase to make it obvious it deals with
  URLs. The fact it can also be pronounced 'see URL' also helped, it works as
  an abbreviation for "Client URL Request Library" or why not the recursive
  version: "Curl URL Request Library".

  The cURL project produces two products:

  libcurl

    A free and easy-to-use client-side URL transfer library, supporting FTP,
    FTPS, HTTP, HTTPS, TELNET, DICT, FILE and LDAP.  libcurl supports
    HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, kerberos, HTTP
    form based upload, proxies, cookies, user+password authentication, file
    transfer resume, http proxy tunneling and more!

    libcurl is highly portable, it builds and works identically on numerous
    platforms, including Solaris, NetBSD, FreeBSD, OpenBSD, Darwin, HPUX,
    IRIX, AIX, Tru64, Linux, UnixWare, HURD, Windows, Amiga, OS/2, BeOs, Mac
    OS X, Ultrix, QNX, OpenVMS, RISC OS, Novell NetWare, DOS and more...

    libcurl is free, thread-safe, IPv6 compatible, feature rich, well
    supported and fast.

  curl

    A command line tool for getting or sending files using URL syntax.

    Since curl uses libcurl, it supports a range of common Internet protocols,
    currently including HTTP, HTTPS, FTP, FTPS, LDAP, DICT, TELNET and FILE.

  We pronounce curl and cURL with an initial k sound: [kurl].

  There are numerous sub-projects and related projects that also use the word
  curl in the project names in various combinations, but you should take
  notice that this FAQ is directed at the command-line tool named curl (and
  libcurl the library), and may therefore not be valid for other curl-related
  projects. (There is however a small section for the PHP/CURL in this FAQ.)

  1.2 What is libcurl?

  libcurl is a reliable and portable library which provides you with an easy
  interface to a range of common Internet protocols.

  You can use libcurl for free in your application, be it open source,
  commercial or closed-source.

  libcurl is most probably the most portable, most powerful and most often
  used C-based multi-platform file transfer library on this planet - be it
  open source or commercial.

  1.3 What is curl not?

  Curl is *not* a wget clone. That is a common misconception.  Never, during
  curl's development, have we intended curl to replace wget or compete on its
  market. Curl is targeted at single-shot file transfers.

  Curl is not a web site mirroring program. If you want to use curl to mirror
  something: fine, go ahead and write a script that wraps around curl to make
  it reality (like curlmirror.pl does).

  Curl is not an FTP site mirroring program. Sure, get and send FTP with curl
  but if you want systematic and sequential behavior you should write a
  script (or write a new program that interfaces libcurl) and do it.

  Curl is not a PHP tool, even though it works perfectly well when used from
  or with PHP (when using the PHP/CURL module).

  Curl is not a single-OS program. Curl exists, compiles, builds and runs
  under a wide range of operating systems, including all modern Unixes (and a
  bunch of older ones too), Windows, Amiga, BeOS, OS/2, OS X, QNX etc.

  1.4 When will you make curl do XXXX ?

  We love suggestions of what to change in order to make curl and libcurl
  better. We do however believe in a few rules when it comes to the future of
  curl:

  * Curl -- the command line tool -- is to remain a non-graphical command line
    tool. If you want GUIs or fancy scripting capabilities, you should look
    for another tool that uses libcurl.

  * We do not add things to curl that other small and available tools already
    do very fine at the side. Curl's output is fine to pipe into another
    program or redirect to another file for the next program to interpret.

  * We focus on protocol related issues and improvements. If you wanna do more
    magic with the supported protocols than curl currently does, chances are
    big we will agree. If you wanna add more protocols, we may very well
    agree.

  * If you want someone else to make all the work while you wait for us to
    implement it for you, that is not a very friendly attitude. We spend a
    considerable time already on maintaining and developing curl. In order to
    get more out of us, you should consider trading in some of your time and
    efforts in return.

  * If you write the code, chances are bigger that it will get into curl
    faster.

  1.5 Who makes curl?

  curl and libcurl are not made by any single individual. Daniel Stenberg is
  project leader and main developer, but other persons' submissions are
  important and crucial. Anyone can contribute and post their changes and
  improvements and have them inserted in the main sources (of course on the
  condition that developers agree on that the fixes are good).

  The full list of the more than 450 contributors is found in the docs/THANKS
  file.

  curl is developed by a community, with Daniel at the wheel.

  1.6 What do you get for making curl?

  Project cURL is entirely free and open. No person gets paid for developing
  (lib)curl. We do this voluntarily on our spare time.

  We get some help from companies. Contactor Data hosts the curl web site,
  Haxx owns the curl web site's domain and sourceforge.net hosts project
  services we take advantage from, like the bug tracker. Also, some companies
  have sponsored certain parts of the development in the past and I hope some
  will continue to do so in the future.

  If you want to support our project, consider a donation or a banner-program
  or even better: by helping us coding, documenting, testing etc.

  1.7 What about CURL from curl.com?

  During the summer 2001, curl.com was busy advertising their client-side
  programming language for the web, named CURL.

  We are in no way associated with curl.com or their CURL programming
  language.

  Our project name curl has been in effective use since 1998. We were not the
  first computer related project to use the name "curl" and do not claim any
  first-hand rights to the name.

  We recognize that we will be living in parallel with curl.com and wish them
  every success.

  1.8 I have a problem who do I mail?

  Please do not mail any single individual unless you really need to. Keep
  curl-related questions on a suitable mailing list. All available mailing
  lists are listed in the MANUAL document and online at
  http://curl.haxx.se/mail/

  Keeping curl-related questions and discussions on mailing lists allows
  others to join in and help, to share their ideas, contribute their
  suggestions and spread their wisdom. Keeping discussions on public mailing
  lists also allows for others to learn from this (both current and future
  users thanks to the web based archives of the mailing lists), thus saving us
  from having to repeat ourselves even more. Thanks for respecting this.

  If you have found or simply suspect a security problem in curl or libcurl,
  mail curl-security at haxx.se (closed list of receivers, mails are not
  disclosed) and tell. Then we can produce a fix in a timely manner before the
  flaw is announced to the world, thus lessen the impact the problem will have
  on existing users.

  1.9 Where do I buy commercial support for curl?

  curl is fully open source. It means you can hire any skilled engineer to fix
  your curl-related problems.

  We list available alternatives on the curl web site:
  http://curl.haxx.se/support.html

  1.10 How many are using curl?

  It is impossible to tell.

  We don't know how many users that knowingly have installed and use curl.

  We don't know how many users that use curl without knowing that they are in
  fact using it.

  We don't know how many users that downloaded or installed curl and then
  never use it.

  Some facts to use as input to the math:

  curl packages have been downloaded from the curl.haxx.se site well over a
  million times. curl is installed by default with most Linux
  distributions. curl is installed by default with Mac OS X. curl and libcurl
  as used by numerous applications that include libcurl binaries in their
  distribution packages (like Adobe Acrobat Reader and Google Earth).

  More than 40 known named companies use curl in commercial environments and
  products. More than 100 known named open source projects depend on
  (lib)curl.

  In a poll on the curl web site mid-2005, more than 50% of the 300+ answers
  estimated a user base of one million users or more.

  In March 2005, the "Linux Counter project" estimated a total Linux user base
  of some 29 millions, while Netcraft detected some 4 million "active" Linux
  based web servers. A guess is that a fair amount of these Linux
  installations have curl installed.

  http://curl.haxx.se/docs/companies.html
  http://curl.haxx.se/docs/programs.html
  http://curl.haxx.se/libcurl/using/apps.html
  http://counter.li.org/estimates.php
  http://news.netcraft.com/archives/2005/03/14/fedora_makes_rapid_progress.html


2. Install Related Problems

  2.1 configure doesn't find OpenSSL even when it is installed

  This may be because of several reasons.

    2.1.1 native linker doesn't find openssl

    Affected platforms:
      Solaris (native cc compiler)
      HPUX (native cc compiler)
      SGI IRIX (native cc compiler)
      SCO UNIX (native cc compiler)

    When configuring curl, I specify --with-ssl. OpenSSL is installed in
    /usr/local/ssl Configure reports SSL in /usr/local/ssl, but fails to find
    CRYPTO_lock in -lcrypto

    Cause: The cc for this test places the -L/usr/local/ssl/lib AFTER
    -lcrypto, so ld can't find the library. This is due to a bug in the GNU
    autoconf tool.

    Workaround: Specifying "LDFLAGS=-L/usr/local/ssl/lib" in front of
    ./configure places the -L/usr/local/ssl/lib early enough in the command
    line to make things work

    Solution submitted by: Bob Allison <[e-mail]>

    2.1.2 only the libssl lib is missing

    If all include files and the libcrypto lib is present, with only the
    libssl being missing according to configure, this is mostly likely because
    a few functions are left out from the libssl.

    If the function names missing include RSA or RSAREF you can be certain
    that this is because libssl requires the RSA and RSAREF libs to build.

    See the INSTALL file section that explains how to add those libs to
    configure. Make sure that you remove the config.cache file before you
    rerun configure with the new flags.

  2.2 Does curl work/build with other SSL libraries?

  Curl has been written to use OpenSSL or GnuTLS, although there should not be
  many problems using a different library. If anyone does "port" curl to use a
  different SSL library, we are of course very interested in getting the
  patch!

  2.3 Where can I find a copy of LIBEAY32.DLL?

  That is an OpenSSL binary built for Windows.

  Curl uses OpenSSL to do the SSL stuff. The LIBEAY32.DLL is what curl needs
  on a windows machine to do https://. Check out the curl web site to find
  accurate and up-to-date pointers to recent OpenSSL DLLs and other binary
  packages.

  2.4 Does curl support Socks (RFC 1928) ?