faq

功能最强大的网络爬虫,希望大家好好学习啊,好好研究啊

  you have.

  If there is a bug, read the BUGS document first. Then report it as described
  in there.

  4.9 Curl can't authenticate to the server that requires NTLM?

  This is supported in curl 7.10.6 or later. No earlier curl version knows
  of this magic.

  NTLM is a Microsoft proprietary protocol. Proprietary formats are evil. You
  should not use such ones.

  4.10 My HTTP request using HEAD, PUT or DELETE doesn't work!

  Many web servers allow or demand that the administrator configures the
  server properly for these requests to work on the web server.

  Some servers seem to support HEAD only on certain kinds of URLs.

  To fully grasp this, try the documentation for the particular server
  software you're trying to interact with. This is not anything curl can do
  anything about.

  4.11 Why does my HTTP range requests return the full document?

  Because the range may not be supported by the server, or the server may
  choose to ignore it and return the full document anyway.

  4.12 Why do I get "certificate verify failed" ?

  You invoke curl 7.10 or later to communicate on a https:// URL and get an
  error back looking something similar to this:

      curl: (35) SSL: error:14090086:SSL routines:
      SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

  Then it means that curl couldn't verify that the server's certificate was
  good. Curl verifies the certificate using the CA cert bundle that comes with
  the curl installation.

  To disable the verification (which makes it act like curl did before 7.10),
  use -k. This does however enable man-in-the-middle attacks.

  If you get this failure but are having a CA cert bundle installed and used,
  the server's certificate is not signed by one of the CA's in the bundle. It
  might for example be self-signed. You then correct this problem by obtaining
  a valid CA cert for the server. Or again, decrease the security by disabling
  this check.

  Details are also in the SSLCERTS file in the release archives, found online
  here: http://curl.haxx.se/docs/sslcerts.html

  4.13 Why is curl -R on Windows one hour off?

  During daylight savings time, when -R is used, curl will set a time that
  appears one hour off. This happens due to a flaw in how Windows stores and
  uses file modification times and it is not easily worked around. For details
  on this problem, read this: http://www.codeproject.com/datetime/dstbugs.asp

  4.14 Redirects work in browser but not with curl!

  curl supports HTTP redirects fine (see item 3.8). Browsers generally support
  at least two other ways to perform directs that curl does not:

  - Meta tags. You can write a HTML tag that will cause the browser to
    redirect to another given URL after a certain time.

  - Javascript. You can write a javascript program embeded in a HTML page
    that redirects the browser to another given URL.

  There is no way to make curl follow these redirects. You must either
  manually figure out what the page is set to do, or you write a script that
  parses the results and fetches the new URL.


5. libcurl Issues

  5.1 Is libcurl thread-safe?

  Yes.

  We have written the libcurl code specifically adjusted for multi-threaded
  programs. libcurl will use thread-safe functions instead of non-safe ones if
  your system has such.

  If you use a OpenSSL-powered libcurl in a multi-threaded environment, you
  need to provide one or two locking functions:

    http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION

  If you use a GnuTLS-powered libcurl in a multi-threaded environment, you
  need to provide locking function(s) for libgcrypt (which is used by GnuTLS
  for the crypto functions).

    http://www.gnu.org/software/gnutls/manual/html_node/Multi_002dthreaded-applications.html

  5.2 How can I receive all data into a large memory chunk?

  [ See also the examples/getinmemory.c source ]

  You are in full control of the callback function that gets called every time
  there is data received from the remote server. You can make that callback do
  whatever you want. You do not have to write the received data to a file.

  One solution to this problem could be to have a pointer to a struct that you
  pass to the callback function. You set the pointer using the
  CURLOPT_WRITEDATA option. Then that pointer will be passed to the callback
  instead of a FILE * to a file:

        /* imaginary struct */
        struct MemoryStruct {
          char *memory;
          size_t size;
        };

        /* imaginary callback function */
        size_t
        WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data)
        {
          size_t realsize = size * nmemb;
          struct MemoryStruct *mem = (struct MemoryStruct *)data;

          mem->memory = (char *)realloc(mem->memory, mem->size + realsize + 1);
          if (mem->memory) {
            memcpy(&(mem->memory[mem->size]), ptr, realsize);
            mem->size += realsize;
            mem->memory[mem->size] = 0;
          }
          return realsize;
        }

  5.3 How do I fetch multiple files with libcurl?

  libcurl has excellent support for transferring multiple files. You should
  just repeatedly set new URLs with curl_easy_setopt() and then transfer it
  with curl_easy_perform(). The handle you get from curl_easy_init() is not
  only reusable, but you're even encouraged to reuse it if you can, as that
  will enable libcurl to use persistent connections.

  5.4 Does libcurl do Winsock initialization on win32 systems?

  Yes, if told to in the curl_global_init() call.

  5.5 Does CURLOPT_WRITEDATA and CURLOPT_READDATA work on win32 ?

  Yes, but you cannot open a FILE * and pass the pointer to a DLL and have
  that DLL use the FILE * (as the DLL and the client application cannot access
  each others' variable memory areas). If you set CURLOPT_WRITEDATA you must
  also use CURLOPT_WRITEFUNCTION as well to set a function that writes the
  file, even if that simply writes the data to the specified FILE *.
  Similarly, if you use CURLOPT_READDATA you must also specify
  CURLOPT_READFUNCTION.

  (Provided by Joel DeYoung and Bob Schader)

  5.6 What about Keep-Alive or persistent connections?

  curl and libcurl have excellent support for persistent connections when
  transferring several files from the same server.  Curl will attempt to reuse
  connections for all URLs specified on the same command line/config file, and
  libcurl will reuse connections for all transfers that are made using the
  same libcurl handle.

  5.7 Link errors when building libcurl on Windows!

  You need to make sure that your project, and all the libraries (both static
  and dynamic) that it links against, are compiled/linked against the same run
  time library.

  This is determined by the /MD, /ML, /MT (and their corresponding /M?d)
  options to the command line compiler. /MD (linking against MSVCRT dll) seems
  to be the most commonly used option.

  (Provided by Andrew Francis)

  When building an application that uses the static libcurl library, you must
  add -DCURL_STATICLIB to your CFLAGS. Otherwise the linker will look for
  dynamic import symbols. If you get linker error like "unknown symbol
  __imp__curl_easy_init ..." you have linked against the wrong (static)
  library.  If you want to use the libcurl.dll and import lib, you don't need
  any extra CFLAGS, but use one of the import libraries below. These are the
  libraries produced by the various lib/Makefile.* files:

  Target:          static lib.   import lib for libcurl*.dll.
  -----------------------------------------------------------
  MingW:           libcurl.a     libcurldll.a
  MSVC (release):  libcurl.lib   libcurl_imp.lib
  MSVC (debug):    libcurld.lib  libcurld_imp.lib
  Borland:         libcurl.lib   libcurl_imp.lib


  5.8 libcurl.so.3: open failed: No such file or directory

  This is an error message you might get when you try to run a program linked
  with a shared version of libcurl and your run-time linker (ld.so) couldn't
  find the shared library named libcurl.so.3.

  You need to make sure that ld.so finds libcurl.so.3. You can do that
  multiple ways, and it differs somewhat between different operating systems,
  but they are usually:

  * Add an option to the linker command line that specify the hard-coded path
    the run-time linker should check for the lib (usually -R)

  * Set an environment variable (LD_LIBRARY_PATH for example) where ld.so
    should check for libs

  * Adjust the system's config to check for libs in the directory where you've
    put the dir (like Linux's /etc/ld.so.conf)

  'man ld.so' and 'man ld' will tell you more details

  5.9 How does libcurl resolve host names?

  libcurl supports a large a number of different name resolve functions. One
  of them is picked at build-time and will be used unconditionally. Thus, if
  you want to change name resolver function you must rebuild libcurl and tell
  it to use a different function.

  - The non-ipv6 resolver that can use one out of four host name resolve calls
    (depending on what your system supports):

    A - gethostbyname()
    B - gethostbyname_r() with 3 arguments
    C - gethostbyname_r() with 5 arguments
    D - gethostbyname_r() with 6 arguments

  - The ipv6-resolver that uses getaddrinfo()

  - The c-ares based name resolver that uses the c-ares library for resolves.
    Using this offers asynchronous name resolves but it currently has no IPv6
    support.

  - The Windows threaded resolver. It use:

    A - gethostbyname() on plain ipv4 windows hosts
    B - getaddrinfo() on ipv6-enabled windows hosts

  Also note that libcurl never resolves or reverse-lookups addresses given as
  pure numbers, such as 127.0.0.1 or ::1.

  5.10 How do I prevent libcurl from writing the response to stdout?

  libcurl provides a default built-in write function that writes received data
  to stdout. Set the CURLOPT_WRITEFUNCTION to receive the data, or possibly
  set CURLOPT_WRITEDATA to a different FILE * handle.

  5.11 How do I make libcurl not receive the whole HTTP response?

  You make the write callback (or progress callback) return an error and
  libcurl will then abort the transfer.

  5.12 Can I make libcurl fake or hide my real IP address?

  No. libcurl operates on a higher level than so. Besides, faking IP address
  would imply sending IP packages with a made-up source address, and then you
  normally get a problem with intercepting the packages sent back as they
  would then not be routed to you!

  If you use a proxy to access remote sites, the sites will not see your local
  IP address but instead the address of the proxy.

  Also note that on many networks NATs or other IP-munging techniques are used
  that makes you see and use a different IP address locally than what the
  remote server will see you coming from.


6. License Issues

  Curl and libcurl are released under a MIT/X derivate license. The license is
  very liberal and should not impose a problem for your project. This section
  is just a brief summary for the cases we get the most questions. (Parts of
  this section was much enhanced by Bjorn Reese.)

  We are not lawyers and this is not legal advice. You should probably consult
  one if you want true and accurate legal insights without our prejudice.

  6.1 I have a GPL program, can I use the libcurl library?

  Yes!

  Since libcurl may be distributed under the MIT/X derivate license, it can be
  used together with GPL in any software.

  6.2 I have a closed-source program, can I use the libcurl library?

  Yes!

  libcurl does not put any restrictions on the program that uses the library.

  6.3 I have a BSD licensed program, can I use the libcurl library?

  Yes!

  libcurl does not put any restrictions on the program that uses the library.

  6.4 I have a program that uses LGPL libraries, can I use libcurl?

  Yes!

  The LGPL license doesn't clash with other licenses.

  6.5 Can I modify curl/libcurl for my program and keep the changes secret?

  Yes!

  The MIT/X derivate license practically allows you to do almost anything with
  the sources, on the condition that the copyright texts in the sources are
  left intact.

  6.6 Can you please change the curl/libcurl license to XXXX?

  No.

  We have carefully picked this license after years of development and
  discussions and a large amount of people have contributed with source code
  knowing that this is the license we use. This license puts the restrictions
  we want on curl/libcurl and it does not spread to other programs or
  libraries that use it. It should be possible for everyone to use libcurl or
  curl in their projects, no matter what license they already have in use.

  6.7 What are my obligations when using libcurl in my commerical apps?

  Next to none. All you need to adhere to is the MIT-style license (stated in
  the COPYING file) which basically says you have to include the copyright
  notice in "all copies" and that you may not use the copyright holder's name
  when promoting your software.

  You do not have to release any of your source code.

  You do not have to reveal or make public any changes to the libcurl source
  code.

  You do not have to reveal or make public that you are using libcurl within
  your app.

  As can be seen here: http://curl.haxx.se/docs/companies.html and
  elsewhere, more and more companies are dicovering the power
  of libcurl and take advantage of it even in commercial environments.


7. PHP/CURL Issues

  7.1 What is PHP/CURL?

  The module for PHP that makes it possible for PHP programs to access curl-
  functions from within PHP.

  In the cURL project we call this module PHP/CURL to differentiate it from
  curl the command line tool and libcurl the library. The PHP team however
  does not refer to it like this (for unknown reasons). They call it plain
  CURL (often using all caps) which causes much confusion to users which in
  turn gives us a higher question load.

  7.2 Who write PHP/CURL?

  PHP/CURL is a module that comes with the regular PHP package. It depends and
  uses libcurl, so you need to have libcurl installed properly first before
  PHP/CURL can be used. PHP/CURL is written by Sterling Hughes.

  7.3 Can I perform multiple requests using the same handle?

  Yes - at least in PHP version 4.3.8 and later (this has been known to not
  work in earlier versions, but the exact version when it started to work is
  unknown to me).

  After a transfer, you just set new options in the handle and make another
  transfer. This will make libcurl to re-use the same connection if it can.