faq

harvest是一个下载html网页得机器人

    The request could not be understood by the server due to malformed
    syntax. The client SHOULD NOT repeat the request without modifications.

    4.5.2 "401 Unauthorized"

    The request requires user authentication.

    4.5.3 "403 Forbidden"

    The server understood the request, but is refusing to fulfill it.
    Authorization will not help and the request SHOULD NOT be repeated.

    4.5.4 "404 Not Found"

    The server has not found anything matching the Request-URI. No indication
    is given of whether the condition is temporary or permanent.

    4.5.5 "405 Method Not Allowed"

    The method specified in the Request-Line is not allowed for the resource
    identified by the Request-URI. The response MUST include an Allow header
    containing a list of valid methods for the requested resource.

    4.5.6 "301 Moved Permanently"
 
    If you get this return code and an HTML output similar to this:

       <H1>Moved Permanently</H1> The document has moved <A
       HREF="http://same_url_now_with_a_trailing_slash/">here</A>.

    it might be because you request a directory URL but without the trailing
    slash. Try the same operation again _with_ the trailing URL, or use the
    -L/--location option to follow the redirection.

  4.6. Can you tell me what error code 142 means?

  All error codes that are larger than the highest documented error code means
  that curl has exited due to a crash. This is a serious error, and we
  appreciate a detailed bug report from you that describes how we could go
  ahead and repeat this!

  4.7. How do I keep user names and passwords secret in Curl command lines?

  This problem has two sides:

  The first part is to avoid having clear-text passwords in the command line
  so that they don't appear in 'ps' outputs and similar. That is easily
  avoided by using the "-K" option to tell curl to read parameters from a
  file or stdin to which you can pass the secret info.

  To keep the passwords in your account secret from the rest of the world is
  not a task that curl addresses. You could of course encrypt them somehow to
  at least hide them from being read by human eyes, but that is not what
  anyone would call security.

  Also note that regular HTTP and FTP passwords are sent in clear across the
  network. All it takes for anyone to fetch them is to listen on the network.
  Eavesdropping is very easy.

  4.8 I found a bug!

  It is not a bug if the behavior is documented. Read the docs first.

  If it is a problem with a binary you've downloaded or a package for your
  particular platform, try contacting the person who built the package/archive
  you have.

  If there is a bug, read the BUGS document first. Then report it as described
  in there.

  4.9. Curl can't authenticate to the server that requires NTLM?

  This is supported in curl 7.10.6 or later. No earlier curl version knows
  of this magic.

  NTLM is a Microsoft proprietary protocol. Proprietary formats are evil. You
  should not use such ones.

  4.10 My HTTP request using HEAD, PUT or DELETE doesn't work!

  Many web servers allow or demand that the administrator configures the
  server properly for these requests to work on the web server.

  Some servers seem to support HEAD only on certain kinds of URLs.

  To fully grasp this, try the documentation for the particular server
  software you're trying to interact with. This is not anything curl can do
  anything about.

  4.11 Why does my HTTP range requests return the full document?

  Because the range may not be supported by the server, or the server may
  choose to ignore it and return the full document anyway.

  4.12 Why do I get "certificate verify failed" ?

  You invoke curl 7.10 or later to communicate on a https:// URL and get an
  error back looking something similar to this:

      curl: (35) SSL: error:14090086:SSL routines: 
      SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

  Then it means that curl couldn't verify that the server's certificate was
  good. Curl verifies the certificate using the CA cert bundle that comes with
  the curl installation.

  To disable the verification (which makes it act like curl did before 7.10),
  use -k. This does however enable man-in-the-middle attacks.

  If you get this failure but are having a CA cert bundle installed and used,
  the server's certificate is not signed by one of the CA's in the bundle. It
  might for example be self-signed. You then correct this problem by obtaining
  a valid CA cert for the server. Or again, decrease the security by disabling
  this check.

  Details are also in the SSLCERTS file in the release archives, found online
  here: http://curl.haxx.se/docs/sslcerts.html

5. libcurl Issues

  5.1. Is libcurl thread-safe?

  Yes.

  We have written the libcurl code specificly adjusted for multi-threaded
  programs. libcurl will use thread-safe functions instead of non-safe ones if
  your system has such.

  We would appreciate some kind of report or README file from those who have
  used libcurl in a threaded environment.

  5.2 How can I receive all data into a large memory chunk?

  [ See also the examples/getinmemory.c source ]

  You are in full control of the callback function that gets called every time
  there is data received from the remote server. You can make that callback do
  whatever you want. You do not have to write the received data to a file.

  One solution to this problem could be to have a pointer to a struct that you
  pass to the callback function. You set the pointer using the
  curl_easy_setopt(CURLOPT_FILE) function. Then that pointer will be passed to
  the callback instead of a FILE * to a file:

        /* imaginary struct */
        struct MemoryStruct {
          char *memory;
          size_t size;
        };

        /* imaginary callback function */
        size_t
        WriteMemoryCallback(void *ptr, size_t size, size_t nmemb, void *data)
        {
          register int realsize = size * nmemb;
          struct MemoryStruct *mem = (struct MemoryStruct *)data;
        
          mem->memory = (char *)realloc(mem->memory, mem->size + realsize + 1);
          if (mem->memory) {
            memcpy(&(mem->memory[mem->size]), ptr, realsize);
            mem->size += realsize;
            mem->memory[mem->size] = 0;
          }
          return realsize;
        }

  5.3 How do I fetch multiple files with libcurl?

  libcurl has excellent support for transferring multiple files. You should
  just repeatedly set new URLs with curl_easy_setopt() and then transfer it
  with curl_easy_perform(). The handle you get from curl_easy_init() is not
  only reusable, but you're even encouraged to reuse it if you can, as that
  will enable libcurl to use persistent connections.

  5.4 Does libcurl do Winsock initialization on win32 systems?

  Yes, if told to in the curl_global_init() call.

  5.5 Does CURLOPT_FILE and CURLOPT_INFILE work on win32 ?

  Yes, but you cannot open a FILE * and pass the pointer to a DLL and have
  that DLL use the FILE *. If you set CURLOPT_FILE you must also use
  CURLOPT_WRITEFUNCTION as well to set a function that writes the file, even
  if that simply writes the data to the specified FILE*. Similarly, if you use
  CURLOPT_INFILE you must also specify CURLOPT_READFUNCTION.

  (Provided by Joel DeYoung and Bob Schader)

  5.6 What about Keep-Alive or persistent connections?

  curl and libcurl have excellent support for persistent connections when
  transferring several files from the same server.  Curl will attempt to reuse
  connections for all URLs specified on the same command line/config file, and
  libcurl will reuse connections for all transfers that are made using the
  same libcurl handle.

  5.7 Link errors when building libcurl on Windows!

  You need to make sure that your project, and all the libraries (both static
  and dynamic) that it links against, are compiled/linked against the same run
  time library.

  This is determined by the /MD, /ML, /MT (and their corresponding /M?d)
  options to the command line compiler. /MD (linking against MSVCRT dll) seems
  to be the most commonly used option.

  (Provided by Andrew Francis)


6. License Issues

  Curl and libcurl are released under a MIT/X derivate license. The license is
  very liberal and should not impose a problem for your project. This section
  is just a brief summary for the cases we get the most questions. (Parts of
  this section was much enhanced by Bjorn Reese.)

  6.1. I have a GPL program, can I use the libcurl library?

  Yes!

  Since libcurl may be distributed under the MIT/X derivate license, it can be
  used together with GPL in any software.

  6.2. I have a closed-source program, can I use the libcurl library?

  Yes!

  libcurl does not put any restrictions on the program that uses the library.

  6.3. I have a BSD licensed program, can I use the libcurl library?

  Yes!

  libcurl does not put any restrictions on the program that uses the library.

  6.4. I have a program that uses LGPL libraries, can I use libcurl?

  Yes!

  The LGPL license doesn't clash with other licenses.

  6.5. Can I modify curl/libcurl for my program and keep the changes secret?

  Yes!

  The MIT/X derivate license practically allows you to do almost anything with
  the sources, on the condition that the copyright texts in the sources are
  left intact.

  6.6. Can you please change the curl/libcurl license to XXXX?

  No.

  We have carefully picked this license after years of development and
  discussions and a large amount of people have contributed with source code
  knowing that this is the license we use. This license puts the restrictions
  we want on curl/libcurl and it does not spread to other programs or
  libraries that use it. It should be possible for everyone to use libcurl or
  curl in their projects, no matter what license they already have in use.