readme.encoding

harvest是一个下载html网页得机器人

		    Content Encoding Support for libcurl

* About content encodings: 

HTTP/1.1 [RFC 2616] specifies that a client may request that a server encode
its response. This is usually used to compress a response using one of a set
of commonly available compression techniques. These schemes are `deflate' (the
zlib algorithm), `gzip' and `compress' [sec 3.5, RFC 2616]. A client requests
that the sever perform an encoding by including an Accept-Encoding header in
the request document. The value of the header should be one of the recognized
tokens `deflate', ... (there's a way to register new schemes/tokens, see sec
3.5 of the spec). A server MAY honor the client's encoding request. When a
response is encoded, the server includes a Content-Encoding header in the
response. The value of the Content-Encoding header indicates which scheme was
used to encode the data.

A client may tell a server that it can understand several different encoding
schemes. In this case the server may choose any one of those and use it to
encode the response (indicating which one using the Content-Encoding header).
It's also possible for a client to attach priorities to different schemes so
that the server knows which it prefers. See sec 14.3 of RFC 2616 for more
information on the Accept-Encoding header.

* Current support for content encoding:

Support for the 'deflate' and 'gzip' content encoding are supported by
libcurl. Both regular and chunked transfers should work fine.  The library
zlib is required for this feature. 'deflate' support was added by James
Gallagher, and support for the 'gzip' encoding was added by Dan Fandrich.

* The libcurl interface:

To cause libcurl to request a content encoding use: 

    curl_easy_setopt(curl, CURLOPT_ENCODING, <string>) 

where <string> is the intended value of the Accept-Encoding header.

Currently, libcurl only understands how to process responses that use the
"deflate" or "gzip" Content-Encoding, so the only values for CURLOPT_ENCODING
that will work (besides "identity," which does nothing) are "deflate" and
"gzip" If a response is encoded using the "compress" or methods, libcurl will
return an error indicating that the response could not be decoded.  If
<string> is NULL no Accept-Encoding header is generated.  If <string> is a
zero-length string, then an Accept-Encoding header containing all supported
encodings will be generated.

The CURLOPT_ENCODING must be set to any non-NULL value for content to be
automatically decoded.  If it is not set and the server still sends encoded
content (despite not having been asked), the data is returned in its raw form
and the Content-Encoding type is not checked.

* The curl interface:

Use the --compressed option with curl to cause it to ask servers to compress
responses using deflate.

James Gallagher <[e-mail]>
Dan Fandrich <[e-mail]>