theartofhttpscripting

功能最强大的网络爬虫,希望大家好好学习啊,好好研究啊

 If your proxy requires the authentication to be done using the NTLM method,
 use --proxy-ntlm, if it requires Digest use --proxy-digest.

 If you use any one these user+password options but leave out the password
 part, curl will prompt for the password interactively.

 Do note that when a program is run, its parameters might be possible to see
 when listing the running processes of the system. Thus, other users may be
 able to watch your passwords if you pass them as plain command line
 options. There are ways to circumvent this.

7. Referer

 A HTTP request may include a 'referer' field (yes it is misspelled), which
 can be used to tell from which URL the client got to this particular
 resource. Some programs/scripts check the referer field of requests to verify
 that this wasn't arriving from an external site or an unknown page. While
 this is a stupid way to check something so easily forged, many scripts still
 do it. Using curl, you can put anything you want in the referer-field and
 thus more easily be able to fool the server into serving your request.

 Use curl to set the referer field with:

        curl -e http://curl.haxx.se daniel.haxx.se

8. User Agent

 Very similar to the referer field, all HTTP requests may set the User-Agent
 field. It names what user agent (client) that is being used. Many
 applications use this information to decide how to display pages. Silly web
 programmers try to make different pages for users of different browsers to
 make them look the best possible for their particular browsers. They usually
 also do different kinds of javascript, vbscript etc.

 At times, you will see that getting a page with curl will not return the same
 page that you see when getting the page with your browser. Then you know it
 is time to set the User Agent field to fool the server into thinking you're
 one of those browsers.

 To make curl look like Internet Explorer on a Windows 2000 box:

        curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]

 Or why not look like you're using Netscape 4.73 on a Linux (PIII) box:

        curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]

9. Redirects

 When a resource is requested from a server, the reply from the server may
 include a hint about where the browser should go next to find this page, or a
 new page keeping newly generated output. The header that tells the browser
 to redirect is Location:.

 Curl does not follow Location: headers by default, but will simply display
 such pages in the same manner it display all HTTP replies. It does however
 feature an option that will make it attempt to follow the Location: pointers.

 To tell curl to follow a Location: 
 
        curl -L www.sitethatredirects.com

 If you use curl to POST to a site that immediately redirects you to another
 page, you can safely use -L and -d/-F together. Curl will only use POST in
 the first request, and then revert to GET in the following operations.

10. Cookies

 The way the web browsers do "client side state control" is by using
 cookies. Cookies are just names with associated contents. The cookies are
 sent to the client by the server. The server tells the client for what path
 and host name it wants the cookie sent back, and it also sends an expiration
 date and a few more properties.

 When a client communicates with a server with a name and path as previously
 specified in a received cookie, the client sends back the cookies and their
 contents to the server, unless of course they are expired.

 Many applications and servers use this method to connect a series of requests
 into a single logical session. To be able to use curl in such occasions, we
 must be able to record and send back cookies the way the web application
 expects them. The same way browsers deal with them.

 The simplest way to send a few cookies to the server when getting a page with
 curl is to add them on the command line like:

        curl -b "name=Daniel" www.cookiesite.com

 Cookies are sent as common HTTP headers. This is practical as it allows curl
 to record cookies simply by recording headers. Record cookies with curl by
 using the -D option like:

        curl -D headers_and_cookies www.cookiesite.com

 (Take note that the -c option described below is a better way to store
 cookies.)

 Curl has a full blown cookie parsing engine built-in that comes to use if you
 want to reconnect to a server and use cookies that were stored from a
 previous connection (or handicrafted manually to fool the server into
 believing you had a previous connection). To use previously stored cookies,
 you run curl like:

        curl -b stored_cookies_in_file www.cookiesite.com

 Curl's "cookie engine" gets enabled when you use the -b option. If you only
 want curl to understand received cookies, use -b with a file that doesn't
 exist. Example, if you want to let curl understand cookies from a page and
 follow a location (and thus possibly send back cookies it received), you can
 invoke it like:

        curl -b nada -L www.cookiesite.com

 Curl has the ability to read and write cookie files that use the same file
 format that Netscape and Mozilla do. It is a convenient way to share cookies
 between browsers and automatic scripts. The -b switch automatically detects
 if a given file is such a cookie file and parses it, and by using the
 -c/--cookie-jar option you'll make curl write a new cookie file at the end of
 an operation:

        curl -b cookies.txt -c newcookies.txt www.cookiesite.com

11. HTTPS

 There are a few ways to do secure HTTP transfers. The by far most common
 protocol for doing this is what is generally known as HTTPS, HTTP over
 SSL. SSL encrypts all the data that is sent and received over the network and
 thus makes it harder for attackers to spy on sensitive information.

 SSL (or TLS as the latest version of the standard is called) offers a
 truckload of advanced features to allow all those encryptions and key
 infrastructure mechanisms encrypted HTTP requires.

 Curl supports encrypted fetches thanks to the freely available OpenSSL
 libraries. To get a page from a HTTPS server, simply run curl like:

        curl https://that.secure.server.com

 11.1 Certificates

  In the HTTPS world, you use certificates to validate that you are the one
  you you claim to be, as an addition to normal passwords. Curl supports
  client-side certificates. All certificates are locked with a pass phrase,
  which you need to enter before the certificate can be used by curl. The pass
  phrase can be specified on the command line or if not, entered interactively
  when curl queries for it. Use a certificate with curl on a HTTPS server
  like:

        curl -E mycert.pem https://that.secure.server.com

  curl also tries to verify that the server is who it claims to be, by
  verifying the server's certificate against a locally stored CA cert
  bundle. Failing the verification will cause curl to deny the connection. You
  must then use -k in case you want to tell curl to ignore that the server
  can't be verified.

  More about server certificate verification and ca cert bundles can be read
  in the SSLCERTS document, available online here:

        http://curl.haxx.se/docs/sslcerts.html

12. Custom Request Elements

 Doing fancy stuff, you may need to add or change elements of a single curl
 request.

 For example, you can change the POST request to a PROPFIND and send the data
 as "Content-Type: text/xml" (instead of the default Content-Type) like this:

        curl -d "<xml>" -H "Content-Type: text/xml" -X PROPFIND url.com

 You can delete a default header by providing one without content. Like you
 can ruin the request by chopping off the Host: header:

        curl -H "Host:" http://mysite.com

 You can add headers the same way. Your server may want a "Destination:"
 header, and you can add it:

        curl -H "Destination: http://moo.com/nowhere" http://url.com

13. Debug

 Many times when you run curl on a site, you'll notice that the site doesn't
 seem to respond the same way to your curl requests as it does to your
 browser's.

 Then you need to start making your curl requests more similar to your
 browser's requests:

 * Use the --trace-ascii option to store fully detailed logs of the requests
   for easier analyzing and better understanding

 * Make sure you check for and use cookies when needed (both reading with -b
   and writing with -c)

 * Set user-agent to one like a recent popular browser does

 * Set referer like it is set by the browser

 * If you use POST, make sure you send all the fields and in the same order as
   the browser does it. (See chapter 4.5 above)

 A very good helper to make sure you do this right, is the LiveHTTPHeader tool
 that lets you view all headers you send and receive with Mozilla/Firefox
 (even when using HTTPS).

 A more raw approach is to capture the HTTP traffic on the network with tools
 such as ethereal or tcpdump and check what headers that were sent and
 received by the browser. (HTTPS makes this technique inefficient.)

14. References

 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
 protocol.

 RFC 2396 explains the URL syntax.

 RFC 2109 defines how cookies are supposed to work.

 RFC 1867 defines the HTTP post upload format.

 http://www.openssl.org is the home of the OpenSSL project

 http://curl.haxx.se is the home of the cURL project