theartofhttpscripting

harvest是一个下载html网页得机器人

5. PUT

 The perhaps best way to upload data to a HTTP server is to use PUT. Then
 again, this of course requires that someone put a program or script on the
 server end that knows how to receive a HTTP PUT stream.

 Put a file to a HTTP server with curl:

        curl -T uploadfile www.uploadhttp.com/receive.cgi

6. AUTHENTICATION

 Authentication is the ability to tell the server your username and password
 so that it can verify that you're allowed to do the request you're doing. The
 basic authentication used in HTTP is *plain* *text* based, which means it
 sends username and password only slightly obfuscated, but still fully
 readable by anyone that sniffs on the network between you and the remote
 server.

 To tell curl to use a user and password for authentication:

        curl -u name:password www.secrets.com
 
 Sometimes your HTTP access is only available through the use of a HTTP
 proxy. This seems to be especially common at various companies. A HTTP proxy
 may require its own user and password to allow the client to get through to
 the Internet. To specify those with curl, run something like:

        curl -U proxyuser:proxypassword curl.haxx.se

 If you use any one these user+password options but leave out the password
 part, curl will prompt for the password interactively.

 Do note that when a program is run, its parameters are possible to see when
 listing the running processes of the system. Thus, other users may be able to
 watch your passwords if you pass them as plain command line options. There
 are ways to circumvent this.

7. REFERER

 A HTTP request may include a 'referer' field, which can be used to tell from
 which URL the client got to this particular resource. Some programs/scripts
 check the referer field of requests to verify that this wasn't arriving from
 an external site or an unknown page. While this is a stupid way to check
 something so easily forged, many scripts still do it. Using curl, you can put
 anything you want in the referer-field and thus more easily be able to fool
 the server into serving your request.

 Use curl to set the referer field with:

        curl -e http://curl.haxx.se daniel.haxx.se

8. USER AGENT

 Very similar to the referer field, all HTTP requests may set the User-Agent
 field. It names what user agent (client) that is being used. Many
 applications use this information to decide how to display pages. Silly web
 programmers try to make different pages for users of different browsers to
 make them look the best possible for their particular browsers. They usually
 also do different kinds of javascript, vbscript etc.

 At times, you will see that getting a page with curl will not return the same
 page that you see when getting the page with your browser. Then you know it
 is time to set the User Agent field to fool the server into thinking you're
 one of those browsers.

 To make curl look like Internet Explorer on a Windows 2000 box:

        curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]

 Or why not look like you're using Netscape 4.73 on a Linux (PIII) box:

        curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]

9. REDIRECTS

 When a resource is requested from a server, the reply from the server may
 include a hint about where the browser should go next to find this page, or a
 new page keeping newly generated output. The header that tells the browser
 to redirect is Location:.

 Curl does not follow Location: headers by default, but will simply display
 such pages in the same manner it display all HTTP replies. It does however
 feature an option that will make it attempt to follow the Location: pointers.

 To tell curl to follow a Location: 
 
        curl -L www.sitethatredirects.com

 If you use curl to POST to a site that immediately redirects you to another
 page, you can safely use -L and -d/-F together. Curl will only use POST in
 the first request, and then revert to GET in the following operations.

10. COOKIES

 The way the web browsers do "client side state control" is by using
 cookies. Cookies are just names with associated contents. The cookies are
 sent to the client by the server. The server tells the client for what path
 and host name it wants the cookie sent back, and it also sends an expiration
 date and a few more properties.

 When a client communicates with a server with a name and path as previously
 specified in a received cookie, the client sends back the cookies and their
 contents to the server, unless of course they are expired.

 Many applications and servers use this method to connect a series of requests
 into a single logical session. To be able to use curl in such occasions, we
 must be able to record and send back cookies the way the web application
 expects them. The same way browsers deal with them.

 The simplest way to send a few cookies to the server when getting a page with
 curl is to add them on the command line like:

        curl -b "name=Daniel" www.cookiesite.com

 Cookies are sent as common HTTP headers. This is practical as it allows curl
 to record cookies simply by recording headers. Record cookies with curl by
 using the -D option like:

        curl -D headers_and_cookies www.cookiesite.com

 Curl has a full blown cookie parsing engine built-in that comes to use if you
 want to reconnect to a server and use cookies that were stored from a
 previous connection (or handicrafted manually to fool the server into
 believing you had a previous connection). To use previously stored cookies,
 you run curl like:

        curl -b stored_cookies_in_file www.cookiesite.com

 Curl's "cookie engine" gets enabled when you use the -b option. If you only
 want curl to understand received cookies, use -b with a file that doesn't
 exist. Example, if you want to let curl understand cookies from a page and
 follow a location (and thus possibly send back cookies it received), you can
 invoke it like:

        curl -b nada -L www.cookiesite.com

 Curl has the ability to read and write cookie files that use the same file
 format that Netscape and Mozilla do. It is a convenient way to share cookies
 between browsers and automatic scripts. The -b switch automatically detects
 if a given file is such a cookie file and parses it, and by using the
 -c/--cookie-jar option you'll make curl write a new cookie file at the end of
 an operation:

        curl -b cookies.txt -c newcookies.txt www.cookiesite.com

11. HTTPS

 There are a few ways to do secure HTTP transfers. The by far most common
 protocol for doing this is what is generally known as HTTPS, HTTP over
 SSL. SSL encrypts all the data that is sent and received over the network and
 thus makes it harder for attackers to spy on sensitive information.

 SSL (or TLS as the latest version of the standard is called) offers a
 truckload of advanced features to allow all those encryptions and key
 infrastructure mechanisms encrypted HTTP requires.

 Curl supports encrypted fetches thanks to the freely available OpenSSL
 libraries. To get a page from a HTTPS server, simply run curl like:

        curl https://that.secure.server.com

 11.1 CERTIFICATES

  In the HTTPS world, you use certificates to validate that you are the one
  you you claim to be, as an addition to normal passwords. Curl supports
  client-side certificates. All certificates are locked with a PIN-code, why
  you need to enter the unlock-code before the certificate can be used by
  curl. The PIN-code can be specified on the command line or if not, entered
  interactively when curl queries for it. Use a certificate with curl on a
  HTTPS server like:

        curl -E mycert.pem https://that.secure.server.com

12. REFERENCES

 RFC 2616 is a must to read if you want in-depth understanding of the HTTP
 protocol.

 RFC 2396 explains the URL syntax.

 RFC 2109 defines how cookies are supposed to work.

 RFC 1867 defines the HTTP post upload format.

 http://www.openssl.org is the home of the OpenSSL project

 http://curl.haxx.se is the home of the cURL project