wget-pavuk.howto

网络爬虫程序

Wget-Pavuk.HOWTO
****************

This file contains base information for wget users who are starting using
pavuk.

I am not very authoritative person to write this kind of document.I am not
regulary using wget, so some of informations contained in this document may
be confused. I just was asked to provide such kind of informations, so I am
trying to write it.

I will appreciate any corrections and contributions to this document.

* Stevo Ondrejicka


Mapping of wget options to pavuk options
----------------------------------------

It is not possible to map exectly all options, because wget have some features
which pavuk lacks and vice versa. Also some of options doesn't behave exactly
same in pavuk.

_________________________________________________________________________
| Wget                             | Pavuk                              |
|-----------------------------------------------------------------------|
| * version information                                                 |
| -V, --version                    | -v                                 |
|-----------------------------------------------------------------------|
| * options help                                                        |
| -h, --help                       | -h                                 |
|-----------------------------------------------------------------------|
| * puting job at background                                            |
| -b, --background                 | -bg                                |
|-----------------------------------------------------------------------|
| * log messages                                                        |
| -a, --append-file=FILE           | -log_file FILE                     |
| -o, --output-file=FILE           | no option to force overwrite of    |
|                                  | log file, always appending         |
|-----------------------------------------------------------------------|
| * debuging                                                            |
| -d, --debug                      | -debug -debug_level LEVEL          |
|                                  | pavuk allows to focus on interested|
|                                  | messages by selecting of debug     |
|                                  | level - html,protos,protoc,protod  |
|                                  |         procs,locks,net,misc,user  |
|                                  |         mtlock,mtthr               |
|-----------------------------------------------------------------------|
| * verbosity of messages                                               |
| -q, --quiet                      | -quiet                             |
| -v, --verbose                    | -verbose                           |
| -nv, --non-verbose               | this option doesn't have any       |
|                                  | meaning for pavuk, because pavuk   |
|                                  | have better tunning of verbosity   |
|                                  | using -debug_level option          |
|-----------------------------------------------------------------------|
| * input of starting URLs from file                                    |
| -i, --input-file=FILE            | -urls_file FILE                    |
|-----------------------------------------------------------------------|
| * force processing of all files as HTML file                          |
| -F,  --force-html                | no similar option, pavuk rely on   |
|                                  | proper MIME types sent by servers. |
|                                  | For local files pavuk tries to     |
|                                  | guess HTML files by extensions or  |
|                                  | by contents                        |
|-----------------------------------------------------------------------|
| * number of attempts to get document after fail                       |
| -t, --tries=NUMBER               | -retry NUMBER                      |
| 0 means unlimited                | 0 means unlimited                  |
|-----------------------------------------------------------------------|
| * writing document to specified file                                  |
| -O, --output-document=FILE       | -store_name FILE                   |
|-----------------------------------------------------------------------|
| * reusing of already downloaded file                                  |
| -nc, --no-clobber                | this is default pavuk behavior     |
|                                  | and can't be turned off            |
|-----------------------------------------------------------------------|
| * resuming of broken downloads                                        |
| -c, --continue                   | this is pavuk default behaviour    |
|                                  | pavuk always prefix document names |
|                                  | with ".in_" and just after download|
|                                  | is finished it renames file to     |
|                                  | regular name, this way it can      |
|                                  | properly detect and restart broken |
|                                  | downloads                          |
|-----------------------------------------------------------------------|
| * progress meter                                                      |
| --dot-style=STYLE                | pavuk have only one style of       |
|                                  | progress (numeric) and it can be   |
|                                  | only turned on/off using -progress |
|                                  | and -noprogress option (default is |
|                                  | turned off)                        |
|-----------------------------------------------------------------------|
| * synchronizing                                                       |
| -N, --timestamping               | -mode sync                         |
|-----------------------------------------------------------------------|
| * output of protocol level communication with servers                 |
| -S, --server-response            | -debug -debug_level LEVEL          |
|                                  | LEVEL protos - for server responses|
|                                  | LEVEL protoc - for pavuk requests  |
|                                  | LEVEL protod - POST data sent in   |
|                                  |                HTTP POST request   |
|                                  | you can activate multiple debug    |
|                                  | levels - for example :             |
|                                  |  -debug_level protos,protoc        |
|-----------------------------------------------------------------------|
| * checking of URLs validity                                           |
| --spider                         | something similar cand be achived  |
|                                  | by -mode remind option.             |
|                                  | it reports broken and modified     |
|                                  | links not everything               |
|                                  | you must use also -remind_cmd      |
|                                  | option to set program which will   |
|                                  | process output (default is mailx)  |
|-----------------------------------------------------------------------| 
| * communication timeout                                               |
| -T, --timeout=SECONDS            | -timeout MINUTES                   |
|-----------------------------------------------------------------------|
| * delaying of transfers                                               |
| -w, --wait=SECONDS               | -sleep SECONDS                     |
|-----------------------------------------------------------------------|
| * enabling/disabling connections through proxy                        |
| -Y, --proxy=on/off               | no similar option, if proxy is     |
|                                  | specified, it is always used       |
|-----------------------------------------------------------------------|
| * retrieval quota                                                     |
| -Q, --quota=NUMBER               | -trans_quota NUMBER                |
| wget stops download just after   | pavuk stops download right after   |
| file download is finished after  | transfer quota is exeede (also in  |
| quota is exeeded                 | midle of transfer)                 |