wget.texi

wget (command line browser) source code

@node Option Syntax, Basic Startup Options, URL Format, Invoking
@section Option Syntax
@cindex option syntax
@cindex syntax of options

Since Wget uses GNU getopts to process its arguments, every option has a
short form and a long form.  Long options are more convenient to
remember, but take time to type.  You may freely mix different option
styles, or specify options after the command-line arguments.  Thus you
may write:

@example
wget -r --tries=10 http://fly.srk.fer.hr/ -o log
@end example

The space between the option accepting an argument and the argument may
be omitted.  Instead @samp{-o log} you can write @samp{-olog}.

You may put several options that do not require arguments together,
like:

@example
wget -drc @var{URL}
@end example

This is a complete equivalent of:

@example
wget -d -r -c @var{URL}
@end example

Since the options can be specified after the arguments, you may
terminate them with @samp{--}.  So the following will try to download
@sc{url} @samp{-x}, reporting failure to @file{log}:

@example
wget -o log -- -x
@end example

The options that accept comma-separated lists all respect the convention
that specifying an empty list clears its value.  This can be useful to
clear the @file{.wgetrc} settings.  For instance, if your @file{.wgetrc}
sets @code{exclude_directories} to @file{/cgi-bin}, the following
example will first reset it, and then set it to exclude @file{/~nobody}
and @file{/~somebody}.  You can also clear the lists in @file{.wgetrc}
(@pxref{Wgetrc Syntax}).

@example
wget -X '' -X /~nobody,/~somebody
@end example

@c man begin OPTIONS

@node Basic Startup Options, Logging and Input File Options, Option Syntax, Invoking
@section Basic Startup Options

@table @samp
@item -V
@itemx --version
Display the version of Wget.

@item -h
@itemx --help
Print a help message describing all of Wget's command-line options.

@item -b
@itemx --background
Go to background immediately after startup.  If no output file is
specified via the @samp{-o}, output is redirected to @file{wget-log}.

@cindex execute wgetrc command
@item -e @var{command}
@itemx --execute @var{command}
Execute @var{command} as if it were a part of @file{.wgetrc}
(@pxref{Startup File}).  A command thus invoked will be executed
@emph{after} the commands in @file{.wgetrc}, thus taking precedence over
them.
@end table

@node Logging and Input File Options, Download Options, Basic Startup Options, Invoking
@section Logging and Input File Options

@table @samp
@cindex output file
@cindex log file
@item -o @var{logfile}
@itemx --output-file=@var{logfile}
Log all messages to @var{logfile}.  The messages are normally reported
to standard error.

@cindex append to log
@item -a @var{logfile}
@itemx --append-output=@var{logfile}
Append to @var{logfile}.  This is the same as @samp{-o}, only it appends
to @var{logfile} instead of overwriting the old log file.  If
@var{logfile} does not exist, a new file is created.

@cindex debug
@item -d
@itemx --debug
Turn on debug output, meaning various information important to the
developers of Wget if it does not work properly.  Your system
administrator may have chosen to compile Wget without debug support, in
which case @samp{-d} will not work.  Please note that compiling with
debug support is always safe---Wget compiled with the debug support will
@emph{not} print any debug info unless requested with @samp{-d}.
@xref{Reporting Bugs}, for more information on how to use @samp{-d} for
sending bug reports.

@cindex quiet
@item -q
@itemx --quiet
Turn off Wget's output.

@cindex verbose
@item -v
@itemx --verbose
Turn on verbose output, with all the available data.  The default output
is verbose.

@item -nv
@itemx --non-verbose
Non-verbose output---turn off verbose without being completely quiet
(use @samp{-q} for that), which means that error messages and basic
information still get printed.

@cindex input-file
@item -i @var{file}
@itemx --input-file=@var{file}
Read @sc{url}s from @var{file}, in which case no @sc{url}s need to be on
the command line.  If there are @sc{url}s both on the command line and
in an input file, those on the command lines will be the first ones to
be retrieved.  The @var{file} need not be an @sc{html} document (but no
harm if it is)---it is enough if the @sc{url}s are just listed
sequentially.

However, if you specify @samp{--force-html}, the document will be
regarded as @samp{html}.  In that case you may have problems with
relative links, which you can solve either by adding @code{<base
href="@var{url}">} to the documents or by specifying
@samp{--base=@var{url}} on the command line.

@cindex force html
@item -F
@itemx --force-html
When input is read from a file, force it to be treated as an @sc{html}
file.  This enables you to retrieve relative links from existing
@sc{html} files on your local disk, by adding @code{<base
href="@var{url}">} to @sc{html}, or using the @samp{--base} command-line
option.

@cindex base for relative links in input file
@item -B @var{URL}
@itemx --base=@var{URL}
When used in conjunction with @samp{-F}, prepends @var{URL} to relative
links in the file specified by @samp{-i}.
@end table

@node Download Options, Directory Options, Logging and Input File Options, Invoking
@section Download Options

@table @samp
@cindex bind() address
@cindex client IP address
@cindex IP address, client
@item --bind-address=@var{ADDRESS}
When making client TCP/IP connections, @code{bind()} to @var{ADDRESS} on
the local machine.  @var{ADDRESS} may be specified as a hostname or IP
address.  This option can be useful if your machine is bound to multiple
IPs.

@cindex retries
@cindex tries
@cindex number of retries
@item -t @var{number}
@itemx --tries=@var{number}
Set number of retries to @var{number}.  Specify 0 or @samp{inf} for
infinite retrying.  The default is to retry 20 times, with the exception
of fatal errors like ``connection refused'' or ``not found'' (404),
which are not retried.

@item -O @var{file}
@itemx --output-document=@var{file}
The documents will not be written to the appropriate files, but all will
be concatenated together and written to @var{file}.  If @var{file}
already exists, it will be overwritten.  If the @var{file} is @samp{-},
the documents will be written to standard output.  Including this option
automatically sets the number of tries to 1.

@cindex clobbering, file
@cindex downloading multiple times
@cindex no-clobber
@item -nc
@itemx --no-clobber
If a file is downloaded more than once in the same directory, Wget's
behavior depends on a few options, including @samp{-nc}.  In certain
cases, the local file will be @dfn{clobbered}, or overwritten, upon
repeated download.  In other cases it will be preserved.

When running Wget without @samp{-N}, @samp{-nc}, or @samp{-r},
downloading the same file in the same directory will result in the
original copy of @var{file} being preserved and the second copy being
named @samp{@var{file}.1}.  If that file is downloaded yet again, the
third copy will be named @samp{@var{file}.2}, and so on.  When
@samp{-nc} is specified, this behavior is suppressed, and Wget will
refuse to download newer copies of @samp{@var{file}}.  Therefore,
``@code{no-clobber}'' is actually a misnomer in this mode---it's not
clobbering that's prevented (as the numeric suffixes were already
preventing clobbering), but rather the multiple version saving that's
prevented.

When running Wget with @samp{-r}, but without @samp{-N} or @samp{-nc},
re-downloading a file will result in the new copy simply overwriting the
old.  Adding @samp{-nc} will prevent this behavior, instead causing the
original version to be preserved and any newer copies on the server to
be ignored.

When running Wget with @samp{-N}, with or without @samp{-r}, the
decision as to whether or not to download a newer copy of a file depends
on the local and remote timestamp and size of the file
(@pxref{Time-Stamping}).  @samp{-nc} may not be specified at the same
time as @samp{-N}.

Note that when @samp{-nc} is specified, files with the suffixes
@samp{.html} or (yuck) @samp{.htm} will be loaded from the local disk
and parsed as if they had been retrieved from the Web.

@cindex continue retrieval
@cindex incomplete downloads
@cindex resume download
@item -c
@itemx --continue
Continue getting a partially-downloaded file.  This is useful when you
want to finish up a download started by a previous instance of Wget, or
by another program.  For instance:

@example
wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
@end example

If there is a file named @file{ls-lR.Z} in the current directory, Wget
will assume that it is the first portion of the remote file, and will
ask the server to continue the retrieval from an offset equal to the
length of the local file.

Note that you don't need to specify this option if you just want the
current invocation of Wget to retry downloading a file should the
connection be lost midway through.  This is the default behavior.
@samp{-c} only affects resumption of downloads started @emph{prior} to
this invocation of Wget, and whose local files are still sitting around.

Without @samp{-c}, the previous example would just download the remote
file to @file{ls-lR.Z.1}, leaving the truncated @file{ls-lR.Z} file
alone.

Beginning with Wget 1.7, if you use @samp{-c} on a non-empty file, and
it turns out that the server does not support continued downloading,
Wget will refuse to start the download from scratch, which would
effectively ruin existing contents.  If you really want the download to
start from scratch, remove the file.

Also beginning with Wget 1.7, if you use @samp{-c} on a file which is of
equal size as the one on the server, Wget will refuse to download the
file and print an explanatory message.  The same happens when the file
is smaller on the server than locally (presumably because it was changed
on the server since your last download attempt)---because ``continuing''
is not meaningful, no download occurs.

On the other side of the coin, while using @samp{-c}, any file that's
bigger on the server than locally will be considered an incomplete
download and only @code{(length(remote) - length(local))} bytes will be
downloaded and tacked onto the end of the local file.  This behavior can
be desirable in certain cases---for instance, you can use @samp{wget -c}
to download just the new portion that's been appended to a data
collection or log file.

However, if the file is bigger on the server because it's been
@emph{changed}, as opposed to just @emph{appended} to, you'll end up
with a garbled file.  Wget has no way of verifying that the local file
is really a valid prefix of the remote file.  You need to be especially
careful of this when using @samp{-c} in conjunction with @samp{-r},
since every file will be considered as an "incomplete download" candidate.

Another instance where you'll get a garbled file if you try to use
@samp{-c} is if you have a lame @sc{http} proxy that inserts a
``transfer interrupted'' string into the local file.  In the future a
``rollback'' option may be added to deal with this case.

Note that @samp{-c} only works with @sc{ftp} servers and with @sc{http}
servers that support the @code{Range} header.

@cindex progress indicator
@cindex dot style
@item --progress=@var{type}
Select the type of the progress indicator you wish to use.  Legal
indicators are ``dot'' and ``bar''.

The ``bar'' indicator is used by default.  It draws an @sc{ascii} progress
bar graphics (a.k.a ``thermometer'' display) indicating the status of
retrieval.  If the output is not a TTY, the ``dot'' bar will be used by
default.

Use @samp{--progress=dot} to switch to the ``dot'' display.  It traces
the retrieval by printing dots on the screen, each dot representing a
fixed amount of downloaded data.

When using the dotted retrieval, you may also set the @dfn{style} by
specifying the type as @samp{dot:@var{style}}.  Different styles assign
different meaning to one dot.  With the @code{default} style each dot
represents 1K, there are ten dots in a cluster and 50 dots in a line.
The @code{binary} style has a more ``computer''-like orientation---8K
dots, 16-dots clusters and 48 dots per line (which makes for 384K
lines).  The @code{mega} style is suitable for downloading very large
files---each dot represents 64K retrieved, there are eight dots in a
cluster, and 48 dots on each line (so each line contains 3M).

Note that you can set the default style using the @code{progress}
command in @file{.wgetrc}.  That setting may be overridden from the
command line.  The exception is that, when the output is not a TTY, the
``dot'' progress will be favored over ``bar''.  To force the bar output,
use @samp{--progress=bar:force}.

@item -N
@itemx --timestamping
Turn on time-stamping.  @xref{Time-Stamping}, for details.

@cindex server response, print
@item -S
@itemx --server-response
Print the headers sent by @sc{http} servers and responses sent by
@sc{ftp} servers.

@cindex Wget as spider