wget.texi

wget (command line browser) source code

presenting a broken link.  The fact that the former links are converted
to relative links ensures that you can move the downloaded hierarchy to
another directory.

Note that only at the end of the download can Wget know which links have
been downloaded.  Because of that, the work done by @samp{-k} will be
performed at the end of all the downloads.

@cindex backing up converted files
@item -K
@itemx --backup-converted
When converting a file, back up the original version with a @samp{.orig}
suffix.  Affects the behavior of @samp{-N} (@pxref{HTTP Time-Stamping
Internals}).

@item -m
@itemx --mirror
Turn on options suitable for mirroring.  This option turns on recursion
and time-stamping, sets infinite recursion depth and keeps @sc{ftp}
directory listings.  It is currently equivalent to
@samp{-r -N -l inf -nr}.

@cindex page requisites
@cindex required images, downloading
@item -p
@itemx --page-requisites
This option causes Wget to download all the files that are necessary to
properly display a given @sc{html} page.  This includes such things as
inlined images, sounds, and referenced stylesheets.

Ordinarily, when downloading a single @sc{html} page, any requisite documents
that may be needed to display it properly are not downloaded.  Using
@samp{-r} together with @samp{-l} can help, but since Wget does not
ordinarily distinguish between external and inlined documents, one is
generally left with ``leaf documents'' that are missing their
requisites.

For instance, say document @file{1.html} contains an @code{<IMG>} tag
referencing @file{1.gif} and an @code{<A>} tag pointing to external
document @file{2.html}.  Say that @file{2.html} is similar but that its
image is @file{2.gif} and it links to @file{3.html}.  Say this
continues up to some arbitrarily high number.

If one executes the command:

@example
wget -r -l 2 http://@var{site}/1.html
@end example

then @file{1.html}, @file{1.gif}, @file{2.html}, @file{2.gif}, and
@file{3.html} will be downloaded.  As you can see, @file{3.html} is
without its requisite @file{3.gif} because Wget is simply counting the
number of hops (up to 2) away from @file{1.html} in order to determine
where to stop the recursion.  However, with this command:

@example
wget -r -l 2 -p http://@var{site}/1.html
@end example

all the above files @emph{and} @file{3.html}'s requisite @file{3.gif}
will be downloaded.  Similarly,

@example
wget -r -l 1 -p http://@var{site}/1.html
@end example

will cause @file{1.html}, @file{1.gif}, @file{2.html}, and @file{2.gif}
to be downloaded.  One might think that:

@example
wget -r -l 0 -p http://@var{site}/1.html
@end example

would download just @file{1.html} and @file{1.gif}, but unfortunately
this is not the case, because @samp{-l 0} is equivalent to
@samp{-l inf}---that is, infinite recursion.  To download a single @sc{html}
page (or a handful of them, all specified on the command-line or in a
@samp{-i} @sc{url} input file) and its (or their) requisites, simply leave off
@samp{-r} and @samp{-l}:

@example
wget -p http://@var{site}/1.html
@end example

Note that Wget will behave as if @samp{-r} had been specified, but only
that single page and its requisites will be downloaded.  Links from that
page to external documents will not be followed.  Actually, to download
a single page and all its requisites (even if they exist on separate
websites), and make sure the lot displays properly locally, this author
likes to use a few options in addition to @samp{-p}:

@example
wget -E -H -k -K -p http://@var{site}/@var{document}
@end example

To finish off this topic, it's worth knowing that Wget's idea of an
external document link is any URL specified in an @code{<A>} tag, an
@code{<AREA>} tag, or a @code{<LINK>} tag other than @code{<LINK
REL="stylesheet">}.

@cindex @sc{html} comments
@cindex comments, @sc{html}
@item --strict-comments
Turn on strict parsing of @sc{html} comments.  The default is to terminate
comments at the first occurrence of @samp{-->}.

According to specifications, @sc{html} comments are expressed as @sc{sgml}
@dfn{declarations}.  Declaration is special markup that begins with
@samp{<!} and ends with @samp{>}, such as @samp{<!DOCTYPE ...>}, that
may contain comments between a pair of @samp{--} delimiters.  @sc{html}
comments are ``empty declarations'', @sc{sgml} declarations without any
non-comment text.  Therefore, @samp{<!--foo-->} is a valid comment, and
so is @samp{<!--one-- --two-->}, but @samp{<!--1--2-->} is not.

On the other hand, most @sc{html} writers don't perceive comments as anything
other than text delimited with @samp{<!--} and @samp{-->}, which is not
quite the same.  For example, something like @samp{<!------------>}
works as a valid comment as long as the number of dashes is a multiple
of four (!).  If not, the comment technically lasts until the next
@samp{--}, which may be at the other end of the document.  Because of
this, many popular browsers completely ignore the specification and
implement what users have come to expect: comments delimited with
@samp{<!--} and @samp{-->}.

Until version 1.9, Wget interpreted comments strictly, which resulted in
missing links in many web pages that displayed fine in browsers, but had
the misfortune of containing non-compliant comments.  Beginning with
version 1.9, Wget has joined the ranks of clients that implements
``naive'' comments, terminating each comment at the first occurrence of
@samp{-->}.

If, for whatever reason, you want strict comment parsing, use this
option to turn it on.
@end table

@node Recursive Accept/Reject Options,  , Recursive Retrieval Options, Invoking
@section Recursive Accept/Reject Options

@table @samp
@item -A @var{acclist} --accept @var{acclist}
@itemx -R @var{rejlist} --reject @var{rejlist}
Specify comma-separated lists of file name suffixes or patterns to
accept or reject (@pxref{Types of Files} for more details).

@item -D @var{domain-list}
@itemx --domains=@var{domain-list}
Set domains to be followed.  @var{domain-list} is a comma-separated list
of domains.  Note that it does @emph{not} turn on @samp{-H}.

@item --exclude-domains @var{domain-list}
Specify the domains that are @emph{not} to be followed.
(@pxref{Spanning Hosts}).

@cindex follow FTP links
@item --follow-ftp
Follow @sc{ftp} links from @sc{html} documents.  Without this option,
Wget will ignore all the @sc{ftp} links.

@cindex tag-based recursive pruning
@item --follow-tags=@var{list}
Wget has an internal table of @sc{html} tag / attribute pairs that it
considers when looking for linked documents during a recursive
retrieval.  If a user wants only a subset of those tags to be
considered, however, he or she should be specify such tags in a
comma-separated @var{list} with this option.

@item -G @var{list}
@itemx --ignore-tags=@var{list}
This is the opposite of the @samp{--follow-tags} option.  To skip
certain @sc{html} tags when recursively looking for documents to download,
specify them in a comma-separated @var{list}.  

In the past, the @samp{-G} option was the best bet for downloading a
single page and its requisites, using a command-line like:

@example
wget -Ga,area -H -k -K -r http://@var{site}/@var{document}
@end example

However, the author of this option came across a page with tags like
@code{<LINK REL="home" HREF="/">} and came to the realization that
@samp{-G} was not enough.  One can't just tell Wget to ignore
@code{<LINK>}, because then stylesheets will not be downloaded.  Now the
best bet for downloading a single page and its requisites is the
dedicated @samp{--page-requisites} option.

@item -H
@itemx --span-hosts
Enable spanning across hosts when doing recursive retrieving
(@pxref{Spanning Hosts}).

@item -L
@itemx --relative
Follow relative links only.  Useful for retrieving a specific home page
without any distractions, not even those from the same hosts
(@pxref{Relative Links}).

@item -I @var{list}
@itemx --include-directories=@var{list}
Specify a comma-separated list of directories you wish to follow when
downloading (@pxref{Directory-Based Limits} for more details.)  Elements
of @var{list} may contain wildcards.

@item -X @var{list}
@itemx --exclude-directories=@var{list}
Specify a comma-separated list of directories you wish to exclude from
download (@pxref{Directory-Based Limits} for more details.)  Elements of
@var{list} may contain wildcards.

@item -np
@item --no-parent
Do not ever ascend to the parent directory when retrieving recursively.
This is a useful option, since it guarantees that only the files
@emph{below} a certain hierarchy will be downloaded.
@xref{Directory-Based Limits}, for more details.
@end table

@c man end

@node Recursive Retrieval, Following Links, Invoking, Top
@chapter Recursive Retrieval
@cindex recursion
@cindex retrieving
@cindex recursive retrieval

GNU Wget is capable of traversing parts of the Web (or a single
@sc{http} or @sc{ftp} server), following links and directory structure.
We refer to this as to @dfn{recursive retrieval}, or @dfn{recursion}.

With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from
the given @sc{url}, documents, retrieving the files the @sc{html}
document was referring to, through markup like @code{href}, or
@code{src}.  If the freshly downloaded file is also of type
@code{text/html} or @code{application/xhtml+xml}, it will be parsed and 
followed further.

Recursive retrieval of @sc{http} and @sc{html} content is
@dfn{breadth-first}.  This means that Wget first downloads the requested
@sc{html} document, then the documents linked from that document, then the
documents linked by them, and so on.  In other words, Wget first
downloads the documents at depth 1, then those at depth 2, and so on
until the specified maximum depth.

The maximum @dfn{depth} to which the retrieval may descend is specified
with the @samp{-l} option.  The default maximum depth is five layers.

When retrieving an @sc{ftp} @sc{url} recursively, Wget will retrieve all
the data from the given directory tree (including the subdirectories up
to the specified depth) on the remote server, creating its mirror image
locally.  @sc{ftp} retrieval is also limited by the @code{depth}
parameter.  Unlike @sc{http} recursion, @sc{ftp} recursion is performed
depth-first.

By default, Wget will create a local directory tree, corresponding to
the one found on the remote server.

Recursive retrieving can find a number of applications, the most
important of which is mirroring.  It is also useful for @sc{www}
presentations, and any other opportunities where slow network
connections should be bypassed by storing the files locally.

You should be warned that recursive downloads can overload the remote
servers.  Because of that, many administrators frown upon them and may
ban access from your site if they detect very fast downloads of big
amounts of content.  When downloading from Internet servers, consider
using the @samp{-w} option to introduce a delay between accesses to the
server.  The download will take a while longer, but the server
administrator will not be alarmed by your rudeness.

Of course, recursive download may cause problems on your machine.  If
left to run unchecked, it can easily fill up the disk.  If downloading
from local network, it can also take bandwidth on the system, as well as
consume memory and CPU.

Try to specify the criteria that match the kind of download you are
trying to achieve.  If you want to download only one page, use
@samp{--page-requisites} without any additional recursion.  If you want
to download things under one directory, use @samp{-np} to avoid
downloading things from other directories.  If you want to download all
the files from one directory, use @samp{-l 1} to make sure the recursion
depth never exceeds one.  @xref{Following Links}, for more information
about this.

Recursive retrieval should be used with care.  Don't say you were not
warned.

@node Following Links, Time-Stamping, Recursive Retrieval, Top
@chapter Following Links
@cindex links
@cindex following links

When retrieving recursively, one does not wish to retrieve loads of
unnecessary data.  Most of the time the users bear in mind exactly what
they want to download, and want Wget to follow only specific links.

For example, if you wish to download the music archive from
@samp{fly.srk.fer.hr}, you will not want to download all the home pages
that happen to be referenced by an obscure part of the archive.

Wget possesses several mechanisms that allows you to fine-tune which
links it will follow.

@menu
* Spanning Hosts::         (Un)limiting retrieval based on host name.
* Types of Files::         Getting only certain files.
* Directory-Based Limits:: Getting only certain directories.
* Relative Links::         Follow relative links only.
* FTP Links::              Following FTP links.
@end menu

@node Spanning Hosts, Types of Files, Following Links, Following Links
@section Spanning Hosts
@cindex spanning hosts
@cindex hosts, spanning

Wget's recursive retrieval normally refuses to visit hosts different
than the one you specified on the command line.  This is a reasonable
default; without it, every retrieval would have the potential to turn
your Wget into a small version of google.

However, visiting different hosts, or @dfn{host spanning,} is sometimes
a useful option.  Maybe the images are served from a different server.
Maybe you're mirroring a site that consists of pages interlinked between
three servers.  Maybe the server has two equivalent names, and the @sc{html}
pages refer to both interchangeably.

@table @asis
@item Span to any host---@samp{-H}

The @samp{-H} option turns on host spanning, thus allowing Wget's
recursive run to visit any host referenced by a link.