wget.texi

wget (command line browser) source code

\input texinfo   @c -*-texinfo-*-

@c %**start of header
@setfilename wget.info
@include version.texi
@set UPDATED May 2003
@settitle GNU Wget @value{VERSION} Manual
@c Disable the monstrous rectangles beside overfull hbox-es.
@finalout
@c Use `odd' to print double-sided.
@setchapternewpage on
@c %**end of header

@iftex
@c Remove this if you don't use A4 paper.
@afourpaper
@end iftex

@c Title for man page.  The weird way texi2pod.pl is written requires
@c the preceding @set.
@set Wget Wget
@c man title Wget The non-interactive network downloader.

@dircategory Network Applications
@direntry
* Wget: (wget).         The non-interactive network downloader.
@end direntry

@ifnottex
This file documents the the GNU Wget utility for downloading network
data.

@c man begin COPYRIGHT
Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2002, 2003 Free
Software Foundation, Inc.

Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.

@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries a copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
@end ignore
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.1 or
any later version published by the Free Software Foundation; with the
Invariant Sections being ``GNU General Public License'' and ``GNU Free
Documentation License'', with no Front-Cover Texts, and with no
Back-Cover Texts.  A copy of the license is included in the section
entitled ``GNU Free Documentation License''.
@c man end
@end ifnottex

@titlepage
@title GNU Wget @value{VERSION}
@subtitle The non-interactive download utility
@subtitle Updated for Wget @value{VERSION}, @value{UPDATED}
@author by Hrvoje Nik@v{s}i@'{c} and the developers

@ignore
@c man begin AUTHOR
Originally written by Hrvoje Niksic <[e-mail]>.
@c man end
@c man begin SEEALSO
GNU Info entry for @file{wget}.
@c man end
@end ignore

@page
@vskip 0pt plus 1filll
Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2003 Free Software
Foundation, Inc.

Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.1 or
any later version published by the Free Software Foundation; with the
Invariant Sections being ``GNU General Public License'' and ``GNU Free
Documentation License'', with no Front-Cover Texts, and with no
Back-Cover Texts.  A copy of the license is included in the section
entitled ``GNU Free Documentation License''.
@end titlepage

@ifnottex
@node Top, Overview, (dir), (dir)
@top Wget @value{VERSION}

This manual documents version @value{VERSION} of GNU Wget, the freely
available utility for network downloads.

Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2003 Free Software
Foundation, Inc.

@menu
* Overview::            Features of Wget.
* Invoking::            Wget command-line arguments.
* Recursive Retrieval:: Description of recursive retrieval.
* Following Links::     The available methods of chasing links.
* Time-Stamping::       Mirroring according to time-stamps.
* Startup File::        Wget's initialization file.
* Examples::            Examples of usage.
* Various::             The stuff that doesn't fit anywhere else.
* Appendices::          Some useful references.
* Copying::             You may give out copies of Wget and of this manual.
* Concept Index::       Topics covered by this manual.
@end menu
@end ifnottex

@node Overview, Invoking, Top, Top
@chapter Overview
@cindex overview
@cindex features

@c man begin DESCRIPTION
GNU Wget is a free utility for non-interactive download of files from
the Web.  It supports @sc{http}, @sc{https}, and @sc{ftp} protocols, as
well as retrieval through @sc{http} proxies.

@c man end
This chapter is a partial overview of Wget's features.

@itemize @bullet
@item
@c man begin DESCRIPTION
Wget is non-interactive, meaning that it can work in the background,
while the user is not logged on.  This allows you to start a retrieval
and disconnect from the system, letting Wget finish the work.  By
contrast, most of the Web browsers require constant user's presence,
which can be a great hindrance when transferring a lot of data.
@c man end

@sp 1
@item
@ignore
@c man begin DESCRIPTION

@c man end
@end ignore
@c man begin DESCRIPTION
Wget can follow links in @sc{html} and @sc{xhtml} pages and create local 
versions of remote web sites, fully recreating the directory structure of 
the original site.  This is sometimes referred to as ``recursive
downloading.''  While doing that, Wget respects the Robot Exclusion
Standard (@file{/robots.txt}).  Wget can be instructed to convert the
links in downloaded @sc{html} files to the local files for offline
viewing.
@c man end

@sp 1
@item
File name wildcard matching and recursive mirroring of directories are
available when retrieving via @sc{ftp}.  Wget can read the time-stamp
information given by both @sc{http} and @sc{ftp} servers, and store it
locally.  Thus Wget can see if the remote file has changed since last
retrieval, and automatically retrieve the new version if it has.  This
makes Wget suitable for mirroring of @sc{ftp} sites, as well as home
pages.

@sp 1
@item
@ignore
@c man begin DESCRIPTION

@c man end
@end ignore
@c man begin DESCRIPTION
Wget has been designed for robustness over slow or unstable network
connections; if a download fails due to a network problem, it will
keep retrying until the whole file has been retrieved.  If the server
supports regetting, it will instruct the server to continue the
download from where it left off.
@c man end

@sp 1
@item
Wget supports proxy servers, which can lighten the network load, speed
up retrieval and provide access behind firewalls.  However, if you are
behind a firewall that requires that you use a socks style gateway, you
can get the socks library and build Wget with support for socks.  Wget
also supports the passive @sc{ftp} downloading as an option.

@sp 1
@item
Built-in features offer mechanisms to tune which links you wish to follow
(@pxref{Following Links}).

@sp 1
@item
The retrieval is conveniently traced with printing dots, each dot
representing a fixed amount of data received (1KB by default).  These
representations can be customized to your preferences.

@sp 1
@item
Most of the features are fully configurable, either through command line
options, or via the initialization file @file{.wgetrc} (@pxref{Startup
File}).  Wget allows you to define @dfn{global} startup files
(@file{/usr/local/etc/wgetrc} by default) for site settings.

@ignore
@c man begin FILES
@table @samp
@item /usr/local/etc/wgetrc
Default location of the @dfn{global} startup file.

@item .wgetrc
User startup file.
@end table
@c man end
@end ignore

@sp 1
@item
Finally, GNU Wget is free software.  This means that everyone may use
it, redistribute it and/or modify it under the terms of the GNU General
Public License, as published by the Free Software Foundation
(@pxref{Copying}).
@end itemize

@node Invoking, Recursive Retrieval, Overview, Top
@chapter Invoking
@cindex invoking
@cindex command line
@cindex arguments
@cindex nohup

By default, Wget is very simple to invoke.  The basic syntax is:

@example
@c man begin SYNOPSIS
wget [@var{option}]@dots{} [@var{URL}]@dots{}
@c man end
@end example

Wget will simply download all the @sc{url}s specified on the command
line.  @var{URL} is a @dfn{Uniform Resource Locator}, as defined below.

However, you may wish to change some of the default parameters of
Wget.  You can do it two ways: permanently, adding the appropriate
command to @file{.wgetrc} (@pxref{Startup File}), or specifying it on
the command line.

@menu
* URL Format::
* Option Syntax::
* Basic Startup Options::
* Logging and Input File Options::
* Download Options::
* Directory Options::
* HTTP Options::
* FTP Options::
* Recursive Retrieval Options::
* Recursive Accept/Reject Options::
@end menu

@node URL Format, Option Syntax, Invoking, Invoking
@section URL Format
@cindex URL
@cindex URL syntax

@dfn{URL} is an acronym for Uniform Resource Locator.  A uniform
resource locator is a compact string representation for a resource
available via the Internet.  Wget recognizes the @sc{url} syntax as per
@sc{rfc1738}.  This is the most widely used form (square brackets denote
optional parts):

@example
http://host[:port]/directory/file
ftp://host[:port]/directory/file
@end example

You can also encode your username and password within a @sc{url}:

@example
ftp://user:password@@host/path
http://user:password@@host/path
@end example

Either @var{user} or @var{password}, or both, may be left out.  If you
leave out either the @sc{http} username or password, no authentication
will be sent.  If you leave out the @sc{ftp} username, @samp{anonymous}
will be used.  If you leave out the @sc{ftp} password, your email
address will be supplied as a default password.@footnote{If you have a
@file{.netrc} file in your home directory, password will also be
searched for there.}

@strong{Important Note}: if you specify a password-containing @sc{url}
on the command line, the username and password will be plainly visible
to all users on the system, by way of @code{ps}.  On multi-user systems,
this is a big security risk.  To work around it, use @code{wget -i -}
and feed the @sc{url}s to Wget's standard input, each on a separate
line, terminated by @kbd{C-d}.

You can encode unsafe characters in a @sc{url} as @samp{%xy}, @code{xy}
being the hexadecimal representation of the character's @sc{ascii}
value.  Some common unsafe characters include @samp{%} (quoted as
@samp{%25}), @samp{:} (quoted as @samp{%3A}), and @samp{@@} (quoted as
@samp{%40}).  Refer to @sc{rfc1738} for a comprehensive list of unsafe
characters.

Wget also supports the @code{type} feature for @sc{ftp} @sc{url}s.  By
default, @sc{ftp} documents are retrieved in the binary mode (type
@samp{i}), which means that they are downloaded unchanged.  Another
useful mode is the @samp{a} (@dfn{ASCII}) mode, which converts the line
delimiters between the different operating systems, and is thus useful
for text files.  Here is an example:

@example
ftp://host/directory/file;type=a
@end example

Two alternative variants of @sc{url} specification are also supported,
because of historical (hysterical?) reasons and their widespreaded use.

@sc{ftp}-only syntax (supported by @code{NcFTP}):
@example
host:/dir/file
@end example

@sc{http}-only syntax (introduced by @code{Netscape}):
@example
host[:port]/dir/file
@end example

These two alternative forms are deprecated, and may cease being
supported in the future.

If you do not understand the difference between these notations, or do
not know which one to use, just use the plain ordinary format you use
with your favorite browser, like @code{Lynx} or @code{Netscape}.