webalizer.1

c语言实现的web http报文分析

.TH webalizer 1 "22-Oct-2001" "Version 2.01" "The Webalizer"
.SH NAME
webalizer - A web server log file analysis tool.
.SH SYNOPSIS
.B webalizer
[\fI option ... \fP] [\fI log-file \fP]
.PP
.B webazolver
[\fI option ... \fP] [\fI log-file \fP]
.PP
.SH DESCRIPTION
The \fIWebalizer\fP is a web server log file analysis program which produces
usage statistics in HTML format for viewing with a browser.  The results
are presented in both columnar and graphical format, which facilitates
interpretation.  Yearly, monthly, daily and hourly usage statistics are
presented, along with the ability to display usage by site, URL, referrer,
user agent (browser), username, search strings, entry/exit pages,  and
country (some information may not be available if not present in the log
file being processed).
.PP
The \fIWebalizer\fP supports \fBCLF\fP (common log format) log files,
as well as \fBCombined\fP log formats as defined by NCSA and others,
and variations of these which it attempts to handle intelligently.  In
addition, the \fIWebalizer\fP also supports \fIwu-ftpd\fP \fBxferlog\fP
formatted log files, allowing analysis of ftp servers, and
\fBsquid\fP proxy logs.  Logs may also be compressed, via \fBgzip\fP.
If a compressed log file is detected, it will be automatically uncompressed
while it is read.  Compressed logs must have the standard \fIgzip\fP
extension of \fB.gz\fP.
.PP
\fIwebazolver\fP is normally just a symbolic link to the \fIwebalizer\fP.
When run as \fIwebazolver\fP, only DNS file creation/updates are performed,
and the program will exit once complete.  All normal options and
configuration directives are available, however many will not be used.
In addition, a DNS cache file must be specified.  If the number of DNS
children processes to use are not specified, the \fIwebazolver\fP will
default to \fB5\fP.
.PP
This documentation applies to The Webalizer Version 2.01
.SH RUNNING THE WEBALIZER
The \fIWebalizer\fP was designed to be run from a Unix command line prompt or
as a \fBcrond(8)\fP job. Once executed, the general flow of the program is:
.TP 8
.B o
A default configuration file is scanned for.  A file named
\fIwebalizer.conf\fP is searched for in the current directory, and if
found, it's configuration data is parsed.  If the file is not
present in the current directory,  the file \fI/etc/webalizer.conf\fP
is searched for and, if found, is used instead.
.TP 8
.B o
Any command line arguments given to the program are parsed.  This
may include the specification of a configuration file, which is
processed at the time it is encountered.
.TP 8
.B o
If a log file was specified, it is opened and made ready for
processing.  If no log file was given, \fISTDIN\fP is used for input.
If the log filename '\fB-\fP' is specified, \fISTDIN\fP will be forced.
.TP 8
.B o
If an output directory was specified, the program does a \fBchdir(2)\fP to
that directory in prepration for generating output.  If no output
directory was given, the current directory is used.
.TP 8
.B o
If a non-zero number of DNS Children processes were specified, they will
be started, and the specified log file will be processed, creating or
updating the specified DNS cache file.
.TP 8
.B o
If no hostname was given, the program attempts to get the hostname
using a \fBuname(2)\fP system call.  If that fails, \fIlocalhost\fP
is used.
.TP 8
.B o
A history file is searched for in the current directory (output
directory) and read if found.  This file keeps totals for previous
months, which is used in the main \fIindex.html\fP HTML document.
.B Note:
The file location can now be specified with the \fIHistoryName\fP
configuration option.
.TP 8
.B o
If incremental processing was specified, a data file is searched for
and loaded if found, containing the 'internal state' data of the
program at the end of a previous run.
.B Note:
The file location can now be specified with the \fIIncrementalName\fP
configuration option.
.TP 8
.B o
Main processing begins on the log file.  If the log spans multiple
months, a seperate HTML document is created for each month.
.TP 8
.B o
After main processing, the main \fIindex.html\fP page is created, which
has totals by month and links to each months HTML document.
.TP 8
.B o
A new history file is saved to disk, which includes totals generated
by The Webalizer during the current run.
.TP 8
.B o
If incremental processing was specified, a data file is written that
contains the 'internal state' data at the end of this run.
.SH INCREMENTAL PROCESSING
Version 1.2x of The \fIWebalizer\fP adds incremental run capability.  Simply
put, this allows processing large log files by breaking them up into
smaller pieces, and processing these pieces instead.  What this means
in real terms is that you can now rotate your log files as often as you
want, and still be able to produce monthly usage statistics without the
loss of any detail.  Basically, The \fIWebalizer\fP saves and restores all
internal data in a file named \fIwebalizer.current\fP.  This allows the
program to 'start where it left off' so to speak, and allows the
preservation of detail from one run to the next.  The data file is
placed in the current output directory, and is a plain ascii text
file that can be viewed with any standard text editor.  It's location
and name may be changed using the \fIIncrementalName\fP configuration
keyword.
.PP
Some special precautions need to be taken when using the incremental
run capability of The \fIWebalizer\fP.  Configuration options should not be
changed between runs, as that could cause corruption of the internal
data stored.  For example, changing the \fIMangleAgents\fP level will cause
different representations of user agents to be stored, producing invalid
results in the user agents section of the report.  If you need to change
configuration options, do it at the end of the month after normal
processing of the previous month and before processing the current month.
You may also want to delete the \fIwebalizer.current\fP file as well.
.PP
The \fIWebalizer\fP also attempts to prevent data duplication by keeping
track of the timestamp of the last record processed.  This timestamp
is then compared to current records being processed, and any records
that were logged previous to that timestamp are ignored.  This, in
theory, should allow you to re-process logs that have already been
processed, or process logs that contain a mix of processed/not yet
processed records, and not produce duplication of statistics.  The
only time this may break is if you have duplicate timestamps in two
seperate log files... any records in the second log file that do have
the same timestamp as the last record in the previous log file processed,
will be discarded as if they had already been processed.  There are
lots of ways to prevent this however, for example, stopping the web
server before rotating logs will prevent this situation.  This setup
also necessitates that you always process logs in chronological order,
otherwise data loss will occur as a result of the timestamp compare.
.SH REVERSE DNS LOOKUPS
The Webalizer supports reverse DNS lookups through a \fIDNS cache file\fP
that is either created/updated at run-time, or has been previously
created, either by a previous run of the \fIwebalizer\fP, or by running
the stand-alone version, \fIwebazolver\fP.  In order to perform reverse
DNS lookups, a \fBDNSCache\fP filename must be specified.  In order to
create/update the cache file at run-time, the \fBDNSChildren\fP number
must be non-zero.  The \fBDNSChildren\fP value specifies the number of
children processes to fork, each of which will perform reverse DNS
lookups in order to create/update the DNS cache file.  See the file
\fIDNS.README\fP for additional information.
.SH COMMAND LINE OPTIONS
The Webalizer supports many different configuration options that will
alter the way the program behaves and generates output.  Most of these
can be specified on the command line, while some can only be specified
in a configuration file. The command line options are listed below,
with references to the corresponding configuration file keywords.
.PP
.I General Options
.TP 8
.B \-h
Display all available command line options and exit program.
.TP 8
.B \-v \-V
Display program version and exit program.
.TP 8
.B \-d
\fBDebug\fP.  Display debugging information for errors and warnings.
.TP 8
.B \-i
\fBIgnoreHist\fP.  Ignore history.  \fBUSE WITH CAUTION\fP. This
will cause The \fIWebalizer\fP to ignore any previous monthly history
file only.  Incremental data (if present) is still processed.
.TP 8
.B \-p
\fBIncremental\fP.  Preserve internal data between runs.
.TP 8
.B \-q
\fBQuiet\fP.  Supress informational messages.  Does not supress
warnings or errors.
.TP 8
.B \-Q
\fBReallyQuiet\fP.  Supress all messages including warnings and errors.
.TP 8
.B \-T
\fBTimeMe\fP.  Force display of timing information at end of processing.
.TP 8
.B \-c \fIfile\fP
Use configuration file \fIfile\fP.
.TP 8
.B \-n \fIname\fP
\fBHostName\fP.  Use the hostname \fIname\fP.
.TP 8
.B \-o \fIdir\fP
\fBOutputDir\fP.  Use output directory \fIdir\fP.
.TP 8
.B \-t \fIname\fP
\fBReportTitle\fP.  Use \fIname\fP for report title.
.TP 8
.B \-F \fP( \fBc\fPlf | \fBf\fPtp | \fBs\fPquid )
\fBLogType\fP.  Specify log type to be processed.  Value can be either
\fIc\fPlf, \fIf\fPtp or \fIs\fPquid format.  If not specified, will
default to \fBCLF\fP format.  \fIFTP\fP logs must be in standard
wu-ftpd \fIxferlog\fP format.
.TP 8
.B \-f
\fBFoldSeqErr\fP.  Fold out of sequence log records back into analysis,
by treating as if they were the same date/time as the last good record.
Normally, out of sequence log records are simply ignored.
.TP 8
.B \-Y
\fBCountryGraph\fP. Supress country graph.
.TP 8
.B \-G
\fBHourlyGraph\fP.  Supress hourly graph.
.TP 8
.B \-x \fIname\fP
\fBHTMLExtension\fP.  Defines HTML file extension to use.  If not
specified, defaults to \fIhtml\fP.  Do not include the leading
period.
.TP 8
.B \-H
\fBHourlyStats\fP.  Supress hourly statistics.
.TP 8
.B \-L
\fBGraphLegend\fP.  Supress color coded graph legends.
.TP 8
.B \-l \fInum\fP
\fBGraphLines\fP.  Specify number of background lines. Default
is 2.  Use zero ('0') to disable the lines.
.TP 8
.B \-P \fIname\fP
\fBPageType\fP.  Specify file extensions that are considered \fIpages\fP.
Sometimes referred to as \fIpageviews\fP.
.TP 8
.B \-m \fInum\fP
\fBVisitTimeout\fP.  Specify the Visit timeout period.  Specified in
number of seconds.  Default is 1800 seconds (30 minutes).
.TP 8
.B \-I \fIname\fP
\fBIndexAlias\fP.  Use the filename \fIname\fP as an additional alias
for \fIindex.\fP.
.TP 8
.B \-M \fInum\fP
\fBMangleAgents\fP.  Mangle user agent names according to the mangle
level specified by \fInum\fP.  Mangle levels are:
.TP 12
.ti 8
.B 5
Browser name and major version.
.TP 12
.ti 8
.B 4
Browser name, major and minor version.
.TP 12
.ti 8
.B 3
Browser name, major version, minor version to two decimal places.
.TP 12
.ti 8
.B 2
Browser name, major and minor versions and sub-version.
.TP 12
.ti 8
.B 1
Browser name, version and machine type if possible.
.TP 12
.ti 8
.B 0
All informaiton (left unchanged).
.TP 8
.B \-g \fInum\fP
\fBGroupDomains\fP. Automatically group sites by domain.  The
grouping level specified by \fInum\fP can be thought of as 'the
number of dots' to display in the grouping.  The default value
of \fB0\fP disables any domain grouping.
.TP 8
.B \-D \fIname\fP
\fBDNSCache\fP.  Use the DNS cache file \fIname\fP.
.TP 8
.B \-N \fInum\fP
\fBDNSChildren\fP.  Use \fInum\fP DNS children processes to perform DNS
lookups, either creating or updateing the DNS cache file.  Specify zero
(\fB0\fP) to disable cache file creation/updates.  If given, a DNS cache
filename must be specified.
.PP
.I Hide Options
.TP 8
.B \-a \fIname\fP
\fBHideAgent\fP.  Hide user agents matching \fIname\fP.
.TP 8
.B \-r \fIname\fP
\fBHideReferrer\fP.  Hide referrer matching \fIname\fP.
.TP 8
.B \-s \fIname\fP
\fBHideSite\fP.  Hide site matching \fIname\fP.
.TP 8
.B \-X \fIname\fP
\fBHideAllSites\fP.  Hide all individual sites (only display groups).
.TP 8
.B \-u \fIname\fP
\fBHideURL\fP.  Hide URL matching \fIname\fP.
.PP
.I Table size options
.TP 8
.B \-A \fInum\fP
\fBTopAgents\fP.  Display the top \fInum\fP user agents table.
.TP 8
.B \-R \fInum\fP
\fBTopReferrers\fP.  Display the top \fInum\fP referrers table.
.TP 8
.B \-S \fInum\fP
\fBTopSites\fP.  Display the top \fInum\fP sites table.
.TP 8
.B \-U \fInum\fP
\fBTopURLs\fP.  Display the top \fInum\fP URL's table.
.TP 8
.B \-C \fInum\fP
\fBTopCountries\fP.  Display the top \fInum\fP countries table.
.TP 8
.B \-e \fInum\fP
\fBTopEntry\fP.  Display the top \fInum\fP entry pages table.
.TP 8
.B \-E \fInum\fP
\fBTopExit\fP.  Display the top \fInum\fP exit pages table.
.SH CONFIGURATION FILES
Configuration files are standard \fBascii(7)\fP text files that may be created
or edited using any standard editor.  Blank lines and lines that begin
with a pound sign ('#') are ignored.  Any other lines are considered to
be configurgation lines, and have the form "Keyword Value", where the
\'Keyword\' is one of the currently available configuration keywords defined
below, and 'Value' is the value to assign to that particular option.  Any
text found after the keyword up to the end of the line is considered the
keyword's value, so you should not include anything after the actual value
on the line that is not actually part of the value being assigned.  The
file \fIsample.conf\fP provided with the distribution contains lots of useful
documentation and examples as well.

.I General Configuration Keywords
.TP 8
.B LogFile \fIname\fP
Use log file named \fIname\fP.  If none specified, \fISTDIN\fP will be used.
.TP 8
.B LogType \fIname\fP