glimpseindex.1

harvest是一个下载html网页得机器人

.TH GLIMPSEINDEX l "November 10, 1997"
.SH NAME
\fIglimpseindex 4.1\fP - index whole file systems to be searched by glimpse
.SH OVERVIEW
\fIGlimpse\fP (which stands for GLobal IMPlicit SEarch)
is a popular UNIX indexing and query system that allows you to search through
a large set of files very quickly.
Glimpseindex is the indexing program for glimpse.
Glimpse supports most of \fIagrep\fP's options 
(\fIagrep\fP is our powerful version of \fIgrep\fP) 
including approximate matching (e.g., finding misspelled words), 
Boolean queries, and even some limited forms of regular expressions.  
It is used in the same way, except that you don't have to
specify file names.
So, if you are looking for a \fIneedle\fP
anywhere in your file system, all you have to do is say
\fIglimpse needle\fR
and all lines containing \fIneedle\fP will appear preceded
by the file name.
See man glimpse for details on how to use glimpse.
.LP
Glimpseindex provides three indexing options: a tiny index (2-3% of
the total size of all files), a small index (7-8%) and a medium-size
index (20-30%).  Search times are normally better with larger indexes
(although unless files are quite large, the small index is just
about as good as the medium one).
To index all your files, you say
\fIglimpseindex ~\fR
for tiny index (where ~ stands for the home directory), 
\fIglimpseindex -o ~\fR 
for small index, and
\fIglimpseindex -b ~\fR 
for medium.
.LP
Mail [e-mail] to be added to the glimpse mailing list.
Mail [e-mail] to report bugs, ask questions, discuss tricks 
for using glimpse, etc. (this is a moderated mailing list with very little
traffic, mostly announcements).
HTML version of these manual pages can be found in
http://glimpse.cs.arizona.edu/glimpseindexhelp.html
Also, see the glimpse home pages in
http://glimpse.cs.arizona.edu/
.SH SYNOPSIS
.B glimpseindex 
[
\fB\-abEfFiInostT \-w \fInumber\fP \-dD \fIfilename(s) \-H \fIdirectory\fP 
\-M \fInumber\fP \-S \fInumber\fP\fR 
]
\fIdirectory_name[s]\fR
.SH INTRODUCTION
\fIGlimpseindex\fP 
builds an index of all text files in all
the directories specified and all their subdirectories (recursively).
It is also possible to build several separate indexes (possibly
even overlapping).
The simplest way to index your files is to say
.LP
\fIglimpseindex -o ~\fP
.LP
The index consists of several files (described in detail below),
all with the prefix \fI.glimpse_\fR stored in the user's home directory
(unless otherwise specified with the -H option).
Files with one of the following suffixes 
are not indexed: ".o", ".gz", ".Z", ".z", ".hqx", ".zip", ".tar".
(Unless the -z option is used, see below.)
In addition, glimpseindex attempts to determine whether a file
is a text file and does not index files that it thinks are not text files.
Numbers are not indexed unless the -n option is used.
It is possible to prevent specified files from being
indexed by adding their names to the .glimpse_exclude file (described below).
The -o option builds a larger index than without it
(typically about 7-8% vs. 2-3% without -o)
allowing for a faster search (1-5 times faster).
The -b builds an even larger index and allows an even faster search
some of the time (-b is helpful mostly when large files are
present).
There is an incremental indexing option \fI-f\fR, which updates an 
existing index by determining which files
have been created or modified since the index was built and
adding them to the index (see -f).
Glimpseindex is reasonably fast, taking about 20 minutes to index 
15,000 files of about 200MB (on an Dec Alpha 233) and 2-4 minutes
to update an existing index. (Your mileage may vary.)
It is also possible to increment the index by adding 
a specific file (the -a option).
.LP
Once an index is built, searching for \fIpattern\fP is as easy as saying
.LP
\fIglimpse pattern\fR
.LP
(See man glimpse for all glimpse's options and features.)
.SH "A DETAILED DESCRIPTION OF GLIMPSEINDEX"
.LP
Glimpse does not automatically index files.  You have to tell it to do
it.  This can be done manually, but a better way is to set it to run
every night.  It is probably a good idea to run glimpseindex manually
for the first time to be sure it works properly.
The following is a simple script to run glimpseindex every night.
We assume that this script is stored in a file called glimpse.script:
.LP
glimpseindex -o -t -w 5000 ~ >& .glimpse_out
.br
at -m 0300 glimpse.script
.br
(It might be interesting to collect all the outputs of glimpse by
changing >& to >>& so that the file .glimpse_out maintains a history.
In this case the file must be created before the first time >>& is used.
If you use ksh, replace '>&' with '2>&1'.)
.LP
Glimpseindex stores the names of all the files that it indexed
in the file .glimpse_filenames.
Each file is listed by its full path name as obtained at the time
the files were indexed.
For example, /usr1/udi/file1.
Glimpse uses this full name when it performs the search, so the name
must match the current name.
This may become a problem when the indexing and the search
are done from different machines (e.g., through NFS), which may cause
the path names to be different.
For example, /tmp_mnt/R/xxx/xxx/usr1/udi/file1.
(The same is true for several other .glimpse files.  See below.)
.LP
Glimpseindex does not follow symbolic links unless they are 
explicitly included in the .glimpse_include file (described below).
.LP
Glimpseindex makes an effort to identify non-text files such as
binary files, compressed files, uuencoded files, postscript files,
binhex files, etc.
These files are automatically not indexed.
In addition, all files whose names end with `.o', `.gz', `.Z', `.z', 
`.hqx', `.zip', or `.tar'
will not be indexed (unless they are specifically included
in .glimpse_include - see below).
.LP
The options for glimpseindex are as follows:
.TP
.B \-a 
adds the given file[s] and/or directories to an existing index.
Any given directory will be traversed recursively and all files will
be indexed (unless they appear in .glimpse_exclude; see below).
Using this option is generally much faster than indexing everything 
from scratch, although in rare cases the index may not be as good.  
If for some reason the index is full (which can happen
unless -o or -b are used)
glimpseindex -a will produce
an error message and will exit without changing the original index.
.TP
.B \-b
builds a medium-size index (20-30% of the size of all files),
allowing faster search.  This option forces glimpseindex to store
an exact (byte level) pointer to each occurrence of each word 
(except for some very common words
belonging to the stop list).
.TP
.B \-B
uses a hash table that is 4 times bigger (256k entries instead of 64K) 
to speed up indexing. 
The memory usage will increase typically by about 2 MB.
This option is only for indexing speed; it does not affect the final 
index.
.TP
.B \-d filename(s)
deletes the given file(s) from the index.
.TP
.B \-D filename(s)
deletes the given file(s) from the list of file names, but not
from the index.  This is much faster than -d, and the file(s) will
not be found by glimpse.  However, the index itself will not become
smaller.
.TP
.B \-E
does not run a check on file types.  Glimpse normally attempts to
exclude non-text files, but this attempt is not always perfect.
With \-E, glimpseindex indexes
all files, except those that are specifically excluded in .glimpse_exclude
and those whose file names end with one of the excluded suffixes.
.TP
.B \-f
incremental indexing.  \fIglimpseindex\fP scans all files
and adds to the index only those files that were created or modified
after the current index was built.
If there is no current index or if this procedure fails, \fIglimpseindex\fP
automatically reverts to the default mode 
(which is to index everything from scratch).
This option may create an inefficient index for several reasons, 
one of which is that deleted files are not really deleted from the index.
Unless changes are small, mostly additions, and -o is used,
we suggest to use the default mode as much as possible.
.TP
.B \-F 
Glimpseindex receives the list of files to index from standard input.
.TP
.B \-H directory
Put or update the index and all other .glimpse files (listed below)
in "directory".
The default is the home directory.
When glimpse is run, the -H option must be used to direct glimpse to this
directory, because glimpse assumes that the index is in the home
directory (see also the -H option in glimpse).
.TP
.B \-i
Make .glimpse_include (SEE GLIMPSEINDEX FILES) take precedence 
over .glimpse_exclude,
so that, for example, one can exclude everything (by putting *)
and then explicitly include files.
.TP
.B \-I
Instead of indexing, only show (print to standard out)
the list of files that would be indexed.
It is useful for filtering purposes.
("glimpseindex -I dir | glimpseindex -F" is the same as 
"glimpseindex dir".)
.TP
.B \-M x 
Tells glimpseindex to use x MB of memory for temporary tables.
The more memory you allow the faster glimpseindex will run.
The default is x=2.
The value of x must be a positive integer.
Glimpseindex will need more memory than x for other things, and 
glimpseindex may perform some 'forks', so you'll
have to experiment if you want to use this option.
WARNING:
If x is too large you may run out of swap space.
.TP
.B \-n
Index numbers as well as text.  The default is not to index numbers.
This is useful when searching for dates or other identifying numbers,
but it may make the index very large if there are lots of numbers.
In general, glimpseindex strips away any non-alphabetic character.
For example, the string abc123 will be indexed as abc if the -n option
is not used and as abc123 if it is used.
Glimpse provides warnings (in .glimpse_messages) for all files
in which more than half the words that were added to the index
from that file had digits in them (this is an attempt to identify
data files that should probably not be indexed).
One can use the .glimpse_exclude file to exclude data files or any
other files.
(See GLIMPSEINDEX FILES.)
.TP
.B \-o
Build a small index rather than tiny
(meaning 7-9% of the sizes of all files - your mileage may vary)
allowing faster search.  This option forces glimpseindex to allocate
one block per file (a block usually contains many files).
A detailed explanation of how blocks affect glimpse can be
found in the glimpse article.
(See also LIMITATIONS.)
.TP
.B \-R
Recompute .glimpse_filenames_index from .glimpse_filenames.
The file .glimpse_filenames_index speeds up processing.
Glimpseindex usually computes it automatically.
However, if for some reason one wants to change the path names
of the files listed in .glimpse_filenames, then running
glimpseindex -R recomputes .glimpse_filenames_index.
This is useful if the index is computed on one machine,
but is used on another (with the same hierarchy).
The names of the files listed in .glimpse_filenames are used
in runtime, so changing them can be done at any time in any way
(as long as just the names not the content is changed).
This is not really an option in the regular sense;  rather,
it is a program by itself,  
and it is meant as a post-processing step.
(Avaliable only from version 3.6.)
.TP
.B \-s
supports structured queries.  This option was added to support the
Harvest project and it is applicable mostly in that context.
See STRUCTURED QUERIES below for more information and also
http://harvest.transarc.com for more information
about the Harvest project.
.TP
.B \-S k
The number k determines the size of the \fIstop-list\fP.
The stop-list consists of words that are too common and are not indexed
(e.g., 'the' or 'and').
Instead of having a fixed stop-list, glimpseindex figures out the
words that are too common for every index separately.
The rules are different for the different indexing options.
The tiny index contains all words (the savings from a stop-list are
too small to bother).
The small index (-o), the number k is a percentage threshold.
A word will be in the stop list if it appears in at least k% of all files.