readme.db

一个C语言写的快速贝叶斯垃圾邮件过滤工具

BERKELEY DB ENVIRONMENT CODE
============================

$Id: README.db,v 1.37 2006/01/13 01:03:36 m-a Exp $

This document does not apply when you are installing a bogofilter
version that has been configured to use the QDBM or SQLite3
database libraries. Common entry points are these

QUICK LINKS:
------------
Upgrade of Berkeley DB or operating system    section 2.6
Switching transactions on or off              section 2.2.1 or 2.2.2
Recover broken database                       section 3.2 or 3.3

0. Definitions ---------------------------------------------------------

Whenever ~/.bogofilter appears in the text below, this is the directory
where bogofilter keeps its data base by default. If you are overriding
this directory by configuration or environment variables, replace your
actual bogofilter data base directory.

1. Overview ------------------------------------------------------------

Operating bogofilter with a Berkeley DB back-end requires some
attention. Bogofilter offers two operating modes that can be switched on
or off a directory-by-directory basis:

  - the traditional, simple Berkeley DB Data Store

    easy to use, but also easy to corrupt, an interrupted registration
    of spam or non-spam mail, an application crash, a system crash, a
    disk that runs out of space, a user that runs out of quota.

  - the Berkeley DB Transactional Data Store,
    transactional, TXN or XA in bogofilter jargon for short

    a bit more complex to use, but pretty crash-proof as long as it is
    not abused. Upgrading bogofilter, Berkeley DB, copying around
    databases and backups require special considerations given in this
    document.

Note that the system administrator can disable either mode of operation
at compile time; in the default install, either will be present.

2. Prerequisites and Caveats -------------------------------------------

2.1 Compatibility, Berkeley DB versions

These versions are supported (with all Sleepycat-posted patches applied
- if using a pre-packaged Berkeley DB version, the packager should have
applied the patches, check your vendor's update site regularly):

  Sleepycat Software: Berkeley DB 3.1.17: (July     31, 2000)
  Sleepycat Software: Berkeley DB 3.2.9:  (January  24, 2001)
  Sleepycat Software: Berkeley DB 3.3.11: (July     12, 2001)
  Sleepycat Software: Berkeley DB 4.0.14: (November 18, 2001)
  Sleepycat Software: Berkeley DB 4.1.25: (December 19, 2002)
  Sleepycat Software: Berkeley DB 4.2.52: (December  3, 2003)
  Sleepycat Software: Berkeley DB 4.3.29: (September 6, 2005)
  Sleepycat Software: Berkeley DB 4.4.20: (January  10, 2006)

Other versions of Berkeley DB between the first and last listed above
may or may not work but usually they will.

Berkeley DB versions 4.1 and newer are recommended over the previous
versions, because the newer can detect data corruptions more reliably
(through the use of checksums that detect partially written data base
pages); Berkeley DB 4.2 and 4.3 appear a bit faster under load than 4.1
and older versions, 4.4 has not yet been evaluated.

2.2.1 Upgrading to transactional databases, also from older bogofilter versions

NOTE: for updates of Berkeley DB itself, see section 2.6!

Bogofilter should transparently upgrade the existing data base to the
new transactional data base, but going the reverse way requires manual
intervention, see section 2.2.2 below.

For enhanced reliability (only available with Berkeley DB 4.1 or newer),
it is recommended that you dump and reload the database once so that
Berkeley DB adds page checksums.  Skip this procedure for Berkeley DB
versions 4.0 or older.  For 4.1 and newer, use these commands:

  cd ~/.bogofilter
  bogoutil -d wordlist.db > wordlist.txt
  mv wordlist.db wordlist.db.old
  bogoutil --db-transaction=yes -l wordlist.db < wordlist.txt

And if all commands succeeded: rm wordlist.txt wordlist.db.old

2.2.2 Downgrading to non-transactional database

To downgrade a transactional database to a non-transactional, a
dump/reload cycle is required. These commands should do the job:

  cd ~/.bogofilter
  bogoutil -d wordlist.db > wordlist.txt
  mv wordlist.db wordlist.db.old
  rm -f log.?????????? __db.???
  bogoutil --db-transaction=no -l wordlist.db < wordlist.txt

2.3 Recoverability

The ability to recover the data base after a crash (power failure!)
depends on data being written to the disk (or a battery-backed write
cache) _immediately_ rather than delayed to be written later.

Common disk drives in current PCs and MACs are of the ATA or SATA kind
and usually ship with their write cache enabled. They write fast, but
can lose or corrupt up to a few MB of data when the power fails.
Note: This problem is not specific to bogofilter.

It is possible to sacrifice a bit bit of the the write speed and get
reliability in turn, by switching off the disk's write cache (see
appendix A for instructions).

Switching the write cache off may however adversely affect the
performance below acceptable levels, particularly for large writes such
as recording live audio or video data to hard disk.
If performance is degraded too much, consider getting a separate disk
drive and using one for fast writes (with the write cache on) and one
for reliable writes (with the write cache off, for bogofilter, mail
servers and other applications that need survive a power loss without
data loss).

2.4 Choosing a file system

If your computer saves the data on its own disk drive (a "local file
system"), Berkeley DB should work fine. Such file systems are ext2, ext3,
ffs, jfs, hfs, hfs+, reiserfs, ufs, xfs.

Berkeley DB Transactional and Concurrent data stores do not work
reliably with a networked file system for various technical reasons.
AFS, CIFS, Coda, NFS, SMBFS fall into this category.

Strictly speaking, with Berkeley DB 4.0 and older versions, the data base
block size must be written atomically. The bogofilter maintainers are
not currently aware of a file system that meets this requirement and is
production quality at the same time.

2.5 Making a snapshot backup

The transactional data store is no good if the disk drive has become
inaccessible (which happens after some months or years with every
drive), so you _must_ back up your data base regularly (see the
db_archive utility manual for additional documentation of a "hot"
backup), bogofilter cannot, of course, guess data that got lost through
a hard drive fault.

When copying or archiving directory contents, be sure to copy or archive
the *.db files FIRST, BEFORE archiving/copying the log.* files, this
is needed to let the database copy or archive remain recoverable.

You can use the bf_tar script for convenience. It requires the pax
utility (UNIX standard for over a decade now) and writes a tar archive
to stdout, and it can optionally remove inactive log files before or
after writing the tarball.

Run bf_tar without arguments to see its synopsis.

2.6 Updating Berkeley DB version underneath bogofilter

When upgrading the Berkeley DB library to a new version, or recompiling
bogofilter to use a newer version, two things in the on-disk data format
can change, generally speaking: the database format (we use the Btree
access method), the log file format, or both.

You need a "log file upgrade" for your transactional databases if at
least one of these conditions is true ("-->" means "to")

- you upgraded Berkeley DB from a 3.X version --> a 4.Y version
- you upgraded Berkeley DB from 4.0 or 4.1 --> 4.2, 4.3 or 4.4
- you upgraded Berkeley DB from 4.2 --> 4.3 or 4.4
- you upgraded Berkeley DB from 4.3 --> 4.4

Non-transactional databases do not need log file format upgrades as they
do not use log files.

If you need a log file upgrade, the upgrade procedure is:
(NOTE: DO NOT UPGRADE BERKELEY DB OR BOGOFILTER UNTIL STEP 4!)

  1. shut down your mail system,
  2. run 'bogoutil --db-remove-environment ~/.bogofilter' (for each user!)
     (this will automatically recover the environment)
  3. archive the database for catastrophic recovery (make a backup)
  4. install the new Berkeley DB version, recompile bogofilter (unless
     using a binary package), install the new bogofilter
  5. start your mail system.

If you've been using Berkeley DB 3.0 (only supported with bogofilter
versions 0.17.2 and older) and are about to update to any newer 3.X or
4.X version, you need a database format upgrade. You can either:

- dump the wordlists to a text file, then update Berkeley
  DB and bogofilter, remove the wordlists and load them from the dump.
  The procedure is the same as in section 2.2.1 or 2.2.2 (depending on
  whether you want to use transactions or not).

or

- use the db_upgrade utility to upgrade the databases in place
  (this is dangerous and must not be interrupted, backup first!)

3. Use and troubleshooting ---------------------------------------------

3.1 LOG FILE HANDLING

This section only applies to transactional databases.

The Berkeley DB Transactional Data Store uses log files to store data
base changes so that they can be rolled back or restored after an
application crash.

The logs of the transactional data store, log.NNNNNNNNNN files of up to
1 MB in size (in the default configuration), can consume considerable
amounts of disk space. Bogofilter therefore removes log files that are
no longer part of active transactions by default.

This automatic removal of log files can be disabled by either using
--db-log-autoremove=no on the command line, or by configuring
db_log_autoremove=no in the configuration file.  If automatic removal is
disabled, it can manually be triggered by running
'bogoutil --db-prune ~/.bogofilter'.

Note that removing log files makes catastrophic recovery impossible
without backups, so you *must* make snapshot
backups that contain both the *.db and log.* files (in this order!). See
section 2.5 for details.

Referral: Berkeley DB's db_archive documentation contains suggestions
for several backup strategies.

3.2 RECOVERY AND FAILED RECOVERY, FOR TRANSACTIONAL DATABASES ONLY

The recovery procedures should be tried in the order shown in this
section. If you aren't willing to experiment much, but have kept
sufficient spam and ham that you can easily and quickly retrain
bogofilter from scratch, read only sections 3.2.1 and 3.2.5, skipping
subsections 3.2.2 to 3.2.4.

3.2.1 Regular recovery

Bogofilter and related bogo* utilities will automatically detect when
regular recovery is needed and run it. This process is transparent,
the user will usually be aware this happens.

This process needs the *.db file and the corresponding _active_ log
files.

It is possible to trigger regular recovery by running
bogoutil --db-recover ~/.bogofilter
although this should not be needed.

If this fails, remove the __db.*, *.db and log.* files,
restore from the latest snapshot backup (see section 2.5) and force
recovery as shown in the previous paragraph.

3.2.2 Catastrophic recovery

If regular recovery fails after severe damage to hardware, filesystem,
database files, you can attempt to run catastrophic recovery. If log
files have been damaged, catastrophic recovery may not work.

This may need *all* log files from the backup and is therefore not
available if log files have been pruned.

To run catastrophic recovery, replace the log files from your archives,
then run:
bogoutil --db-recover-harder ~/.bogofilter

If this fails, read on.

3.2.3 Last-resort recovery method #1: nuke the environment

This recovery methods do not guarantee you are getting all of your
database, you may already have lost part or all of your data when this
is required, and you may lose recent updates to the database and corrupt
it. Only attempt this methods if the regular and catastrophic recovery
methods have failed or were unavailable.

To use this method:

  1. remove the __db.* and log.* files
  2. run: bogoutil -v --db-verify ~/.bogofilter/wordlist.db
  3a. if that printed OK, watch carefully if bogofilter performs to 
      the standards you are used to.
  3b. if verify failed, read the next section.

3.2.4 Last-resort recovery method #2: salvage the raw .db file

This recovery methods do not guarantee you are getting all of your
database, you may already have lost part or all of your data when this
is required, and you may lose recent updates to the database and corrupt 
it. Only attempt this methods if the regular and catastrophic recovery 
methods have failed or were unavailable.

To try this method:

  # salvage raw data
  db_dump -r ~/.bogofilter/wordlist.db > ~/.bogofilter/wordlist.saved
  rm ~/.bogofilter/{__db.*,log.*,wordlist.db}
  db_load ~/.bogofilter/wordlist.db < ~/.bogofilter/wordlist.saved
  # convert to transactional store
  bogoutil -d ~/.bogofilter/wordlist.db > ~/.bogofilter/wordlist.saved
  rm ~/.bogofilter/{__db.*,log.*,wordlist.db}
  bogoutil --db-transaction=yes -l ~/.bogofilter/wordlist.db \
    < ~/.bogofilter/wordlist.saved

3.2.5 No recovery possible?

We're sorry. This should happen really rarely. There's nothing left to
try, so you need to retrain from scratch.

First, remove the database and environment, type:

  rm ~/.bogofilter/{__db.*,log.*,wordlist.db}

Then retrain from scratch with the usual bogofilter -n and bogofilter -s
commands.

3.3 RECOVERY OF TRADITIONAL (NON-TRANSACTIONAL) DATABASES