intro

一套客户/服务器模式的备份系统代码,跨平台,支持linux,AIX, IRIX, FreeBSD, Digital Unix (OSF1), Solaris and HP-UX.

          Introduction to AF's Backup system
          ==================================


This file gives a short introduction into AF's backup system,
it's architecture (wow !), the ideas behind it and how it can
be used to serve the individual needs.


Client-Server
-------------

It's a client-server system. The server is a machine that has
access to some kind of storage media, usually a tape drive.
This server side knows how to handle this device, how many
cartridges are in use and what to do when a cartridge gets full
or other special cases occur. What the server side does not
know is anything that has to do with directory structures, files,
attributes of the stored files or even their names. This is all
handled on the client side. The server only knows data streams.
The client demands from the server the functionality to pass it
the next bunch of data from the tape, to write a packet to tape,
to put in a certain cartridge and the server performs the actions
which it can can. The client knows how to pack and unpack the
files and directories and on which cartridges and files they are
written and can be found. For this purpose it maintains file
lists containing the names of all stored filesystem entries in a
local directory. These filelists are not saved to the backup
unless explicitly configured. Compression is done on the client
side to reduce network traffic and to achieve more safety.
If the packed stream had been compressed, a single wrong bit in
the stream would make the whole rest of the backup useless. Thus
the files are compressed individually on the client side.


Server
''''''

Server Identification

An afbackup server is addressed using one of it's hostnames and
a port number. Two services can be configured for each afbackup
server: a single stream and a multi stream server, each serving
on a separate port. It's up to the client, which service it will
use. The multi stream server can handle requests from several
clients in parallel (only for saving, not for restore), thus
multiplexing the data stream. This multiplexing requires some
overhead, so the performance of the multi stream service can be
expected worse than the single stream service. For full backups
the single stream server might be preferred, but for incremental
backups, where a client is spending most of the time searching
for modified filesystem entries, the multi stream server might
be the service to choose. Then the possibly several clients can
scan their disks in parallel, sending the data to backup to the
server as it is found.
Because the two servers are using the same (sets of) tapes, an
additional identifier is introduced to enable clients to figure
out, that they are speaking to the same server through different
services/ports. This together with the strongly recommended use
of a host alias to address the server(s), makes for being able
to move the backup service from one machine to another. Change
the host alias to the other machine, connect the storage media
to it, move over the afbackup installation and that's it. Of
course the clients must see, that the host alias has changed,
so a working name service is necessary.

Storage Devices

Any kind of streamer can be used with this backup system. The
only restriction is that it must have a block- or character-
device entry in the /dev-directory. The blocksize of the device
must be set correctly in the server-side configuration.
Besides streamers data can also be saved to filesystems.
The configuration to do this is a little special and
can be taken from Question 3 of the HOWTO.FAQ.DO-DONT.
With streamers several tape cartridges can and should be used.
There are two basic cases: Either you have a robot or you do not.
If you do not have a robot, you must change the cartridges manually.
The server supplies you with e-mail, whenever another cartridge
should be put into the drive. Having done this, a special
command (cartready) must be run on the server to let it know,
that it can continue now.
If you have a robot, but no special software (commands) to tell
it which cartridge to insert into the drive, you must set the
robot to sequential mode. That is, the robot puts the next tape
into the drive once the current one is ejected. Such a mode is
usually supported in which case the server ejects the cartridges
out of the drive until the right tape is found.
If you have commands to set the cartridge directly, you can
configure them in the configuration file.

Cartridges

The cartridges are identified by numbers beginning with 1. Once
they are in use, they automatically get a magnetic label that
they keep until overwritten with a special command (label_tape).
It is a good idea also to use adhesive labels and to write the
numbers onto the tapes, so they can't be mixed up. The server
recognizes if there is a tape in the drive that has the wrong
label. What happens then depends on the cases mentioned above.
If you have no robot, you get another email, that points you
to the problem. If you have a robot in sequential mode, the
server tries all cartridges it has and attempts to find the right
one. If it finds a tape with the right label, this is used. If it
does not, it stops and writes errors to the serverside error logfile.
If you have a robot and commands to set the cartridge, and the
robot inserts the wrong one, this is considered a severe problem
and the server stops, again writing errors to it's error log. It
then sends an email then waits for a maintainer to solve the problem
and to allow the server to continue by issuing a command.
Once all tapes of a set (for cartridge sets see the next section)
have been used up, the server tries to start over with the first
of the set, overwriting it. The server tracks what tapes are
needed for clients to restore all their data. When full, these
tapes are not overwritten until explicitly permitted or the data
on them is no longer in any index file on a client. Tapes can be
set to read-only state. The server will never write to tapes in
this mode, not even in the variable append operating mode. In
variable append mode, any tape of the correct cartridge set can
be supplied to the streamer device and it will be accepted for
writing. If the data on that tape is still needed by clients, the
next data will be appended to the area on tape already written,
otherwise the tape will be overwritten. If not in variable append
mode, the next writing position commences from where writing stopped
before. This can be overridden using the cartis command or the
client-side option -G.
More cartridges than the robot can juggle can be used. For this
case see the HOWTO.FAQ.DO-DONT Q6.

Cartridge Sets

You can devide your cartridges into sets for several purposes.
A set comprises of several cartridges, e.g. the cartridges
number 1-3. There may be cartridges, that are not in any set
and a cartridge must not be in more than one set. This would
be a configuration error. Nonetheless if you have a robot, you
must tell the backup system the number of cartridges, the robot
is handling. The configuration parameter is named Cartridge-Sets.
If you do not configure sets of cartridges, all of the available
cartridges will be used as the one and only existing set. Which
set to use, can be configured in the client side configuration
file or overriding this setting by using commandline options
(-S). Client access to tapes is handled on a per-set basis.
Restrictions can be configured evaluating the client's official
host name. If the server is unable to determine the client's
hostname, access is denied, if a restriction is configured. If
no restriction is configured, access is granted for any client.
Clients must authenticate to operate a server, so this default
behaviour is not to be considered a security issue. Attempts of
answers to the question, why to use sets, can be found in the
HOWTO.FAQ.DO-DONT Q4.

Storage units

This is a term used for the combination of a backup server's
hostname, the port number to connect and a valid cartridge set
number on this server. Several storage units can be configured
on each client. When starting a backup, the client-side first
checks the servers for availability one after the other and
starts the backup to the first available one found. Running a
restore the user or administrator does not have to remember,
to which storage unit the backup went. This is all handled
transparently.
For each of the three client-side configuration parameters
(server hostname, port number and cartridge set) several
entries are now allowed. They are associated by position.
For further information, see the HOWTO.FAQ.DO-DONT HOWTO Q9.

Read and Write

The read position can be requested by clients without any
restriction assuming the client is granted access to the tape
in question. With the write position it is different. By default
the server continues to write at the position where the most
recent write operation ended. This postition handling is maintained
individually for each cartridge set. For each set the writing
position is stored. Writing position means not only the cartridge
used before, but also the number of the file on tape. Thus
another tape is not started for each and every backup. The default
behaviour can be changed to variable append mode. In this mode
any tape in the right cartridge set with remaining space and not
in read-only state can be supplied to the streamer device and be
accepted for writing. If the tape is containing data needed by
clients for restore, the next data is appended, otherwise the
tape is reused and completely overwritten.
The data stream is written to the tape in pieces (tape files) of
a configurable maximum size. This is done to find the position
faster, where certain data has been stored. The client knows for
every file, on which cartridge the file is and in which tape file
it can be found. Thus a restore of certain requested files is much
faster than starting only one file for each backup, in the worst
case using up a whole cartridge.

Locking

The server process generates a lockfile when starting to prevent
several server processes to access the same tape drive. Nonetheless
if several processes start, the one started later waits a confi-
gurable time for the device to become available sending email
after configurable intervals. Another lock is set for access to
an autochanger if used.


Client
''''''

Full and Incremental Backup, Backup Levels

A full backup means, that everything requested is stored on the
server. An incremental backup stores only these files and direc-
tories, that have changed or are newer than the start of the
previous full or incremental backup. For determining what has
changed, the mtime in the inode is used by default. Additionally
it is possible to configure, that also the inode change time
(ctime) is evaluated. Why this might be done can be read from
the HOWTO.FAQ.DO-DONT, Q2.
Problems may arise if mechanisms are used that create files and
directories while resetting the mtime to an earlier date and time
(e.g. using tar -x or cpio -i). In these cases files are not saved
during the next incremental backup.
If a backup level is supplied, when running incr_backup, then
everything is stored that has changed since the most recent
run with a higher or equal level. Note, that other backup systems
define levels the other way round: the lower the level number,
the more is written to backup.
The full backup can be divided into several parts. This is useful,
for example, if the duration for backing up everything is too
long for the intended time frame. If the full backup should run
during the weekend, but the about 60 hours from friday evening
to monday morning are not sufficient, this is a case for several
parts. These can be configured by listing the files and directories
for each part in different client-side configuration parameters.
Before the parameter NumBackupParts must be set appropriately.

Packing and Unpacking

The client packs the files and directories and sends them to the
server or receives a data stream from the server and unpacks it.
If compression is requested, a configurable compression program
can be used to process each packed file individually. The name of
the appropriate uncompress-program is packed into the stream, so
it needs not be supplied during restore. Nonetheless it must be
ensured that compress and uncompress-programs exist within the
search path of the client processes or are supplied with the full
path.

Filelists

For each stored filesystem entry the client writes a line to a
file containing the cartridge number where it can be found for
restore, the tape file number and (since 2.9) the ID of the
user owning the filesystem entry. This listfile resides locally