error_handling.tex

xorp源码hg

%
% $XORP: xorp/docs/design_arch/error_handling.tex,v 1.42 2007/03/15 00:43:12 pavlin Exp $
%

\documentclass[11pt]{article}

%\usepackage[dvips]{changebar}

\usepackage{subfigure}
\usepackage{fullpage}
\usepackage{setspace}
\usepackage{times}
\usepackage{latexsym}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{xspace}
\usepackage{color}
\usepackage{amsmath}
\usepackage{rotating}
\usepackage{moreverb}
\usepackage{listings}
\usepackage{alltt}
\usepackage{stmaryrd}
%\usepackage[dvipdf]{graphics}
%\usepackage[dvips]{graphicx}
%\usepackage{xorp}

\definecolor{gray}{rgb}{0.5,0.5,0.5}
\newcommand{\etc}{\emph{etc.}\xspace}
\newcommand{\ie}{\emph{i.e.,}\xspace}
\newcommand{\eg}{\emph{e.g.,}\xspace}
%\newcommand{\comment}[1]{{\color{gray}[\textsf{#1}]}}
%\newcommand{\comment}[1]{}

\newcommand{\xorp} {{\em XORP}\@\xspace}
\newcommand{\module} {{\em module}\@\xspace}
\newcommand{\modules} {{\em modules}\@\xspace}
\newcommand{\finder} {{\em Finder}\@\xspace}
\newcommand{\xorpsh} {{\em Xorpsh}\@\xspace}
\newcommand{\cm} {{\em CM}\@\xspace}
\newcommand{\xrl} {{\em XRL}\@\xspace}
\newcommand{\xt} {{\em XRL Target}\@\xspace}
\newcommand{\rtrmgr} {{\em rtrmgr}\@\xspace}


% Changebar stuff
% \newenvironment{colorcode}{\color{blue}}{}
% \renewcommand{\cbstart}{\begin{colorcode}}
% \renewcommand{\cbend}{\end{colorcode}}

% \pagestyle{empty}

\begin{document}

\title{XORP Error Handling \\
\vspace{1ex}
Version 1.4}
\author{ XORP Project					\\
	 International Computer Science Institute	\\
	 Berkeley, CA 94704, USA			\\
         {\it http://www.xorp.org/}			\\
	 {\it [e-mail]}
}
\date{March 20, 2007}

\maketitle


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

A \xorp router is made up of a number of processes that communicate
via XRLs \cite{xorp:xrl} (a messaging system developed for \xorp). In
this document we will focus on how to deal with errors that are
generated directly or indirectly by \xrl calls, and discuss how to
handle process failures and the subsequent restart of failed
processes.  Of course, in an ideal world processes would not fail, but
when they do fail, our goals are to keep as much router
functionality working as possible, to avoid permanent inconsistencies
at all costs, and for the remainder of the functionality to be
restored as quickly as possible.

Many \xorp processes share routing state that must remain
synchronised. For example, the BGP process sends the result of its
routing decisions to the RIB process, which passes these routes on to
the FEA and hence to the forwarding engine's Forwarding Information
Base (FIB). If the RIB process fails, then BGP would lose the ability
to manipulate the FIB, and forwarding would not match the BGP routing
table.  Thus, BGP should withdraw all routes that it told its peers, or
alternatively it might drop all peerings until the RIB has
successfully restarted.

A critical component of the system is the router manager process
(\rtrmgr) which is responsible for starting and stopping routing
processes. When a \xorp process starts or terminates, that
process's XRL client library ensures that the \finder is notified. If
a process has an interest in the status of another process it can
register interest with the \finder.

In a \xorp router, as with any complex system, errors can occur. These
errors can range from a \xorp process simply failing, to an attempt to
install a route into the forwarding engine that already exists. Errors
need to be dealt with in a consistent manner. The types
of error that may occur are categorized below.

The first type of error is {\em Process Failure}.

The second type of error is {\em Communication Error}. At the most
basic level an attempt to send an \xrl has failed. The process that
was the recipient of the \xrl may have failed or be slow to respond.
The message that was being sent may have been lost in transit.

The third type of error, {\em Execution Error}, is when an XRL call
returns an error due to some underlying interaction failure. A simple
example of this type of error is a ``route add'' failing. The attempt to
add a route may fail for many reasons. The identical route may already
be present or a different route may be installed. The error may occur
due to a bug in the router code, because routing state has been
manipulated by non \xorp processes, or due to resource starvation in
the forwarding engine.

The fourth type of error, {\em Type Error}, is when an XRL call fails
because the arguments passed to an XRL are invalid. This error will
most likely be due to a version mismatch between \xorp processes. If
all the processes in a \xorp router have been built from the same
source tree this error should not occur. As we are building an
extensible router it may be the case that a process built from a
different source tree may encounter compatibility problems.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\label{pfailure}Process Failure}

A \xorp router is made up of a number of distinct processes. There are
dependencies between these processes. We define the critical
dependencies and what action to take on detecting failure.

The most critical component of a \xorp router is the \rtrmgr/\finder
process. One of the functions of this component is to start/re-start
processes. If process A is dependent on the status (\eg alive, dead,
restarted) of process B, then process A registers this interest with
the \finder. This dependency on the \rtrmgr/\finder for managing and
monitoring process liveness state means that a \xorp router cannot
survive the failure of this process. If we attempted to survive a
\finder restart, it is conceivable that, in the same time window,
another monitored process could restart, in which case the restarting
of the monitored process could be missed by the \finder. To guard
against this possible race, a \xorp process that detects the loss of
the \finder must exit. There is one exception to this rule, the
\xorpsh process, that will be discussed later in section \ref{xorpsh}.

Each process in a \xorp router is described with how it should behave
when another process in the system fails. Processes can explicitly
register interest in the status of other processes through the
\finder. If process A is dependent on the state of process B then
process A must register interest in process B.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Implementing process failure detection}

The \finder process will send keepalive messages to all processes at
thirty second intervals. If a process does not respond to a keepalive
it is considered dead. The keepalive messages are sent over a reliable
transport such as TCP. A process dying should therefore be easy to
detect.

The \rtrmgr might also be able to detect that a process has died (but
not if it is simply not responding), as it will normally receive a
SIGCHILD signal.  On discovering a process has died, the \rtrmgr will
send a hint to the \finder, which will immediately try and send a
keepalive.  Again if the process has died it should be easy to detect.

If a process is not responding to keepalives but it is still alive, it
will be marked as dead and all interested processes will be notified.
Most importantly, the \rtrmgr will be notified and it will kill the
running process and start a new process.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Actions to take on detecting process failure}

Table \ref{failure_table} indicates what action a process should take on
detecting failure in other processes~\footnote{Note that currently this
document does not describe the policy manager. Such description will be
included in the future. For all practical reasons, the policy manager is
as important as the rtrmgr/finder, even though it is running as a
separate process.}. The ``(G)'' denotes that the process should attempt
to exit gracefully. Figure \ref{failure_fig} shows the relationship
between the various processes. The thick arrows should be modelled as a
signal sent from a process dying to its dependent processes.

\begin{figure}
  \begin{center}
    \includegraphics[width=0.9\textwidth]{figs/error_dependency.eps}
    \caption{Process relationship on failure}
    \label{failure_fig}
  \end{center}
\end{figure}

\begin{table}[ht]
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
\hline
Process fails   &                 &          &      &      &      &         &      &      &      &         \\\hline
                & \rtrmgr/        & FEA      & MFEA & RIB  & IGMP & PIM     & BGP  & RIP  & OSPF & \xorpsh \\
                & \finder         &          &      &      &      &         &      &      &      &         \\\hline
\rtrmgr/        & /               & Withdraw & Exit & Exit & Exit & Exit    & Exit & Exit & Exit & Report  \\
\finder         &                 & All      &      &      &      &         &      &      &      & Problem \\
                &                 & Unicast  &      &      &      &         &      &      &      & Wait    \\
                &                 & Routes   &      &      &      &         &      &      &      &         \\
                &                 & Exit     &      &      &      &         &      &      &      &         \\\hline
FEA(*)          &  Restart        & /        & Exit & Exit & Exit & Exit    & Exit & Exit & Exit & -       \\\hline
MFEA(*)         &  Restart        & -        & /    & -    & Exit & Exit    & -    & -    & -    & -       \\\hline
RIB             &  Restart        & Withdraw & /    & -    & Exit & Exit    & Exit & Exit & Exit & -       \\
                &                 & All      &      &      & (G)  & (G)     & (G)  & (G)  & (G)  &         \\
                &                 & Unicast  &      &      &      &         &      &      &      &         \\
                &                 & Routes   &      &      &      &         &      &      &      &         \\\hline
IGMP            &  Restart        & -        & -    & -    & /    & Delete  & -    & -    & -    & -       \\
                &                 &          &      &      &      & Local   &      &      &      &         \\
                &                 &          &      &      &      & Members &      &      &      &         \\
                &                 &          &      &      &      & After   &      &      &      &         \\
                &                 &          &      &      &      & Timeout &      &      &      &         \\\hline
PIM             &  Restart        & -        & -    & -    & -    & /       & -    & -    & -    & -       \\\hline
BGP             &  Restart        & -        & -    & -    & -    & -       & /    & -    & -    & -       \\\hline
BGP             &  Restart        & -        & -    & -    & -    & -       & /    & -    & -    & -       \\\hline
RIP             &  Restart        & -        & -    & -    & -    & -       & -    & /    & -    & -       \\\hline
OSPF            &  Restart        & -        & -    & -    & -    & -       & -    & -    & /    & -       \\\hline
\xorpsh         &  Restart        & -        & -    & -    & -    & -       & -    & -    & -    & /       \\\hline
\end{tabular}
\end{center}
{\small Note(*): Typically, the MFEA would be part of the FEA process}
\caption{\label{failure_table}Action to take on detecting process failure}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{\rtrmgr/\finder - Router manager}