2.t

早期freebsd实现

.\" Copyright (c) 1993
.\"	The Regents of the University of California.  All rights reserved.
.\"
.\" This document is derived from software contributed to Berkeley by
.\" Rick Macklem at The University of Guelph.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\" 3. All advertising materials mentioning features or use of this software
.\"    must display the following acknowledgement:
.\"	This product includes software developed by the University of
.\"	California, Berkeley and its contributors.
.\" 4. Neither the name of the University nor the names of its contributors
.\"    may be used to endorse or promote products derived from this software
.\"    without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\"	@(#)2.t	8.1 (Berkeley) 6/8/93
.\"
.sh 1 "Not Quite NFS, Crash Tolerant Cache Consistency for NFS"
.pp
Not Quite NFS (NQNFS) is an NFS like protocol designed to maintain full cache
consistency between clients in a crash tolerant manner.
It is an adaptation of the NFS protocol such that the server supports both NFS
and NQNFS clients while maintaining full consistency between the server and
NQNFS clients.
This section borrows heavily from work done on Spritely-NFS [Srinivasan89],
but uses Leases [Gray89] to avoid the need to recover server state information
after a crash.
The reader is strongly encouraged to read these references before
trying to grasp the material presented here.
.sh 2 "Overview"
.pp
The protocol maintains cache consistency by using a somewhat
Sprite [Nelson88] like protocol,
but is based on short term leases\** instead of hard state information
about open files.
.(f
\** A lease is a ticket permitting an activity that is
valid until some expiry time.
.)f
The basic principal is that the protocol will disable client caching of a
file whenever that file is write shared\**.
.(f
\** Write sharing occurs when at least one client is modifying a file while
other client(s) are reading the file.
.)f
Whenever a client wishes to cache data for a file it must hold a valid lease.
There are three types of leases: read caching, write caching and non-caching.
The latter type requires that all file operations be done synchronously with
the server via. RPCs.
A read caching lease allows for client data caching, but no file modifications
may be done.
A write caching lease allows for client caching of writes,
but requires that all writes be pushed to the server when the lease expires.
If a client has dirty buffers\**
.(f
\** Cached write data is not yet pushed (written) to the server.
.)f
when a write cache lease has almost expired, it will attempt to
extend the lease but is required to push the dirty buffers if extension fails.
A client gets leases by either doing a \fBGetLease RPC\fR or by piggybacking
a \fBGetLease Request\fR onto another RPC. Piggybacking is supported for the
frequent RPCs Getattr, Setattr, Lookup, Readlink, Read, Write and Readdir
in an effort to minimize the number of \fBGetLease RPCs\fR required.
All leases are at the granularity of a file, since all NFS RPCs operate on
individual files and NFS has no intrinsic notion of a file hierarchy.
Directories, symbolic links and file attributes may be read cached but
are not write cached.
The exception here is the attribute file_size, which is updated during cached
writing on the client to reflect a growing file.
.pp
It is the server's responsibility to ensure that consistency is maintained
among the NQNFS clients by disabling client caching whenever a server file
operation would cause inconsistencies.
The possibility of inconsistencies occurs whenever a client has
a write caching lease and any other client,
or local operations on the server,
tries to access the file or when
a modify operation is attempted on a file being read cached by client(s).
At this time, the server sends an \fBeviction notice\fR to all clients holding
the lease and then waits for lease termination.
Lease termination occurs when a \fBvacated the premises\fR message has been
received from all the clients that have signed the lease or when the lease
expires via. timeout.
The message pair \fBeviction notice\fR and \fBvacated the premises\fR roughly
correspond to a Sprite server\(->client callback, but are not implemented as an
actual RPC, to avoid the server waiting indefinitely for a reply from a dead
client.
.pp
Server consistency checking can be viewed as issuing intrinsic leases for a
file operation for the duration of the operation only. For example, the
\fBCreate RPC\fR will get an intrinsic write lease on the directory in which
the file is being created, disabling client read caches for that directory.
.pp
By relegating this responsibility to the server, consistency between the
server and NQNFS clients is maintained when NFS clients are modifying the
file system as well.\**
.(f
\** The NFS clients will continue to be \fIapproximately\fR consistent with
the server.
.)f
.pp
The leases are issued as time intervals to avoid the requirement of time of day
clock synchronization. There are three important time constants known to
the server. The \fBmaximum_lease_term\fR sets an upper bound on lease duration.
The \fBclock_skew\fR is added to all lease terms on the server to correct for
differing clock speeds between the client and server and \fBwrite_slack\fR is
the number of seconds the server is willing to wait for a client with
an expired write caching lease to push dirty writes.
.pp
The server maintains a \fBmodify_revision\fR number for each file. It is
defined as a unsigned quadword integer that is never zero and that must
increase whenever the corresponding file is modified on the server.
It is used
by the client to determine whether or not cached data for the file is
stale.
Generating this value is easier said than done. The current implementation
uses the following technique, which is believed to be adequate.
The high order longword is stored in the ufs inode and is initialized to one
when an inode is first allocated.
The low order longword is stored in main memory only and is initialized to
zero when an inode is read in from disk.
When the file is modified for the first time within a given second of
wall clock time, the high order longword is incremented by one and
the low order longword reset to zero.
For subsequent modifications within the same second of wall clock
time, the low order longword is incremented. If the low order longword wraps
around to zero, the high order longword is incremented again.
Since the high order longword only increments once per second and the inode
is pushed to disk frequently during file modification, this implies
0 \(<= Current\(miDisk \(<= 5.
When the inode is read in from disk, 10
is added to the high order longword, which ensures that the quadword
is greater than any value it could have had before a crash.
This introduces apparent modifications every time the inode falls out of
the LRU inode cache, but this should only reduce the client caching performance
by a (hopefully) small margin.
.sh 2 "Crash Recovery and other Failure Scenarios"
.pp
The server must maintain the state of all the current leases held by clients.
The nice thing about short term leases is that maximum_lease_term seconds
after the server stops issuing leases, there are no current leases left.
As such, server crash recovery does not require any state recovery. After
rebooting, the server refuses to service any RPCs except for writes until
write_slack seconds after the last lease would have expired\**.
.(f
\** The last lease expiry time may be safely estimated as
"boottime+maximum_lease_term+clock_skew" for machines that cannot store
it in nonvolatile RAM.
.)f
By then, the server would not have any outstanding leases to recover the
state of and the clients have had at least write_slack seconds to push dirty
writes to the server and get the server sync'd up to date. After this, the
server simply services requests in a manner similar to NFS.
In an effort to minimize the effect of "recovery storms" [Baker91],
the server replies \fBtry_again_later\fR to the RPCs it is not
yet ready to service.
.pp
After a client crashes, the server may have to wait for a lease to timeout
before servicing a request if write sharing of a file with a cachable lease
on the client is about to occur.
As for the client, it simply starts up getting any leases it now needs. Any
outstanding leases for that client on the server prior to the crash will either be renewed or expire
via timeout.
.pp
Certain network partitioning failures are more problematic. If a client to
server network connection is severed just before a write caching lease expires,
the client cannot push the dirty writes to the server. After the lease expires
on the server, the server permits other clients to access the file with the
potential of getting stale data. Unfortunately I believe this failure scenario
is intrinsic in any delay write caching scheme unless the server is required to
wait \fBforever\fR for a client to regain contact\**.
.(f
\** Gray and Cheriton avoid this problem by using a \fBwrite through\fR policy.
.)f
Since the write caching lease has expired on the client,
it will sync up with the
server as soon as the network connection has been re-established.
.pp
There is another failure condition that can occur when the server is congested.
The worst case scenario would have the client pushing dirty writes to the server
but a large request queue on the server delays these writes for more than
\fBwrite_slack\fR seconds. It is hoped that a congestion control scheme using
the \fBtry_again_later\fR RPC reply after booting combined with
the following lease termination rule for write caching leases
can minimize the risk of this occurrence.
A write caching lease is only terminated on the server when there are have
been no writes to the file and the server has not been overloaded during
the previous write_slack seconds. The server has not been overloaded
is approximated by a test for sleeping nfsd(s) at the end of the write_slack
period.
.sh 2 "Server Disk Full"
.pp
There is a serious unresolved problem for delayed write caching with respect to
server disk space allocation.
When the disk on the file server is full, delayed write RPCs can fail
due to "out of space".
For NFS, this occurrence results in an error return from the close system
call on the file, since the dirty blocks are pushed on close.
Processes writing important files can check for this error return
to ensure that the file was written successfully.
For NQNFS, the dirty blocks are not pushed on close and as such the client
may not attempt the write RPC until after the process has done the close
which implies no error return from the close.
For the current prototype,
the only solution is to modify programs writing important
file(s) to call fsync and check for an error return from it instead of close.
.sh 2 "Protocol Details"
.pp
The protocol specification is identical to that of NFS [Sun89] except for
the following changes.
.ip \(bu
RPC Information
.(l
        Program Number 300105
        Version Number 1
.)l
.ip \(bu
Readdir_and_Lookup RPC
.(l
        struct readdirlookargs {
                fhandle file;
                nfscookie cookie;
                unsigned count;
                unsigned duration;
        };

        struct entry {
                unsigned cachable;
                unsigned duration;
                modifyrev rev;
                fhandle entry_fh;
                nqnfs_fattr entry_attrib;
                unsigned fileid;
                filename name;
                nfscookie cookie;
                entry *nextentry;
        };

        union readdirlookres switch (stat status) {
        case NFS_OK:
                struct {
                        entry *entries;
                        bool eof;
                } readdirlookok;
        default:
                void;
        };