nqnfs.me

早期freebsd实现

"(not cached)" at 0.550,4.733 ljust
"Read data" at 0.550,4.921 ljust
"Reply  data" at 1.675,4.421 ljust
"Read request" at 1.675,4.921 ljust
"lease" at 1.675,5.233 ljust
"Reply non-caching" at 1.675,5.421 ljust
"Reply" at 3.737,5.733 ljust
"Write" at 3.175,5.983 ljust
"Reply" at 3.737,6.171 ljust
"Write" at 3.175,6.421 ljust
"Eviction Notice" at 3.175,6.796 ljust
"Get read lease" at 1.675,7.046 ljust
"Read syscall" at 0.550,6.983 ljust
"being cached" at 4.675,7.171 ljust
"Delayed writes" at 4.675,7.358 ljust
"lease" at 3.175,7.233 ljust
"Reply write caching" at 3.175,7.421 ljust
"Get  write lease" at 3.175,7.983 ljust
"Write syscall" at 4.675,7.983 ljust
"with same modrev" at 1.675,8.358 ljust
"Lease" at 0.550,8.171 ljust
"Renewed" at 0.550,8.358 ljust
"Reply" at 1.675,8.608 ljust
"Get Lease Request" at 1.675,8.983 ljust
"Read syscall" at 0.550,8.733 ljust
"from cache" at 0.550,9.108 ljust
"Read syscall" at 0.550,9.296 ljust
"Reply " at 1.675,9.671 ljust
"plus lease" at 2.050,9.983 ljust
"Read Request" at 1.675,10.108 ljust
.ps
.ft
.PE
.sp
.)z
A write-caching lease is not used in the Stanford V Distributed System [Gray89],
since synchronous writing is always used. A side effect of this change
is that the five to ten second lease duration recommended by Gray was found
to be insufficient to achieve good performance for the write-caching lease.
Experimentation showed that thirty seconds was about optimal for cases where
the client and server are connected to the same local area network, so
thirty seconds is the default lease duration for NQNFS.
A maximum of twice that value is permitted, since Gray showed that for some
network topologies, a larger lease duration functions better.
Although there is an explicit get_lease RPC defined for the protocol,
most lease requests are piggybacked onto the other RPCs to minimize the
additional overhead introduced by leasing.
.sh 2 "Rationale"
.pp
Leasing was chosen over hard server state information for the following
reasons:
.ip 1.
The server must maintain state information about all current
client leases.
Since at most one lease is allocated for each RPC and the leases expire
after their lease term,
the upper bound on the number of current leases is the product of the
lease term and the server RPC rate.
In practice, it has been observed that less than 10% of RPCs request new leases
and since most leases have a term of thirty seconds, the following rule of
thumb should estimate the number of server lease records:
.sp
.nf
	Number of Server Lease Records \(eq 0.1 * 30 * RPC rate
.fi
.sp
Since each lease record occupies 64 bytes of server memory, storing the lease
records should not be a serious problem.
If a server has exhausted lease storage, it can simply wait a few seconds
for a lease to expire and free up a record.
On the other hand, a Sprite-like server must store records for all files
currently open by all clients, which can require significant storage for
a large, heavily loaded server.
In [Mogul93], it is proposed that a mechanism vaguely similar to paging could be
used to deal with this for Spritely NFS, but this
appears to introduce a fair amount of complexity and may limit the
usefulness of open records for storing other state information, such
as file locks.
.ip 2.
After a server crashes it must recover lease records for
the current outstanding leases, which actually implies that if it waits
until all leases have expired, there is no state to recover.
The server must wait for the maximum lease duration of one minute, and it must serve
all outstanding write requests resulting from terminated write-caching
leases before issuing new leases. The one minute delay can be overlapped with
file system consistency checking (eg. fsck).
Because no state must be recovered, a lease-based server, like an NFS server,
avoids the problem of state recovery after a crash.
.sp
There can, however, be problems during crash recovery
because of a potentially large number of write backs due to terminated
write-caching leases.
One of these problems is a "recovery storm" [Baker91], which could occur when
the server is overloaded by the number of write RPC requests.
The NQNFS protocol deals with this by replying
with a return status code called
try_again_later to all
RPC requests (except write) until the write requests subside.
At this time, there has not been sufficient testing of server crash
recovery while under heavy server load to determine if the try_again_later
reply is a sufficient solution to the problem.
The other problem is that consistency will be lost if other RPCs are performed
before all of the write backs for terminated write-caching leases have completed.
This is handled by only performing write RPCs until
no write RPC requests arrive
for write_slack seconds, where write_slack is set to several times
the client timeout retransmit interval,
at which time it is assumed all clients have had an opportunity to send their writes
to the server.
.ip 3.
Another advantage of leasing is that, since leases are required at times when other I/O operations occur,
lease requests can almost always be piggybacked on other RPCs, avoiding some of the
overhead associated with the explicit open and close RPCs required by a Sprite-like system.
Compared with Sprite cache consistency,
this can result in a significantly lower RPC load (see table #1).
.sh 1 "Limitations of the NQNFS Protocol"
.pp
There is a serious risk when leasing is used for delayed write
caching.
If the server is simply too busy to service a lease renewal before a write-caching
lease terminates, the client will not be able to push the write
data to the server before the lease has terminated, resulting in
inconsistency.
Note that the danger of inconsistency occurs when the server assumes that
a write-caching lease has terminated before the client has
had the opportunity to write the data back to the server.
In an effort to avoid this problem, the NQNFS server does not assume that
a write-caching lease has terminated until three conditions are met:
.sp
.(l
1 - clock time > (expiry time + clock skew)
2 - there is at least one server daemon (nfsd) waiting for an RPC request
3 - no write RPCs received for leased file within write_slack after the corrected expiry time
.)l
.lp
The first condition ensures that the lease has expired on the client.
The clock_skew, by default three seconds, must be
set to a value larger than the maximum time-of-day clock error that is likely to occur
during the maximum lease duration.
The second condition attempts to ensure that the client
is not waiting for replies to any writes that are still queued for service by
an nfsd. The third condition tries to guarantee that the client has
transmitted all write requests to the server, since write_slack is set to
several times the client's timeout retransmit interval.
.pp
There are also certain file system semantics that are problematic for both NFS and NQNFS,
due to the
lack of state information maintained by the
server. If a file is unlinked on one client while open on another it will
be removed from the file server, resulting in failed file accesses on the
client that has the file open.
If the file system on the server is out of space or the client user's disk
quota has been exceeded, a delayed write can fail long after the write system
call was successfully completed.
With NFS this error will be detected by the close system call, since
the delayed writes are pushed upon close. With NQNFS however, the delayed write
RPC may not occur until after the close system call, possibly even after the process
has exited.
Therefore,
if a process must check for write errors,
a system call such as \fIfsync\fR must be used.
.pp
Another problem occurs when a process on one client is
running an executable file
and a process on another client starts to write to the file. The read lease on
the first client is terminated by the server, but the client has no recourse but
to terminate the process, since the process is already in progress on the old
executable.
.pp
The NQNFS protocol does not support file locking, since a file lock would have
to involve hard, recovered after a crash, state information.
.sh 1 "Other NQNFS Protocol Features"
.pp
NQNFS also includes a variety of minor modifications to the NFS protocol, in an
attempt to address various limitations.
The protocol uses 64bit file sizes and offsets in order to handle large files.
TCP transport may be used as an alternative to UDP
for cases where UDP does not perform well.
Transport mechanisms
such as TCP also permit the use of much larger read/write data sizes,
which might improve performance in certain environments.
.pp
The NQNFS protocol replaces the Readdir RPC with a Readdir_and_Lookup
RPC that returns the file handle and attributes for each file in the
directory as well as name and file id number.
This additional information may then be loaded into the lookup and file-attribute
caches on the client.
Thus, for cases such as "ls -l", the \fIstat\fR system calls can be performed
locally without doing any lookup or getattr RPCs.
Another additional RPC is the Access RPC that checks for file
accessibility against the server. This is necessary since in some cases the
client user ID is mapped to a different user on the server and doing the
access check locally on the client using file attributes and client credentials is
not correct.
One case where this becomes necessary is when the NQNFS mount point is using
Kerberos authentication, where the Kerberos authentication ticket is translated
to credentials on the server that are mapped to the client side user id.
For further details on the protocol, see [Macklem93].
.sh 1 "Performance"
.pp
In order to evaluate the effectiveness of the NQNFS protocol,
a benchmark was used that was
designed to typify
real work on the client workstation.
Benchmarks, such as Laddis [Wittle93], that perform server load characterization
are not appropriate for this work, since it is primarily client caching
efficiency that needs to be evaluated.
Since these tests are measuring overall client system performance and
not just the performance of the file system,
each sequence of runs was performed on identical hardware and operating system in order to factor out the system
components affecting performance other than the file system protocol.
.pp
The equipment used for the all the benchmarks are members of the DECstation\(tm\(dg
family of workstations using the MIPS\(tm\(sc RISC architecture.
The operating system running on these systems was a pre-release version of
4.4BSD Unix\(tm\(dd.
For all benchmarks, the file server was a DECstation 2100 (10 MIPS) with 8Mbytes of
memory and a local RZ23 SCSI disk (27msec average access time).
The clients range in speed from DECstation 2100s
to a DECstation 5000/25, and always run with six block I/O daemons
and a 4Mbyte buffer cache, except for the test runs where the
buffer cache size was the independent variable.
In all cases /tmp is mounted on the local SCSI disk\**, all machines were
attached to the same uncongested Ethernet, and ran in single user mode during the benchmarks.
.(f
\**Testing using the 4.4BSD MFS [McKusick90] resulted in slightly degraded performance,
probably since the machines only had 16Mbytes of memory, and so paging
increased.
.)f
Unless noted otherwise, test runs used UDP RPC transport
and the results given are the average values of four runs.
.pp
The benchmark used is the Modified Andrew Benchmark (MAB)
[Ousterhout90],
which is a slightly modified version of the benchmark used to characterize
performance of the Andrew ITC file system [Howard88].
The MAB was set up with the executable binaries in the remote mounted file
system and the final load step was commented out, due to a linkage problem
during testing under 4.4BSD.
Therefore, these results are not directly comparable to other reported MAB
results.
The MAB is made up of five distinct phases:
.sp
.ip "1." 10
Makes five directories (no significant cost)
.ip "2." 10
Copy a file system subtree to a working directory
.ip "3." 10
Get file attributes (stat) of all the working files
.ip "4." 10
Search for strings (grep) in the files
.ip "5." 10
Compile a library of C sources and archive them
.lp
Of the five phases, the fifth is by far the largest and is the one affected most
by client caching mechanisms.
The results for phase #1 are invariant over all
the caching mechanisms.
.sh 2 "Buffer Cache Size Tests"
.pp
The first experiment was done to see what effect changing the size of the
buffer cache would have on client performance. A single DECstation 5000/25
was used to do a series of runs of MAB with different buffer cache sizes
for four variations of the file system protocol. The four variations are
as follows:
.ip "Case 1:" 10
NFS - The NFS protocol as implemented in 4.4BSD
.ip "Case 2:" 10
Leases - The NQNFS protocol using leases for cache consistency
.ip "Case 3:" 10
Leases, Rdirlookup - The NQNFS protocol using leases for cache consistency
and with the readdir RPC replaced by Readdir_and_Lookup
.ip "Case 4:" 10
Leases, Attrib leases, Rdirlookup - The NQNFS protocol using leases for
cache consistency, with the readdir
RPC replaced by the Readdir_and_Lookup,
and requiring a valid lease not only for file-data access, but also for file-attribute access.
.lp
As can be seen in figure 1, the buffer cache achieves about optimal
performance for the range of two to ten megabytes in size. At eleven
megabytes in size, the system pages heavily and the runs did not
complete in a reasonable time. Even at 64Kbytes, the buffer cache improves
performance over no buffer cache by a significant margin of 136-148 seconds
versus 239 seconds.
This may be due, in part, to the fact that the Compile Phase of the MAB
uses a rather small working set of file data.
All variants of NQNFS achieve about
the same performance, running around 30% faster than NFS, with a slightly
larger difference for large buffer cache sizes.
Based on these results, all remaining tests were run with the buffer cache
size set to 4Mbytes.
Although I do not know what causes the local peak in the curves between 0.5 and 2 megabytes,
there is some indication that contention for buffer cache blocks, between the update process
(which pushes delayed writes to the server every thirty seconds) and the I/O
system calls, may be involved.
.(z
.PS
.ps
.ps 10
dashwid = 0.050i
line dashed from 0.900,7.888 to 4.787,7.888
line dashed from 0.900,7.888 to 0.900,10.262
line from 0.900,7.888 to 0.963,7.888
line from 4.787,7.888 to 4.725,7.888
line from 0.900,8.188 to 0.963,8.188
line from 4.787,8.188 to 4.725,8.188
line from 0.900,8.488 to 0.963,8.488
line from 4.787,8.488 to 4.725,8.488
line from 0.900,8.775 to 0.963,8.775
line from 4.787,8.775 to 4.725,8.775
line from 0.900,9.075 to 0.963,9.075
line from 4.787,9.075 to 4.725,9.075
line from 0.900,9.375 to 0.963,9.375
line from 4.787,9.375 to 4.725,9.375
line from 0.900,9.675 to 0.963,9.675
line from 4.787,9.675 to 4.725,9.675
line from 0.900,9.963 to 0.963,9.963
line from 4.787,9.963 to 4.725,9.963
line from 0.900,10.262 to 0.963,10.262
line from 4.787,10.262 to 4.725,10.262
line from 0.900,7.888 to 0.900,7.950
line from 0.900,10.262 to 0.900,10.200
line from 1.613,7.888 to 1.613,7.950
line from 1.613,10.262 to 1.613,10.200
line from 2.312,7.888 to 2.312,7.950