nfs tracing.txt

黑客培训教程

nameserver lookup each time this was necessary; this quickly flooded the
network with a nameserver request for each NFS transaction. The current
version maintains a cache of host names; this requires a only a modest
amount of memory for typical networks of less than a few hundred hosts. For
very large networks or those where NFS service is provided to a large number
of remote hosts, this could still be a potential problem, but as a last
resort remote name resolution could be disabled or rpcspy configured to not
translate IP addresses.

UDP/IP datagrams may be fragmented among several packets if the datagram is
larger than the maximum size of a single Ethernet frame. rpcspy looks only
at the first fragment; in practice, fragmentation occurs only for the data
fields of NFS read and write transactions, which are ignored anyway.

3. nfstrace: The Filesystem Tracing Package

Although rpcspy provides a trace of the low-level NFS commands, it is not,
in and of itself, sufficient for obtaining useful filesystem traces. The
low-level commands do not by themselves reveal user-level activity. Furth
ermore, the volume of data that would need to be recorded is potentially
enormous, on the order of megabytes per hour. More useful would be an
abstraction of the user-level system calls underlying the NFS activity.

nfstrace is a filter for rpcspy that produces a log of a plausible set of
user level filesystem commands that could have triggered the monitored
activity. A record is produced each time a file is opened, giving a summary
of what occurred. This summary is detailed enough for analysis or for use as
input to a filesystem simulator.

The output format of nfstrace consists of 7 fields:

timestamp | command-time | direction | file-id | client | transferred | size

where timestamp is the time the open occurred, command-time is the length of
time between open and close, direc tion is either read or write (mkdir and
readdir count as write and read, respectively). file-id identifies the
server and the file handle, client is the client and user that performed the
open, transferred is the number of bytes of the file actually read or
written (cache hits have a 0 in this field), and size is the size of the
file (in bytes).

An example record might be as follows:

690691919.593442 | 17734 | read | basso:7b1f00000000400f | frejus.321 | 0 |
24576

Here, userid 321 at client frejus read file 7b1f00000000400f on server
basso. The file is 24576 bytes long and was able to be read from the client
cache. The command started at Unix time 690691919.593442 and took 17734
microseconds at the server to execute.

Since it is sometimes useful to know the name corresponding to the handle
and the mode information for each file, nfstrace optionally produces a map
of file handles to file names and modes. When enough information (from
lookup and readdir commands) is received, new names are added. Names can
change over time (as files are deleted and renamed), so the times each
mapping can be considered valid is recorded as well. The mapping infor
mation may not always be complete, however, depending on how much activity
has already been observed. Also, hard links can confuse the name mapping,
and it is not always possible to determine which of several possible names a
file was opened under.

What nfstrace produces is only an approximation of the underlying user
activity. Since there are no NFS open or close commands, the program must
guess when these system calls occur. It does this by taking advantage of the
observation that NFS is fairly consistent in what it does when a file is
opened. If the file is in the local buffer cache, a getattr call is made on
the file to verify that it has not changed since the file was cached.
Otherwise, the actual bytes of the file are fetched as they are read by the
user. (It is possible that part of the file is in the cache and part is not,
in which case the getattr is performed and only the missing pieces are
fetched. This occurs most often when a demand-paged executable is loaded).
nfstrace assumes that any sequence of NFS read calls on the same file issued
by the same user at the same client is part of a single open for read. The
close is assumed to have taken place when the last read in the sequence
completes. The end of a read sequence is detected when the same client reads
the beginning of the file again or when a timeout with no reading has
elapsed. Writes are handled in a similar manner.


Reads that are entirely from the client cache are a bit harder; not every
getattr command is caused by a cache read, and a few cache reads take place
without a getattr. A user level stat system call can sometimes trigger a
getattr, as can an ls -l command. Fortunately, the attribute caching used by
most implementations of NFS seems to eliminate many of these extraneous
getattrs, and ls commands appear to trigger a lookup command most of the
time. nfstrace assumes that a getattr on any file that the client has read
within the past few hours represents a cache read, otherwise it is ignored.
This simple heuristic seems to be fairly accurate in practice. Note also
that a getattr might not be performed if a read occurs very soon after the
last read, but the time threshold is generally short enough that this is
rarely a problem. Still, the cached reads that nfstrace reports are, at
best, an estimate (generally erring on the side of over-reporting). There is
no way to determine the number of bytes actually read for cache hits.

The output of nfstrace is necessarily produced out of chronological order,
but may be sorted easily by a post-processor.

nfstrace has a host of options to control the level of detail of the trace,
the lengths of the timeouts, and so on. To facilitate the production of very
long traces, the output can be flushed and checkpointed at a specified inter
val, and can be automatically compressed.

4. Using rpcspy and nfstrace for Filesystem Tracing

Clearly, nfstrace is not suitable for producing highly accurate traces;
cache hits are only estimated, the timing information is imprecise, and data
from lost (and duplicated) network packets are not accounted for. When such
a highly accurate trace is required, other approaches, such as modification
of the client and server kernels, must be employed.

The main virtue of the passive-monitoring approach lies in its simplicity.
In [5], Baker, et al, describe a trace of a distributed filesystem which
involved low-level modification of several different operating system
kernels. In contrast, our entire filesystem trace package consists of less
than 5000 lines of code written by a single programmer in a few weeks,
involves no kernel modifications, and can be installed to monitor multiple
heterogeneous servers and clients with no knowledge of even what operating
systems they are running.

The most important parameter affecting the accuracy of the traces is the
ability of the machine on which rpcspy is running to keep up with the
network traffic. Although most modern RISC workstations with reasonable
Ethernet interfaces are able to keep up with typical network loads, it is
important to determine how much informa tion was lost due to packet buffer
overruns before relying upon the trace data. It is also important that the
trace be, indeed, non-intrusive. It quickly became obvious, for example,
that logging the traffic to an NFS filesystem can be problematic.

Another parameter affecting the usefulness of the traces is the validity of
the heuristics used to translate from RPC calls into user-level system
calls. To test this, a shell script was written that performed ls -l, touch,
cp and wc commands randomly in a small directory hierarchy, keeping a record
of which files were touched and read and at what time. After several hours,
nfstrace was able to detect 100% of the writes, 100% of the uncached reads,
and 99.4% of the cached reads. Cached reads were over-reported by 11%, even
though ls com mands (which cause the "phantom" reads) made up 50% of the
test activity. While this test provides encouraging evidence of the accuracy
of the traces, it is not by itself conclusive, since the particular workload
being monitored may fool nfstrace in unanticipated ways.

As in any research where data are collected about the behavior of human
subjects, the privacy of the individu als observed is a concern. Although
the contents of files are not logged by the toolkit, it is still possible to
learn something about individual users from examining what files they read
and write. At a minimum, the users of a mon itored system should be informed
of the nature of the trace and the uses to which it will be put. In some
cases, it may be necessary to disable the name translation from nfstrace
when the data are being provided to others. Commercial sites where filenames
might reveal something about proprietary projects can be particularly
sensitive to such concerns.


5. A Trace of Filesystem Activity in the Princeton C.S. Department

A previous paper[14] analyzed a five-day long trace of filesystem activity
conducted on 112 research worksta tions at DEC-SRC. The paper identified a
number of file access properties that affect filesystem caching perfor
mance; it is difficult, however, to know whether these properties were
unique artifacts of that particular environment or are more generally
applicable. To help answer that question, it is necessary to look at similar
traces from other computing environments.

It was relatively easy to use rpcspy and nfstrace to conduct a week long
trace of filesystem activity in the Princeton University Computer Science
Department. The departmental computing facility serves a community of
approximately 250 users, of which about 65% are researchers (faculty,
graduate students, undergraduate researchers, postdoctoral staff, etc), 5%
office staff, 2% systems staff, and the rest guests and other "external"
users. About 115 of the users work full-time in the building and use the
system heavily for electronic mail, netnews, and other such communication
services as well as other computer science research oriented tasks (editing,
compiling, and executing programs, formatting documents, etc).

The computing facility consists of a central Auspex file server (fs) (to
which users do not ordinarily log in directly), four DEC 5000/200s (elan,
hart, atomic and dynamic) used as shared cycle servers, and an assortment of
dedicated workstations (NeXT machines, Sun workstations, IBM-RTs, Iris
workstations, etc.) in indi vidual offices and laboratories. Most users log
in to one of the four cycle servers via X window terminals located in
offices; the terminals are divided evenly among the four servers. There are
a number of Ethernets throughout the building. The central file server is
connected to a "machine room network" to which no user terminals are
directly connected; traffic to the file server from outside the machine room
is gatewayed via a Cisco router. Each of the four cycle servers has a local
/, /bin and /tmp filesystem; other filesystems, including /usr, /usr/local,
and users' home directories are NFS mounted from fs. Mail sent from local
machines is delivered locally to the (shared) fs:/usr/spool/mail; mail from
outside is delivered directly on fs.

The trace was conducted by connecting a dedicated DEC 5000/200 with a local
disk to the machine room net work. This network carries NFS traffic for all
home directory access and access to all non-local cycle-server files
(including the most of the actively-used programs). On a typical weekday,
about 8 million packets are transmitted over this network. nfstrace was
configured to record opens for read and write (but not directory accesses or
individual reads or writes). After one week (wednesday to wednesday),
342,530 opens for read and 125,542 opens for write were recorded, occupying
8 MB of (compressed) disk space. Most of this traffic was from the four
cycle servers.

No attempt was made to "normalize" the workload during the trace period.
Although users were notified that file accesses were being recorded, and
provided an opportunity to ask to be excluded from the data collection, most
users seemed to simply continue with their normal work. Similarly, no
correction is made for any anomalous user activity that may have occurred
during the trace.

5.1. The Workload Over Time

Intuitively, the volume of traffic can be expected to vary with the time of
day. Figure 1 shows the number of reads and writes per hour over the seven
days of the trace; in particular, the volume of write traffic seems to
mirror the general level of departmental activity fairly closely.

An important metric of NFS performance is the client buffer cache hit rate.
Each of the four cycle servers allocates approximately 6MB of memory for the
buffer cache. The (estimated) aggregate hit rate (percentage of reads served
by client caches) as seen at the file server was surprisingly low: 22.2%
over the entire week. In any given hour, the hit rate never exceeded 40%.
Figure 2 plots (actual) server reads and (estimated) cache hits per hour
over the trace week; observe that the hit rate is at its worst during
periods of the heaviest read activity.

Past studies have predicted much higher hit rates than the aggregate
observed here. It is probable that since most of the traffic is generated by
the shared cycle servers, the low hit rate can be attributed to the large
number of users competing for cache space. In fact, the hit rate was
observed to be much higher on the single-user worksta tions monitored in the
study, averaging above 52% overall. This suggests, somewhat
counter-intuitively, that if more computers were added to the network (such
that each user had a private workstation), the server load would decrease
considerably. Figure 3 shows the actual cache misses and estimated cache
hits for a typical private works tation in the study.


Thu 00:00  Thu 06:00  Thu 12:00  Thu 18:00  Fri 00:00  Fri 06:00  Fri 12:00