rfc2123.txt

RFC 的详细文档！

Network Working Group                                       N. Brownlee
Request for Comments: 2123                   The University of Auckland
Category: Informational                                      March 1997


          Traffic Flow Measurement:  Experiences with NeTraMet

Status of this Memo

   This memo provides information for the Internet community.  This memo
   does not specify an Internet standard of any kind.  Distribution of
   this memo is unlimited.

Abstract

   This memo records experiences in implementing and using the Traffic
   Flow Measurement Architecture and Meter MIB. It discusses the
   implementation of NeTraMet (a traffic meter) and NeMaC (a combined
   manager and meter reader), considers the writing of meter rule sets
   and gives some guidance on setting up a traffic flow measurement
   system using NeTraMet.

Table of Contents

 1 Introduction                                                        2
   1.1 NeTraMet structure and development . . . . . . . . . . . . . .  3
   1.2 Scope of this document . . . . . . . . . . . . . . . . . . . .  4
 2 Implementation                                                      4
   2.1 Choice of meter platform . . . . . . . . . . . . . . . . . . .  4
   2.2 Programming support requirements . . . . . . . . . . . . . . .  5
     2.2.1 DOS environment  . . . . . . . . . . . . . . . . . . . . .  6
     2.2.2 Unix environment . . . . . . . . . . . . . . . . . . . . .  7
   2.3 Implementing the meter . . . . . . . . . . . . . . . . . . . .  7
     2.3.1 Data structures  . . . . . . . . . . . . . . . . . . . . .  7
     2.3.2 Packet matching  . . . . . . . . . . . . . . . . . . . . .  8
     2.3.3 Testing groups of rule addresses . . . . . . . . . . . . .  8
     2.3.4 Compression of address masks . . . . . . . . . . . . . . .  9
     2.3.5 Ignoring unwanted flow data  . . . . . . . . . . . . . . . 10
     2.3.6 Observing meter reader activity  . . . . . . . . . . . . . 11
     2.3.7 Meter memory management  . . . . . . . . . . . . . . . . . 12
   2.4 Data collection  . . . . . . . . . . . . . . . . . . . . . . . 14
   2.5 Restarting a meter . . . . . . . . . . . . . . . . . . . . . . 15
   2.6 Performance  . . . . . . . . . . . . . . . . . . . . . . . . . 16
 3 Writing rule sets                                                  16
   3.1 Rule set to observe all flows  . . . . . . . . . . . . . . . . 17
   3.2 Specifying flow direction, using computed attributes . . . . . 18
   3.3 Subroutines  . . . . . . . . . . . . . . . . . . . . . . . . . 21
   3.4 More complicated rule sets . . . . . . . . . . . . . . . . . . 23



Brownlee                     Informational                      [Page 1]

RFC 2123                Traffic Flow Measurement              March 1997


 4 Flow data files                                                    26
   4.1 Sample flow data file  . . . . . . . . . . . . . . . . . . . . 27
   4.2 Flow data file features  . . . . . . . . . . . . . . . . . . . 28
   4.3 Terminating and restarting meter reading . . . . . . . . . . . 29
 5 Analysis applications                                              30
 6 Using NeTraMet in a measurement system                             31
   6.1 Examples of NeTraMet in production use . . . . . . . . . . . . 31
 7 Acknowledgments                                                    33
 8 References                                                         33
 9 Security Considerations                                            34
10 Author's Address                                                   34

1 Introduction

   Early in 1992 my University needed to develop a system for recovering
   the costs of its Internet traffic.  In March of that year I attended
   the Internet Accounting Working Group's session at the San Diego
   IETF, where I was delighted to find that the Group had produced a
   detailed architecture for measuring network traffic and were waiting
   for someone to try implementing it.

   During 1992 I produced a prototype measurement system, using balanced
   binary trees to store information about traffic flows.  This work was
   reported at the Washington IETF in November 1992.  The prototype
   performed well, but it made no attempt to recover memory from old
   flows, and the overheads in managing the balanced trees proved to be
   unacceptably high.  I moved on to develop a production-quality
   system, this time using hash tables to index the flow information.

   This version was called NeTraMet (the Network Traffic Meter), and was
   released as free software in October 1993.  Since then I have
   continued working on NeTraMet, producing new releases two or three
   times each year.  NeTraMet is now in production use at many sites
   around the world.  It is difficult to estimate the number of sites,
   but there is an active NeTraMet mailing list, which had about 130
   subscribers in March 1996.

   Early in 1996 the Realtime Traffic Flow Measurement Working Group
   (RTFM) was chartered to move the Traffic Flow Measurement
   Architecture on to the IETF standards track.  This document records
   traffic flow measurement experience gained through three years
   experience with NeTraMet.









Brownlee                     Informational                      [Page 2]

RFC 2123                Traffic Flow Measurement              March 1997


1.1 NeTraMet structure and development

   The Traffic Flow Architecture document [1] describes four components:

     - METERS, which are attached to the network at the points where
       it is desired to measure the traffic,

     - METER READERS, which read data from meters and store it for later
       use,

     - MANAGERS, which configure meters and control meter readers, and

     - ANALYSIS APPLICATIONS, which process the data from meter readers
       so as to produce whatever reports are required.

   NeTraMet is a computer program which implements the Traffic Meter,
   stores the measured flow data in memory, and provides an SNMP agent
   so as to make it available to Meter Readers.  The NeTraMet
   distribution files include NeMaC, which is a combined Manager and
   Meter Reader capable of managing an arbitrary number of meters, each
   of which may be using its own rule set, and having its flow data
   collected at its own specified intervals.  The NeTraMet distribution
   also includes several rudimentary Analysis Applications, allowing
   users to produce simple plots from NeMaC's flow data files (fd_filter
   and fd_extract) and to monitor - in real time - the flows at a remote
   meter (nm_rc and nifty).

   Since the first release the Traffic Meter MIB [2] has been both
   improved and simplified.  Significant changes have included better
   ways to specify traffic flows (i.e. more actions and better control
   structures for the Packet Matching Engine), and computed attributes
   (class and kind).  These changes have been prompted by operational
   requirements at sites using NeTraMet, and have been tested
   extensively in successive versions of NeTraMet.

   NeTraMet is widely used to collect usage data for Internet Service
   Providers.  This is especially so in Australia and New Zealand, but
   there are also active users at sites around the world, for example in
   Canada, France, Germany and Poland.

   NeTraMet is very useful as a tool for understanding exactly where
   traffic is flowing in large networks.  Since the Traffic Meters
   perform considerable data reduction (as specified by their rule sets)
   they significantly reduce the volume of data to be read by Meter
   Readers.  This characteristic makes NeTraMet particularly effective
   for networks with many remote sites.  An example of this (the
   Kawaihiko network) is briefly described below.




Brownlee                     Informational                      [Page 3]

RFC 2123                Traffic Flow Measurement              March 1997


   As well as providing data for post-observation analysis, NeTraMet can
   be used for real-time network monitoring and trouble-shooting.  The
   NeTraMet distribution includes 'nifty,' an X/Motif application which
   monitors traffic flows and attempts to highlight those which are
   'interesting.'

1.2 Scope of this document

   This document presents the experience gained from three years work
   with the Traffic Flow Measurement Architecture.  Its contents are
   grouped as follows

     - Implementation issues for NeTraMet and NeMaC,

     - How rule files work, and how to write them for particular
       purposes, and

     - How to use NeTraMet and NeMaC for short-term and long-term flow
       measurement.

2 Implementation

2.1 Choice of meter platform

   As pointed out in the Architecture document [1], the goal of the
   Realtime Traffic Flow Measurement Working Group is to develop a
   standard for the Traffic Meter, with the goal of seeing it
   implemented in network devices such as hubs, switches and routers.
   Until the Architecture is well enough developed to allow this, it has
   sufficed to implement the meter as a program running on a general-
   purpose computer system.

   The choice of computer system for NeTraMet was driven by the need to
   choose one which would be widely available within the Internet
   community.  One strong possibility was a Unix system, since these are
   commonly used for a variety of network support and management tasks.
   For the initial implementation, however, Unix would have had some
   disadvantages:

     - The wide variety of different Unix systems can increase the
       difficulties of software support.

     - The cost of a Unix system as a meter is too high to allow users
       to run meters simultaneously at many points within their
       networks.






Brownlee                     Informational                      [Page 4]

RFC 2123                Traffic Flow Measurement              March 1997


   Another factor in choosing the platform was system performance.  When
   I first started implementing NeTraMet it was impossible to predict
   how much processing workload was needed for a viable meter.
   Similarly, I had no idea how much memory would be required for code
   or data.  I therefore chose to implement NeTraMet on a DOS PC. This
   was because:

     - It is a minimum system in all respects.  If the meter works
       well on such a system, it can be implemented on almost any
       hardware (including routers, switches, etc.)

     - It is an inexpensive system.  Sites can easily afford to have
       many meters around their networks.

     - It is a simple system, and one which I had complete control over.
       This allowed me to implement effective instrumentation to monitor
       the meter's performance, and to include a wide variety of
       performance optimisations in the code.

   Once the meter was running I needed a manager to download rule files
   to it.  Since a single manager and meter reader can effectively
   support a large number of meters, a Unix environment for NeMaC was a
   natural choice.  There are fewer software support problems for NeMaC
   than for NeTraMet since NeMaC has minimal support needs - it only
   needs to open a UDP socket to the SNMP port on each controlled meter.

   Early NeTraMet distributions contained only the PC meter and Unix
   manager.  In later releases I ported NeTraMet (the meter) to Unix,
   and extended the control features of NeMaC (the combined manager and
   meter reader).  I have also experimented with porting NeMaC to the
   DOS system.  This is not difficult, but doesn't seem to be worth
   pursuing.

   The current version of NeTraMet is a production-quality traffic
   measurement system which has been in continuous use at the University
   of Auckland for nearly two years.

2.2 Programming support requirements

   To implement the Traffic Flow Meter I needed a programming
   environment providing good support for the following:

     - observation of packet headers on the network;

     - system timer with better than 10 ms resolution;

     - IP (Internet Protocol), for communications with manager and meter
       reader;



Brownlee                     Informational                      [Page 5]

RFC 2123                Traffic Flow Measurement              March 1997


     - SNMP, for the agent implementing the Meter MIB.