gulp.html

改善linux指令 "tcpdump" 效能的免費open source程式

capacity left to empty Gulp's ring buffer as fast as it fills.  Gulp did
not start dropping packets until its ring buffer eventually filled.
This demonstrates that Linux can be very
successful at capturing packets at high speed and delivering them to
user processes as long as the reading process can read them from the
kernel fast enough that the NIC-driver's relatively small ring
buffer does not overflow.  At very high packet rates, even though the
e1000 NIC driver does interrupt aggregation,
<A HREF=http://xosview.sourceforge.net/>xosview</A> indicated that much of
the CPU was consumed with "hard" and "soft" interrupt processing. </P>

<P> In summary, I believe as long as the average packet size is 300 or
more, our system should be able to capture and write to disk every
packet it receives from a gigabit ethernet.  The larger the average
packet size, the more CPU headroom is available and the more certain is
capturing every packet.
</P>

<P> I should mention that I have been using Shawn Ostermann's
"<A HREF=http://jarok.cs.ohiou.edu/software/tcptrace/>tcptrace</A>"
program to confirm that when tcpdump or Gulp reports that the kernel
dropped no packets, this is indeed true.  Likewise, when the tools
report the kernel dropped some packets, tcptrace agrees.  This means I have
complete confidence in my claims above for capturing iperf data without
loss.  Although the SmartBits did not generate TCP traffic, it offered
counts of how many packets it sent which agree with what was captured. </P>

<H2>Examples of Gulp Usage</H2>

<UL>
<PRE>
0) the <A HREF=gulpman.pdf>Gulp manpage.pdf</A> or <A HREF=gulpman.html>Gulp manpage.html</A> (converted with <A HREF=http://staff.washington.edu/corey/tools.html#bold2html>bold2html</A>).

1) helping tcpdump drop fewer packets when writing to disk:
   (gulp -c can be used in any pipeline as it does no data interpretation)

    <FONT COLOR=0000c0>tcpdump -i eth1 -w - ... | gulp -c &gt; pcapfile</FONT>
      or if you have more than 2 CPUs, run tcpdump and gulp on different ones:
    <FONT COLOR=0000c0>taskset -c 2 tcpdump -i eth1 -w - ... | gulp -c &gt; pcapfile</FONT>

   (gulp uses CPUs #0,1 so taskset runs tcpdump on #2 to reduce interference)

2) a similar but more efficient capture using Gulp's native capture ability:

    <FONT COLOR=0000c0>gulp -i eth1 -f "..." &gt; pcapfile</FONT>

3) capture and GRE-decapsulate an ERSPAN feed and save the result to disk:

    <FONT COLOR=0000c0>gulp -i eth1 -d &gt; pcapfile</FONT>

4) capture, decapsulate and then filter with tcpdump before saving:

    <FONT COLOR=0000c0>gulp -i eth1 -d | tcpdump -r - -s0 -w pcapfile ...</FONT>
      or if you have more than 2 CPUs, run tcpdump and gulp on different ones:
    <FONT COLOR=0000c0>gulp -i eth1 -d | taskset -c 2 tcpdump -r - -s0 -w pcapfile ...</FONT>

5) capture everything to disk; then decapsulate offline:

    <FONT COLOR=0000c0>gulp -i eth1 &gt; pcapfile1; gulp -d -i - &lt; pcapfile1 &gt; pcapfile2</FONT>

6) capture, decapsulate and filter with <A HREF=http://ngrep.sourceforge.net/>ngrep</A>:

    <FONT COLOR=0000c0>gulp -i eth1 -d | ngrep -I - -O pcapfile regex ...</FONT>

7) capture, decapsulate and feed into <A HREF=http://www.ntop.org>ntop</A>:

    <FONT COLOR=0000c0>gulp -i eth1 -d | ntop -f /dev/stdin -m a.b.c.d/x ...</FONT>
      or
    <FONT COLOR=0000c0>mkfifo pipe; chmod 644 pipe; gulp -i eth1 -d > pipe & ntop -u ntop -f pipe -m a.b.c.d/x ...</FONT>

8) capture, decapsulate and feed into <A HREF=http://www.wireshark.org>wireshark</A>:

    <FONT COLOR=0000c0>gulp -i eth1 -d | /usr/sbin/wireshark -i - -k</FONT>

9) capture to 1000MB files, keeping just the most recent 10 (files):

    <FONT COLOR=0000c0>gulp -i eth1 -C 10 -W 10 -o pcapdir</FONT>
      or with help from tcpdump:
    <FONT COLOR=0000c0>gulp -i eth1 | taskset -c 2 tcpdump -r- -C 1000 -W 10 -w pcapname</FONT>
</PRE>
</UL>

<H2> Suggestions for improvements to the Linux code base </H2>

<OL>
<LI> <P> Normally if one is interested in capturing only a subset of the
traffic on an interface, the pcap library can filter out the uninteresting
packets in the kernel (as early as possible) to avoid the overhead of
copying them into userspace and then discarding them. </P>

<P> Because neither the Linux GRE tunnel mechanism, i.e.:

<PRE>
# modprobe ip_gre
# ip tunnel add gre1 local x.y.78.60 remote x.y.78.4 mode gre
# ifconfig gre1 up
# tcpdump -i gre1
</PRE>

<P> nor the pcap code seems to be capable of decapsulating GRE packets with a
non-standard header length (50 bytes in this case) and then applying normal
pcap filters to what remains, I can do no in-kernel filtering on the contents
of the ERSPAN packets--they must all be copied to userspace, decapsulated
and then filtered again by tcpdump (wireshark or equivalent) as per
examples #4-6 above. </P>

<P> Extensions to either the pcap code or the GRE tunnel mechanism should be
able to add the ability to capture a subset of packets more efficiently by
filtering them out in the kernel.  I have not measured the overhead of
"ip tunnel" but I presume doing this in the pcap code would be simplest
and most efficient. </P>

<LI><P> Perhaps select(2) should not always say a descriptor to an open file
on disk will not block for write(2) or alternatively, perhaps the writes
can be made faster so they agree with select(2) and don't block. </P>

<A NAME=64bit>

<LI><P> I think "<CODE>struct&nbsp;pcap_pkthdr</CODE>" in <CODE>pcap.h</CODE>
should be re-defined to be independent of <CODE>sizeof(long)</code>.  In pcap
files, a <CODE>struct&nbsp;pcap_pkthdr</CODE> precedes every packet.
Unfortunately, the size of <CODE>struct&nbsp;pcap_pkthdr</CODE> (which contains
a <CODE>struct&nbsp;timeval</CODE>) depends upon <CODE>sizeof(long)</CODE>.
This makes pcap files from 64-bit linux systems incompatible with those from
32-bit systems.  Apparently as a workaround, some 64-bit linux distributions
are providing tcpdump and wireshark binaries which read/write 32-bit compatible
pcap files (which makes Gulp's pcap output appear to be corrupt).  </P>

<P>
(To build Gulp on 64-bit linux systems so that it reads/writes 32-bit
compatible pcap files, try installng the 32-bit (i386) "libpcap-devel"
package and making Gulp with "-m32" added to CFLAGS.)</P> </A>

</OL>

<H2> Future Work </H2>

<P> To my surprise, I learned after completing this work that Luca
Deri's PF_RING patch is NOT already incorporated in the standard Linux kernel
(as I mistakenly thought) and the packet "ring buffer" that "ethtool"
adjusts is something different.  Though this misunderstanding is
somewhat embarrassing to me, it seems likely that the benefits of Gulp
and PF_RING will be cumulative and since my next obvious goal is 10Gb I look
forward to confirming that.  </P>

<A NAME=links>
<H2> Program Source and Links of Interest</H2>
</A>

<OL>
<LI> <A HREF=gulp.tgz>Gulp Source Code Bundle</A> released under the
     <A HREF=http://www.apache.org/licenses/LICENSE-2.0>Apache License Version 2.0</A>
<LI> <A HREF=gulpman.pdf>Gulp manpage.pdf</A> or
     <A HREF=gulpman.html>Gulp manpage.html</A> (converted with <A HREF=http://staff.washington.edu/corey/tools.html#bold2html>bold2html</A>)
<LI> <A HREF=http://staff.washington.edu/corey/tools/inter-core-benchmark.html>inter-core-benchmark</A>
<LI> <A HREF=http://dast.nlanr.net/Projects/Iperf/>iperf</A>
<LI> <A HREF=http://luca.ntop.org>Luca Deri</A>
<LI> <A HREF=http://gd.tuwien.ac.at/utils/archivers/buffer>McLoughlin's buffer program</A>
<LI> <A HREF=http://ngrep.sourceforge.net/>ngrep</A>
<LI> <A HREF=http://www.ntop.org/PF_RING.html>PF_RING NIC driver</A>
<LI> <A HREF=http://www.tcpdump.org>tcpdump</A>
<LI> <A HREF=http://jarok.cs.ohiou.edu/software/tcptrace/>tcptrace</A>
<LI> <A HREF=http://xosview.sourceforge.net/>xosview</A>
<LI> <A HREF=http://www.ntop.org/>ntop</A>
<LI> <A HREF=http://www.wireshark.org>WireShark</A>
</OL>

<HR><HR>

<P>
<B>Corey Satten</B> <BR>
Email -- <B>corey @ u.washington.edu</B> <BR>
Web -- <A HREF=http://staff.washington.edu/corey/>http://staff.washington.edu/corey/</A> <BR>
Date -- <B>
Tue Mar 18 14:14:09 PDT 2008
</B>

<P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P>
<P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P>
<P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P>
<P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P> <P>&#160;</P>
</BODY>