benchmarks.html

嵌入式http源代码的文档资料

    <TD ALIGN="right">260</TD>
    <TD ALIGN="right">70</TD>
    <TD ALIGN="right">350</TD>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#52" tppabs="http://hoohoo.ncsa.uiuc.edu/">NCSA</A></TH>
</TR><TR>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#65" tppabs="http://home.netscape.com/">Netscape FastTrack</A></TH>
    <TD ALIGN="center">3.01-export-us</TD>
    <TD ALIGN="center">02oct97</TD>
    <TD ALIGN="center">no</TD>
    <TD ALIGN="center">threads</TD>
    <TD ALIGN="center">n/a</TD>
    <TD ALIGN="center">yes</TD>
    <TD ALIGN="center">?</TD>
    <TD ALIGN="center">no</TD>
    <TD ALIGN="right">62,443,520</TD>
    <TD ALIGN="right">n/a</TD>
    <TD ALIGN="right">n/a</TD>
    <TD ALIGN="right">1,896,016</TD>
    <!-- <TD ALIGN="right" BGCOLOR="#99ff99">850</TD> -->
    <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-license">-</A></TD>
    <!-- <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-CGI">5</A></TD> -->
    <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-license">-</A></TD>
    <!-- <TD ALIGN="right">350</TD> -->
    <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-license">-</A></TD>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#65" tppabs="http://home.netscape.com/">Netscape FastTrack</A></TH>
</TR><TR>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#65" tppabs="http://home.netscape.com/">Netscape Enterprise</A></TH>
    <TD ALIGN="center">3.5.1-export-us</TD>
    <TD ALIGN="center">02feb98</TD>
    <TD ALIGN="center">yes</TD>
    <TD ALIGN="center">threads</TD>
    <TD ALIGN="center">n/a</TD>
    <TD ALIGN="center">yes</TD>
    <TD ALIGN="center">?</TD>
    <TD ALIGN="center">no</TD>
    <TD ALIGN="right">130,426,880</TD>
    <TD ALIGN="right">n/a</TD>
    <TD ALIGN="right">n/a</TD>
    <TD ALIGN="right">1,977,568</TD>
    <!-- <TD ALIGN="right" BGCOLOR="#99ff99">750</TD> -->
    <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-license">-</A></TD>
    <!-- <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-CGI">4.5</A></TD> -->
    <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-license">-</A></TD>
    <!-- <TD ALIGN="right">250</TD> -->
    <TD ALIGN="right" BGCOLOR="#cc9999"><A HREF="#Netscape-license">-</A></TD>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#65" tppabs="http://home.netscape.com/">Netscape Enterprise</A></TH>
</TR><TR>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#56" tppabs="http://www.zeus.co.uk/">Zeus</A></TH>
    <TD ALIGN="center">3.1.4</TD>
    <TD ALIGN="center">10jul98</TD>
    <TD ALIGN="center">yes</TD>
    <TD ALIGN="center">select</TD>
    <TD ALIGN="center">n/a</TD>
    <TD ALIGN="center">yes</TD>
    <TD ALIGN="center">no</TD>
    <TD ALIGN="center" BGCOLOR="#99ff99">yes</TD>
    <TD ALIGN="right">3,747,840</TD>
    <TD ALIGN="right">n/a</TD>
    <TD ALIGN="right">n/a</TD>
    <TD ALIGN="right">1,367,976</TD>
    <TD ALIGN="right" BGCOLOR="#99ff99">1050</TD>
    <TD ALIGN="right">70</A></TD>
    <TD ALIGN="right">500</TD>
    <TH ALIGN="center"><A HREF="tppmsgs/msgs0.htm#56" tppabs="http://www.zeus.co.uk/">Zeus</A></TH>
</TR>
</TABLE>

<P>
Notes:

<UL>
    <LI> <A NAME="software">Software - which web server.
    <UL>
	<LI> <A NAME="name">Name - the name of the server, and a link to
	its home page.
	<LI> <A NAME="version">Version - the version tested here.
	<LI> <A NAME="released">Released - when it was released.
	<LI> <A NAME="active">Active devel - whether it is currently
	under development.
    </UL>
    <LI> <A NAME="features">Features - some key server features.
    <UL>
	<LI> <A NAME="model">Model - what kind of server it is.  The models are:
	<UL>
	    <LI> fork - start a new process for each request.
	    <LI> pre-fork - pre-start a pool of processes which each handle
	    multiple requests.
	    <LI> threads - use threads instead of processes.
	    <LI> Java threads - this version of the Java runtime uses
	    "Green threads" instead of native threads, so it acts more like
	    the select-based servers.
	    <LI> select - use non-blocking I/O and the select() system call
	    to handle multiple requests in a single process, single thread.
	</UL>
	<LI> <A NAME="auto-conf">Auto-conf - whether there's a script
	to automatically configure the build process for your OS.
	<UL>
	    <LI> <A NAME="mathopd-dnc">mathopd did not compile out-of-box under
	    Solaris 2.6, I had to make a minor edit.
	    <LI> <A NAME="Roxen-dnc">Roxen did not compile under Solaris 2.6,
	    I had to build under Solaris 2.5.1.
	</UL>
	<LI> <A NAME="basic-auth">Basic auth - whether the server supports
	Basic Authentication, for password-protected web pages.
	<LI> <A NAME="chroot">Chroot - whether the server lets you use
	the chroot() system call to enhance security.
	<LI> <A NAME="throttling">Throttling - the ability to set bandwidth
	limits on certain pages, so they don't use more than their fair share
	of the server's resources.
    </UL>
    <LI> <A NAME="size">Size - various sizes of parts of the server
    <UL>
	<LI> <A NAME="tar-bytes">Tar bytes - the uncompressed source tarchive
	size.
	<LI> <A NAME="files">Source files - how many source and header files.
	<LI> <A NAME="lines">Source lines - how many lines in the source and
	header files.
	<LI> <A NAME="exe">Exe - the executable size.
	For the compiled program this is size of the main executable file,
	stripped.
	For the Java servers it's the total size of the .class files or .zip
	files.
	For Roxen it's the size of the Pike interpreter.
    </UL>
    <LI> The benchmark test system is a 297MHz Ultra Sparc with
    256MB RAM / 512MB swap running Solaris 2.6, otherwise totally quiescent.
    RLIMIT_NOFILE is 256 soft / 1024 hard, and v.v_maxup is 3941.
    <LI> The test loads were generated on a separate machine connected via
    100MB/s ethernet.  The test client is
    <A HREF="tppmsgs/msgs0.htm#66" tppabs="http://www.acme.com/software/http_load/">http_load</A>, doing
    multiple requests in parallel, with checksumming enabled.
    <LI> All the test runs were done at least twice on different days, and
    all the shapes and inflection points etc. proved to be completely
    repeatable.
    <LI> All programs are in their out-of-box configuration, except as
    needed for the tests (e.g. enabling CGI).  And except for mathopd,
    which doesn't have an out-of-box configuration - I used
    <A HREF="mathopd_config.txt" tppabs="http://www.acme.com/software/thttpd/mathopd_config.txt">this one</A>.
    <LI> <A NAME="rps">RPS - maximum requests per second.  This is determined
    by running the test load at various parallel-request rates and finding
    the maximum response rate.  See the graph below for the full data.
    <UL>
	<LI> <A NAME="small">Small files - the small-file test load consists of
	1000 files, each 1KB long, requested randomly.
	<UL>
	    <LI> <A NAME="Acme-small">Acme.Serve exitted at 60 parallel
	    requests, due to running out of file descriptors.  Apparently
	    the Java runtime uses three file descriptors per thread, or
	    something like that.  But still, the program shouldn't exit!
	    <LI> <A NAME="Jigsaw-mystery">Jigsaw mysteriously refused to
	    serve 40 of the 1000 files in the test load.  Same 40 consistently.
	    I took those 40 out of the test and the other 960 files worked
	    fine.  This sort of thing doesn't really build confidence in
	    a server.
	</UL>
	<LI> <A NAME="cgi">CGI - the CGI test load consists of a
	<A HREF="hello_cgi.c.htm" tppabs="http://www.acme.com/software/thttpd/hello_cgi.c">trivial "hello world" C program</A>.
	<UL>
	    <LI> <A NAME="Roxen-CGI">Roxen started returning bad checksums
	    at only 3 parallel CGI requests.
	    <!--
	    <LI> <A NAME="Netscape-CGI">FastTrack and Enterprise started
	    returning bad checksums at 85 and 80 parallel requests,
	    respectively.  No doubt this and their piss-poor CGI service rates
	    are both due to their very large memory footprint, over 60MB.
	    But still, it's a fixable problem.  Netscape could write a
	    separate CGI-proxy process or something.  I guess their theory
	    is you should use NSAPI instead.
	    -->
	    <LI> Some of the servers do not implement the parts of the CGI
	    spec that require header-parsing - thttpd, mathopd, Boa,
	    perhaps others.  This could be seen as giving them an unfair
	    advantage at performance comparisons.
	</UL>
    </UL>
    <LI> <A NAME="max-users">Max users - maximum number of simultaneous users
    that can be handled.  This is determined by running the test load at
    various parallel-request rates and seeing when it starts screwing up.
    Typical screwups are running out of memory or processes, so that requests
    start timing out or getting refused.
    <UL>
	<LI> <A NAME="large">Large files - the large-file test load consists of
	100 files, each 1MB long, requested randomly.  Also, each connection
	is throttled to simulate a 33.6Kbps modem.  Note that 1000 33.6Kbps
	connections is 3/4 of a T3.
    </UL>
    <LI> Some servers that I wanted to benchmark but didn't get to:
    <UL>
	<LI> phttpd I couldn't get to run.
	<LI> <A NAME="Netscape-license">Netscape's
	<A HREF="tppmsgs/msgs0.htm#69" tppabs="http://home.netscape.com/download/server.html">license agreement</A>
	prohibits publishing benchmark results (section 6 viii).
    </UL>
</UL>

<P>
Here's a graph of the full data:

<P>
<IMG SRC="serverperf.png" tppabs="http://www.acme.com/software/thttpd/serverperf.png">

<P>
Comments:

<UL>
    <LI> That nice diagonal line is very interesting.  A bunch of very
    different servers follow it exactly for the first part of their curve.
    Its slope is 5 hits/second per user, indicating that each hit takes a
    minimum of 1/5th second to handle.  But other servers don't follow it,
    showing that it's not an inherent limit in the test setup.
    thttpd-2.00 was following the line, while mathopd, a very similar server,
    does not, so I made some changes to thttpd to try and get its latency
    to be more like mathopd's.  Turns out the change that made the
    difference was sending the response headers and the first load
    of data as a single packet, instead of as two separate packets.
    Apparently this avoids triggering TCP's "delayed ACK", a 1/5th
    second wait to see if more packets are coming in.  thttpd-2.01 has the
    single-packet change.
    <LI> The sharp drop-off at 125 users for a bunch of different servers
    is mysterious, especially considering it applies to servers of three
    different models - pre-fork, threads, and select - and that other
    servers are immune.
</UL>

<HR>
Back to the <A HREF="thttpd.html" tppabs="http://www.acme.com/software/thttpd/thttpd.html">thttpd page</A>.
<P>
<ADDRESS><A HREF="mailto:webmaster&#64;acme.com">ACME Labs Webmaster &lt;webmaster&#64;acme.com&gt;</A></ADDRESS>
</BODY>
</HTML>