timings

100 病毒源碼,原始碼,無毒 ......

Various timing comparisons resolving a typical ~300,000 line Apache
log file.  The results given below were typical, though there was a
standard deviation of about 10% between repeated runs.

logresolve and logresolve.pl are from Apache 1.3.6.
DNSlookups and dns-terror1.x are unreleased software written at UUNET.

Summary of wall clock run times:

program				seconds	ratios

resolving with local cache, not rewriting:

dns-terror-map-nodb		31	1.00
dns-terror-map			32	1.03
dns-terror-skiplist-nodb	33	1.06
dns-terror-skiplist		43	1.39
niced dns-terror-btree		145	4.68
dns-terror-btree		177	5.71
DNSlookups			221	7.13
dns-terror-hash			234	7.55
dns-terror1.2			315	10.16

resolving with local cache and rewriting:

dns-terror-map			64	2.06	1.00
dns-terror-skiplist-nodb	64	2.06	1.00
dns-terror-skiplist		64	2.06	1.00
dns-terror-map-nodb		66	2.13	1.03
dns-terror-btree		228	7.35	3.56
dns-terror-hash			266	8.58	4.16
logresolve.pl			618	19.94	9.66
logresolve			3225	104.03	50.39

Tests were run on an unloaded Pentium Pro 200 with 512MB RAM running
BSD/OS 3.1.

cat /etc/resolv.conf
domain web.us.uu.net
search web.us.uu.net va.pubnix.com
nameserver 127.0.0.1
nameserver 198.6.1.2
nameserver 198.6.1.3
nameserver 198.6.1.1

cat /opt/reports/db/resolv.conf 
nameserver 198.6.1.2
nameserver 198.6.1.3
nameserver 198.6.1.1

unlimit

gtime wc access.log 
  301448 4677785 46916439 access.log
6.55user 0.79system 0:07.37elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+2outputs (0major+0minor)pagefaults 0swaps

## Resolving and rewriting (filtering) logs.

gtime dns-terror -o -d remote.db -m5000 -p300 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote
301448 addresses, 9206 (3.05%) queried, 0 invalid
79.57user 29.36system 27:09.79elapsed 6%CPU (0avgtext+0avgdata 0maxresident)k
543inputs+15146outputs (0major+0minor)pagefaults 0swaps
[mean of 9 outstanding queries per mark]

gtime dns-terror -o -d local.db -m5000 -p300 -c 'nameserver 127.0.0.1' < access.log > dns-terror.local
301448 addresses, 9578 (3.18%) queried, 0 invalid
81.18user 30.54system 44:08.94elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
757inputs+15308outputs (0major+0minor)pagefaults 0swaps
[mean of 9 outstanding queries per mark]

gtime dns-terror -o -d remote6000.db -m5000 -p300 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote6000
301448 addresses, 9578 (3.18%) queried, 0 invalid
82.39user 32.81system 13:27.19elapsed 14%CPU (0avgtext+0avgdata 0maxresident)k
760inputs+15296outputs (0major+0minor)pagefaults 0swaps
[mean of 190 outstanding queries per mark]

gtime dns-terror -o -d remote10000.db -m5000 -p5000 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote10000
301448 addresses, 9578 (3.18%) queried, 0 invalid
82.56user 34.73system 9:33.86elapsed 20%CPU (0avgtext+0avgdata 0maxresident)k
762inputs+15458outputs (0major+0minor)pagefaults 0swaps
[mean of 315 outstanding queries per mark]

gtime dns-terror -o -d remote20000.db -m5000 -p1000 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote20000
301448 addresses, 9578 (3.18%) queried, 0 invalid
85.25user 34.95system 7:46.14elapsed 25%CPU (0avgtext+0avgdata 0maxresident)k
759inputs+15637outputs (0major+0minor)pagefaults 0swaps
[mean of 620 outstanding queries per mark]

gtime dns-terror -o -d remote50000.db -m5000 -p2500 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote50000
301448 addresses, 9578 (3.18%) queried, 0 invalid
96.12user 34.46system 4:25.87elapsed 49%CPU (0avgtext+0avgdata 0maxresident)k
315inputs+15216outputs (0major+0minor)pagefaults 0swaps
[mean of 1473 outstanding queries per mark]

gtime dns-terror-btree -o -d remote-btree.db -m250 -p50000 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote-btree
301448 addresses, 9578 (3.18%) queried, 0 invalid
36.38user 27.14system 3:47.58elapsed 27%CPU (0avgtext+0avgdata 0maxresident)k
763inputs+14813outputs (0major+0minor)pagefaults 0swaps
[mean of 1473 outstanding queries per mark]

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, SkipList.
gtime dns-terror -o -m5000  -d skiplist2.db -c "include /opt/reports/db/resolv.conf" < access.log > dns-terror.skiplist2
301448 entries, 9578 (3.18%) queried, 0 invalid
24.67user 5.36system 1:04.18elapsed 46%CPU (0avgtext+0avgdata 0maxresident)k
554inputs+83outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, SkipList.
# No DB file.
gtime dns-terror -o -m5000  -d '' -c "include /opt/reports/db/resolv.conf" < access.log > dns-terror.nodb
301448 entries, 9578 (3.18%) queried, 0 invalid
23.88user 5.81system 1:03.68elapsed 46%CPU (0avgtext+0avgdata 0maxresident)k
545inputs+61outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, STL map.
gtime dns-terror -o -m5000 -d map-deque2.db -c "include /opt/reports/db/resolv.conf" < access.log > dns-terror.map-deque2
301448 entries, 9578 (3.18%) queried, 0 invalid
19.59user 5.14system 1:03.86elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
557inputs+73outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, STL map.
# No DB file.
gtime dns-terror -o -m5000 -d '' -c "include /opt/reports/db/resolv.conf" < access.log > dns-terror.map-deque6       
301448 entries, 9578 (3.18%) queried, 0 invalid
19.87user 5.43system 1:05.60elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
746inputs+54outputs (0major+0minor)pagefaults 0swaps

gtime dns-terror -o -d remote50000-5.db -m5000 -p2500 -c 'include /opt/reports/db/resolv.conf
options udpmaxretries:5' < access.log > dns-terror.remote50000-5
301448 addresses, 9578 (3.18%) queried, 0 invalid
129.69user 25.05system 4:58.37elapsed 51%CPU (0avgtext+0avgdata 0maxresident)k
767inputs+12494outputs (0major+0minor)pagefaults 0swaps
[mean of 1473 outstanding queries per mark]

gtime dns-terror -o -d remote100000.db -m5000 -p5000 -c 'include /opt/reports/db/resolv.conf' < access.log > dns-terror.remote100000
301448 addresses, 9578 (3.18%) queried, 0 invalid
113.34user 30.98system 4:42.90elapsed 51%CPU (0avgtext+0avgdata 0maxresident)k
665inputs+14700outputs (0major+0minor)pagefaults 0swaps
[mean of 2678 outstanding queries per mark]

gtime dns-terror -o -d remote-jeeves.db -m5000 -p2500 -c 'nameserver 208.240.88.4' < access.log > dns-terror.remote-jeeves
301448 addresses, 9578 (3.18%) queried, 0 invalid
103.09user 30.68system 4:58.43elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k
757inputs+14826outputs (0major+0minor)pagefaults 0swaps
[mean of 1473 outstanding queries per mark]

gtime logresolve.pl < access.log > logresolve.pl.out
42.01user 17.90system 10:18.11elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
746inputs+977outputs (0major+0minor)pagefaults 0swaps

gtime logresolve < access.log > logresolve.out
9.08user 3.71system 53:44.64elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k
752inputs+428outputs (0major+0minor)pagefaults 0swaps

## Producing an Analog DNS cache file.

gtime convert-ip-db remote.db > remote.cache
2.79user 0.12system 0:03.04elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
4inputs+19outputs (0major+0minor)pagefaults 0swaps

## Only resolving to a database, no rewrites.

gtime dns-terror -d nowrite.db -m5000 -p2500 < access.log
301448 addresses, 9578 (3.18%) queried, 0 invalid
75.78user 23.21system 3:54.06elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k
732inputs+12532outputs (0major+0minor)pagefaults 0swaps
[mean of 2245 outstanding queries per mark]

gtime dns-terror-btree -d nowrite-btree.db -m5000 -p2500 < access.log  
301448 addresses, 9578 (3.18%) queried, 0 invalid
22.44user 19.66system 2:56.62elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k
2inputs+13272outputs (0major+0minor)pagefaults 0swaps
[mean of 2245 outstanding queries per mark]

gtime dns-terror-btree -d nowrite-btree.db -m5000 -p2500 -c 'include /opt/reports/db/resolv.conf' < access.log
301448 addresses, 9578 (3.18%) queried, 0 invalid
23.10user 17.97system 2:54.37elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k
2inputs+13360outputs (0major+0minor)pagefaults 0swaps
[mean of 2245 outstanding queries per mark]

# As root.  btree version, udpmaxretries:5.
gtime nice --15 dns-terror -m5000 -p2500 -d nice.db -c "include /opt/reports/db/resolv.conf" < access.log
301448 entries, 9578 (3.18%) queried, 0 invalid
17.44user 12.84system 2:35.46elapsed 19%CPU (0avgtext+0avgdata 0maxresident)k
318inputs+12943outputs (0major+0minor)pagefaults 0swaps
[mean of 2245 outstanding queries per mark]

# As root.  btree version, udpmaxretries:5, DB buffer cache of 20Mb.
gtime nice --15 dns-terror -m5000 -p2500 -d nice.db -c "include /opt/reports/db/resolv.conf" < access.log
301448 entries, 9578 (3.18%) queried, 0 invalid
17.71user 12.30system 2:27.68elapsed 20%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+12926outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 20Mb.
# Ensuring that DB file is held entirely in memory.
gtime dns-terror -m5000 -p2500 -d '' nice.db -c "include /opt/reports/db/resolv.conf" < access.log
301448 entries, 9578 (3.18%) queried, 0 invalid
22.99user 19.05system 2:34.44elapsed 27%CPU (0avgtext+0avgdata 0maxresident)k
25inputs+13405outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, SkipList.
gtime dns-terror -m5000 -p2500 -d skiplist.db -c "include /opt/reports/db/resolv.conf" < access.log
301448 entries, 9578 (3.18%) queried, 0 invalid
18.35user 5.63system 0:42.98elapsed 55%CPU (0avgtext+0avgdata 0maxresident)k
727inputs+18outputs (0major+0minor)pagefaults 0swaps
[mean of 2245 outstanding queries per mark]

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, STL map.
gtime dns-terror -m5000 -p2500 -d map-deque.db -c "include /opt/reports/db/resolv.conf" < access.log 
301448 entries, 9578 (3.18%) queried, 0 invalid
19.09user 5.03system 0:31.83elapsed 75%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+22outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, SkipList.
# No DB file.
gtime dns-terror -m5000 -p2500 -d '' -c "include /opt/reports/db/resolv.conf" < access.log
301448 entries, 9578 (3.18%) queried, 0 invalid
20.59user 5.12system 0:33.30elapsed 77%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+3outputs (0major+0minor)pagefaults 0swaps

# btree version, udpmaxretries:5, DB buffer cache of 2Mb, STL map.
# No DB file.
gtime dns-terror -m5000 -p2500 -d '' -c "include /opt/reports/db/resolv.conf" < access.log
301448 entries, 9578 (3.18%) queried, 0 invalid
19.00user 4.90system 0:31.31elapsed 76%CPU (0avgtext+0avgdata 0maxresident)k

# A perl parallizing front-end to dns-terror:
gtime teamresolve -p12 access.log
22.65user 10.17system 0:47.28elapsed 69%CPU (0avgtext+0avgdata 0maxresident)k
730inputs+636outputs (0major+0minor)pagefaults 0swaps

gtime teamresolve -p5 access.log
26.55user 9.25system 0:36.95elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
1inputs+358outputs (0major+0minor)pagefaults 0swaps

gtime teamresolve -p3 access.log
27.81user 9.30system 0:37.45elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
1inputs+239outputs (0major+0minor)pagefaults 0swaps

gtime dns-terror1.1 < access.log 
-- A total of 9578 [unique] requests have been seen.
2778 unique requests, 74076 total requests, 3.7502% unique.
Issued 6353 queries, avg of 2.2869 queries per request
708 (25.486%) queries never answered.
2810 unique requests, 91438 total requests, 3.07312% unique.
Issued 6382 queries, avg of 2.27117 queries per request
720 (25.6228%) queries never answered.
3990 unique requests, 135934 total requests, 2.93525% unique.
Issued 6361 queries, avg of 1.59424 queries per request
227 (5.68922%) queries never answered.
Had 0 bad lines
167.55user 5.98system 5:20.06elapsed 54%CPU (0avgtext+0avgdata 0maxresident)k