📄 testscantprovetheabsenceofbugs.html
字号:
<head><title>Tests Cant Prove The Absence Of Bugs</title></head><body><h1><img src="logo.gif"> Tests Cant Prove The Absence Of Bugs</h1>Regarding <a href="http://c2.com/cgi/wiki?KentBecksChangeCostXpArticle">KentBecksChangeCostXpArticle</a>,
<p><strong>Issue #2: You didn't really KNOW that the change was safe.</strong>
<p>There being no instances of ">2 line" transactions in the data does not prove that there is no code that can produce them; it only shows that in that period of time none did.
If the system imported transactions from another system, for example, it might be just a coincidence that they had not yet done "year end processing" or some other process that produces "N line" transactions.
And after refactoring this functionality out of the system, what would you do on the day that the system breaks <strong>online</strong>, because you missed one of the cases?
<p>Here's another reason to be nervous: None of the unit tests failed.
How well are you testing the functionality of the Transaction class, if changing some of its methods so they throw away data does not cause any of the tests to fail?
<p>-- <a href="http://c2.com/cgi/wiki?JeffGrigg">JeffGrigg</a>
<hr>
<em>So true, they can't. Consider two new airframes. The one on the left has flown 10,000 hours of tests. At the beginning it crashed a lot. Lately it hasn't crashed at all. The one on the right hasn't been tested. You need to get home from this desert island. Which one will you ride in? Justify your answer. --<a href="RonJeffries.html">RonJeffries</a></em>
<hr>
<em>This concern appears to be based on the notion that the <a href="UnitTests.html">UnitTests</a> were devised at random, not with careful attention. While it is certainly the case that errors do slip through conscientious testing, it's still way ahead of the usual alternative, i.e. </em>
<strong>not testing. </strong>
<em>Jeff, what alternative practice do you follow that provides even better certainty during refactoring?</em>
<hr>
This is the old saw that tests can only prove the presence of features, not the absence of bugs. True, but irrelevant for our purposes. There is some defect rate for any project below which you are willing to act like you don't have defects as long as the tests run.
<p>We crossed this threshold while I was working on <a href="http://c2.com/cgi/wiki?LifeTech">LifeTech</a>. At first nobody could make a single move because they would break something else. (<em><a href="http://c2.com/cgi/wiki?ExtremeFrustration">ExtremeFrustration</a>!</em>) After about four months of OnDemandUnitTesting<a href="http://c2.com/cgi/wiki?edit=OnDemandUnitTesting">?</a> (see <a href="http://c2.com/cgi/wiki?UnitTestTrial">UnitTestTrial</a>, I think), we changed from thinking "all the tests running just proves we don't have enough tests" to thinking "all the tests run, we can't think of anything that might break, so put it into production".
<p>Most times we were right, but occasionally we were wrong and we would put code into production that broke. This happened less and less frequently as we got more tests. Every time something broke in production we learned about another kind of test we needed to write. --<a href="KentBeck.html">KentBeck</a>
<hr>
Would coverage tools help ensure the completeness of testing? Clearly, nothing will let you say, "there are absolutely no bugs", but being able to state "we've executed every line of code, and all our tests passed" adds to the comfort level.
<p>Coverage tools add comfort, for sure. The fact that coverage is complete also increases the number of "edge" cases that are checked. However, it's coverage in combination that really does the checking - this generally requires human thought: "OK, but what if he's at the savings match and gets a raise this month". Don't let your coverage tester discourage thinking and discussion of what to test.
<hr>
When I first met Kent I did not understand this either. Let me try to explain my newfound understanding. Many people assume that <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a> as used within <a href="ExtremeProgramming.html">ExtremeProgramming</a> is the same as <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a> as per some other methodology. There are some subtle differences that make it more effective.
<p>The first is a misconception that comes from familiarity with <a href="BigBangTesting.html">BigBangTesting</a>. That is, you sit down and just test and test your code all in one session. And you always, always miss several big bugs. Thus you find <a href="TestsCantProveTheAbsenceOfBugs.html">TestsCantProveTheAbsenceOfBugs</a>. But this is not XP style. With XP you create tests before you code, while you code, and then even after you code. You create tests whenever a bug is found. You create tests when you look at someone else's code and see a testing hole. And you don't just throw those tests away after you believe you are done. You evolve and expand your test suite over time. The testing suite that results is (trust me) very different than what you might imagine.
<p>The second misconception comes from a lack of <a href="YouArentGonnaNeedIt.html">YouArentGonnaNeedIt</a>. That is you wonder how much of your system is not being exercised by your tests. The XP answer is none of it. You have been very careful not to include anything extra and you tested 100% of everything you did include. If you missed something someone else will add a test for it sooner than later.
<p>After a while (and I am talking about only a few months of evolution) you will have a collection of tests which do in fact prove the absence of many bugs. Not all of course, but any relevant ones will be covered. --<a href="DonWells.html">DonWells</a>
<hr>
I believe the objection is to the use of the word "prove", which has a particular mathematical interpretation. I don't usually use "prove" for exactly this reason, although I can believe I boo-bood in the article. The question is whether or not you can act like you have proved the absence of defects, even though you know you haven't, not in the mathematical sense of the word. --<a href="KentBeck.html">KentBeck</a>
<p><a href="ProofsCantProveTheAbsenceOfBugs.html">ProofsCantProveTheAbsenceOfBugs</a>, either.
<hr>
Passing a test suite just means you are ready to release the software. This is a good thing, but there's always another bug.
<p>To illustrate, a story by <a href="http://c2.com/cgi/wiki?EricUlevik">EricUlevik</a> (don't trust the details):
<p>Doing forex trading, we found a case where our software would get a bad calculation. Once a week or so. Eventually, the problem turned out to be a failure in a CPU cache line refresh. This was a hardware design fault in the PC.
<p>The test suite included running for two weeks at maximum update rate without error, so this bug was found.
<p>If you need this level of quality, a good place to look is Intel's web site for processor errata.
<hr>
Note that after restricting the functionality of the Transaction object, all the unit tests ran successfully.
While this does increase one's confidence that the change did not break existing functionality (the most severe risk of a change like this), it also leads to another, more disturbing conclusion:
<strong>The unit tests for the Transaction class are not sufficient to detect this loss of functionality.</strong>
My response would be to add more Transaction unit tests.
-- <a href="http://c2.com/cgi/wiki?JeffGrigg">JeffGrigg</a>
<p>It depends. If the functionality is required, i.e. you really do have a card for it, then test it and leave it in. The test was erroneously left out. On the other hand, if the functionality is not an immediate requirement, you should invoke YAGNI and remove the functionality rather than test it.
<p>Either way, by paying attention to having tests and functionality aligned, our confidence in the sytem increases.
<p><hr>
A question for the XP gurus:
<p>Doesn't refactoring (which will change the visible interface of classes) lead to a restatement of some tests (unit and functional)?
<p>⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -