testscantprovetheabsenceofbugs.html

极限编程 Extream Programing

Isn't this a problem when you choose to measure stability and productivity in
terms of those tests? (i.e. you can't use <em>exactly</em> the same tests.)
<p>
Is this not a problem in practice? If it is a problem, how do you live with it?
<p>
--<a href="http://c2.com/cgi/wiki?DafyddRees">DafyddRees</a>
<p>
In refactoring, you are either changing implementation, or you are changing interface. It is <em>bad</em> to change both at once. If you are changing only implementation, then the tests do not require changing.
<p>
If you are changing interface, the tests need changing. Generally, of course, they are quite similar to the old ones. 
<p>
It is in no way a problem: if you were implementing <em>new</em> functionality, you'd have to do tests, typically from scratch. New tests are not a problem. If you're refactoring interface, you still have to do tests, and they aren't even as hard as new tests, which aren't hard.
<p>
As always, with new code or refactoring, if a test breaks, there are two possibilities: the code is wrong, or the test is wrong. If the developer can't tell which, he should consider a job in Marketing. ;-&gt;  --<a href="RonJeffries.html">RonJeffries</a>
<p>
<hr>
<p>
What about ProbabilisticProofs<a href="http://c2.com/cgi/wiki?edit=ProbabilisticProofs">?</a>?  The reason we can generate large prime numbers for cryptography is that we use a prime testing algorithm that's very fast, only it goofs up from time to time.  But the algorithm also has the property that the more times we run run it successfully on a number, the more certain we can be that the number is prime.  This really pisses off the purists, but seems to work in the real world.
<p>
Maybe a sufficient suite of unit (and functional) tests serve as a probabilistic proof that the system is correct.
<p>
--<a href="http://c2.com/cgi/wiki?JohnBrewer">JohnBrewer</a>
<p>
It's refreshing to hear this point of view. People get hung up on extremes, either its provable or everything sucks. After all, just because a test can't <em>definitively prove</em> the absence of bugs in a system does not mean that we as engineers shouldn't test that system as rigorously as possible. It's funny how hung-up we get on extremes. So, I would agree, you don't have to convince everyone that software can be proven correct in order to show the value of testing. I think someone (maybe <a href="http://c2.com/cgi/wiki?WattsHumphrey">WattsHumphrey</a>?) came up with the maxim that if you have found a bug, you can be certain there is another one. :) --<a href="http://c2.com/cgi/wiki?RobertDiFalco">RobertDiFalco</a>
<p>
<em>That's a pretty safe conclusion even if you haven't found a bug yet. --<a href="http://c2.com/cgi/wiki?GarethMcCaughan">GarethMcCaughan</a></em>
<p>
I think <a href="http://c2.com/cgi/wiki?WattsHumphrey">WattsHumphrey</a>'s view is that finding a bug in a particular area suggests that there are more bugs in that area.  Example: code that has a lot of syntax errors will likely also have more serious errors.  Not sure how that fits on this page... --<a href="http://c2.com/cgi/wiki?JasonYip">JasonYip</a>
<p>
Yeah, that was it. Maybe I was mistaken, but it seemed relevant to the idea that you cannot prove the <em>absence</em> of bugs. In fact, finding a bug is often an indication that more serious bugs exist. So, while testing can not prove the <em>absence</em> of bugs but it <em>can</em> prove their existence -- and this is an important and good thing. This allows the engineer to remove <em>those</em> detected bugs and, hence, increase confidence in the system. Seems logical to me. Another great thing about <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a> is that (when you are rigorous about them and add them to your daily build) you can waste less of QA's time when they do integration testing. There is no excuse for giving QA a build that has errors in the implementation (un-integrated).
<p>
This will be slightly off-topic, but I have always been pretty religious about developing many small and atomic unit tests that map to component interfaces. Actually, I use to create a single <a href="http://c2.com/cgi/wiki?TestCase">TestCase</a> for every public method. Now I just make sure that each method works, write tests to simulate every fail-condition, and make sure that running everyone's <a href="http://c2.com/cgi/wiki?UnitTest">UnitTest</a><strong></strong>s is an automatic part of the DailyBuild<a href="http://c2.com/cgi/wiki?edit=DailyBuild">?</a> script. However, (while admittedly a subtle distinction) I don't really believe that I am <em>removing</em> any bugs (nor <em>proving the absence of new bugs</em>). Actually, I visualize my unit tests much like an extension to the traditional <a href="http://c2.com/cgi/wiki?DesignByContract">DesignByContract</a>. The TestMethods<a href="http://c2.com/cgi/wiki?edit=TestMethods">?</a> provide a harness for excercising these contracts. My general rule is that I shouldn't have behavior (for failure or success) that can't be asserted in some way. The more rigorous my assertions, the higher my confidence level is in the code's behavior. So, in this way, each <a href="http://c2.com/cgi/wiki?TestCase">TestCase</a> is really just a harness or an <em>artificial agitator</em> for my Contracts.
<p>
Finally, just as I would never <strong>skip</strong> <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a>, I would also never be naive enough to think <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a> alone is sufficient. And so, I <em>also</em> make certain that QA performs black and white box functionality tests that don't test my implementation as much as they test the <em>solutions</em> my units of implementation provide when <em>integrated</em> together. --<a href="http://c2.com/cgi/wiki?RobertDiFalco">RobertDiFalco</a>
<p>
There is evidence of <em>BadModules<a href="http://c2.com/cgi/wiki?edit=BadModules">?</a></em> where one finds 80% of a system's faults in 20% of its modules.  This seems to be true, so far, of all systems whatever methodology or tools are used.
<p>
<hr>
<em>That's a pretty safe conclusion even if you haven't found a bug yet.</em>
<p>
Another thought experiment. Two modules to run your pacemaker have undergone large amounts of testing, equivalent as far as you know. From module A no defects have ever been found. From module B, hundreds have been found so far. Your pacemaker has a JulY2K problem and you must choose a new module NOW, or die. Which one do you choose? Justify your answer.  --<a href="RonJeffries.html">RonJeffries</a>
<p>
<em>I know what you are trying to prove, Ron - but your example has major holes. Obviously, it must depend on the kind of bugs found, how much actual time the two have been in use, the reputation of the manufacturers and many other factors. Otherwise, we would expect a bugridden OS like Windows to fail utterly compared with the much more stable Linux.</em>
<p>
<em>The observation that bugs have been found is certainly a good predictor that still more remain - but it is not a reliable predictor that people will not use the product -- <a href="http://c2.com/cgi/wiki?RussellGold">RussellGold</a></em>
<p>
All you know is that both modules have undergone extensive testing, equivalent as far as you know. One has never exhibited a defect, one has exhibited hundreds. Your heart is stopping. Pick one. Justify your answer. --rj
<p>
<hr>
<p>
Ron, if both underwent rigorous testing, I would have to select module A. Just by the fact that hundreds have been found in module B, I can pretty safely bet that many more are yet to be found. However, since so few (actually none) have been found in module A, chances are that none <em>will</em> be found. Is this where you were heading? Which would you pick? BTW, I think this was a useful thought experiment. --<a href="http://c2.com/cgi/wiki?RobertDiFalco">RobertDiFalco</a>
<p>
<em>A. I can accept no errors in my pacemaker. Only one of the systems has exhibited no errors. End of consideration. For extra credit, consider pacemaker software C, which has shown a declining error curve. It started with lots, but no new defects have been discovered for a &quot;long time&quot; now. Do we prefer it to B? Do we prefer it to A? And why? Your heart is stopping. Pick one. ;-&gt;</em>
<p>
Ron, I would still choose <em>module A</em>. I still prefer code which has never shown bugs in production testing to one that has radically reduced its number of bugs. For anyone who disagrees with the concept of <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a> here's another question. How can you achieve the success of <em>module A</em> without <a href="http://c2.com/cgi/wiki?UnitTesting">UnitTesting</a>? Personally, I think the only answer is either <em>serindipity</em> or a team mate who did some unit testing behind your back. :) --<a href="http://c2.com/cgi/wiki?RobertDiFalco">RobertDiFalco</a>
<p>
This is the <a href="http://c2.com/cgi/wiki?TheresTheBug">TheresTheBug</a> principle.
<p>
<em>My flippant remark appears to be being taken much more seriously than it deserved. Anyway, obviously the answer to the pacemaker question is that I pick module A rather than module B. If pacemakers actually contain non-trivial software (which I rather doubt, but what do I know?) then I'd guess A probably <strong>has</strong> bugs, but clearly any bugs it has show up less than the bugs in B. Do I prefer A or C? Depends on the amount of testing they've had. If the total is the same, then C's latest version, despite not having exhibited any bugs, hasn't been tested as much as A's (presumably only) version. If the total on C's latest version equals the total on A's only version, I might pick C. I don't think any of this is surprising, unless someone here is claiming that testing isn't useful. I doubt anyone is. --<a href="http://c2.com/cgi/wiki?GarethMcCaughan">GarethMcCaughan</a></em>
<p>
I'm going to step out on a limb since I haven't actually worked on a pacemaker but from talking to biomedical engineers, they do consider pacemaker software to be trivial.  Or at least, it is straight forward to formally verify the algorithms.  Software tends not to be the main failure condition but rather rejection and hardware problems. --<a href="http://c2.com/cgi/wiki?JasonYip">JasonYip</a>
<hr>
<a href="http://c2.com/cgi/wiki?edit=TestsCantProveTheAbsenceOfBugs">EditText</a> of this page (last edited July 5, 2000)<br>
<a href="http://c2.com/cgi/wiki?FindPage&value=TestsCantProveTheAbsenceOfBugs">FindPage</a> by browsing or searching<p>
<font color=gray size=-1>This page mirrored in <a href="index.html">ExtremeProgrammingRoadmap</a> as of March 31, 2001</font>
</body>