testing.html

sqlite3源码,适合作为嵌入式（embedded）

buffers, dereferencing NULL pointers, or performing other
unwholesome actions.</p>

<h3>4.3 Boundary Value Tests</h3>

<p>SQLite defines certain <a href="limits.html">limits</a> on its operation, such as the
maximum number of columns in a table, the maximum length of an 
SQL statement, or the maximum value of an integer.  The TCL test
suite contains numerous tests that push SQLite right to the edge
of its defined limits and verify that it performs correctly for
all allowed values.  Additional tests go beyond the defined limits
and verify that SQLite correctly returns errors.</p>

<h2>5.0 Regression Testing</h2>

<p>Whenever a bug is reported against SQLite, that bug is not considered
fixed until new test cases have been added to the TCL test suite which
would exhibit the bug in an unpatched version of SQLite.  Over the years,
this has resulted in thousands and thousands of new tests being added
to the TCL test suite.  These regression tests insure that bugs that have
been fixed in the past are never reintroduced into future versions of
SQLite.</p>

<h2>6.0 Automatic Resource Leak Detection</h2>

<p>Resource leak occurs when system resources
are allocated and never freed.  The most troublesome resource leaks
in many applications are memory leaks - when memory is allocated using
malloc() but never released using free().  But other kinds of resources
can also be linked:  file descriptors, threads, mutexes, etc.</p>

<p>Both the TCL and TH3 test harnesses automatically track system
resources and report resources leaks on <u>every</u> test run.
No special configuration or setup is required.   The test harnesses
are especially vigilant with regard to memory leaks.  If a change
causes a memory leak, the test harnesses will recognize this
quickly.  SQLite is designed to never leak memory, even after
an exception such as an OOM error or disk I/O error.  The test
harnesses are zealous to enforce this.</p>

<h2>7.0 Test Coverage</h2>

<p>The <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
utility is used to measure the "test coverage" of the SQLite test suite.
SQLite strives for but does not yet obtain 100% test coverage.  A major
goal of the SQLite project is to obtain 100% condition/decision coverage
during 2009.</P.

<p>Test coverage can be measured in several ways.  "Statement coverage"
measures (as a percentage of the whole) how many lines of code are
exercised by the test cases. The TCL test suite obtains
99.38% statement coverage on the SQLite
core.  (The SQLite core, in this case, excludes the operating-system
dependent <a href="c3ref/vfs.html">VFS</a> backends.)
"Condition/Decision" or "C/D" coverage measures (again, as a percentage of the
whole) how many machine-code branch instructions are taken at least
once in both directions.  The TCL test suite obtains
94.97% C/D coverage.</p>

<p>To illustrate the difference between statement coverage and
condition/decision coverage, consider the following hypothetical
line of C code:</p>

<blockquote><pre>
if( a>b && c!=25 ){ d++; }
</pre></blockquote>

<p>Such a line of C code might generate a dozen separate machine code
instructions.  If any one of those instructions is ever evaluated, then
we say that the statement has been tested.  So, for example, it might
be the case that the condition is always false and the "d" variable is
never incremented.  Even so, statement coverage counts this line of
code as having been tested.</p>

<p>C/D coverage is more strict.  With C/D coverage, each test and
each subblock within the statement is considered separately.  In order
to achieve 100% C/D coverage in the example above, there must be at
least three test cases:</p>

<p><ul>
<li> a<=b
<li> a>b && c==25
<li> a>b && c!=25
</ul></p>

<p>C/D test coverage is normally less than statement coverage since
a C program will typically contain some defensive tests which in practice
are always true or always false.  
For testing purposes, the SQLite source code defines
macros called ALWAYS() and NEVER().   The ALWAYS() macro
surrounds conditions
which are expected to always evaluated to true and NEVER() surrounds
conditions that are always evaluate to false.  These macros serve as
comments to indicate that the conditions are defensive code.
For standard builds, these macros are pass-throughs:</p>

<blockquote><pre>
#define ALWAYS(X)  (X)
#define NEVER(X)   (X)
</pre></blockquote>

<p>During most testing, however, these macros will throw an assertion
fault if their argument does not have the expected truth value.  This
alerts the developers quickly to incorrect design assumptions.

<blockquote><pre>
#define ALWAYS(X)  ((X)?1:assert(0),0)
#define NEVER(X)   ((X)?assert(0),1:0)
</pre></blockquote>

<p>When measuring test coverage, these macros are defined to be constant
truth values so that they do not generate assembly language branch
instructions, and hence do not come into play when calculating the
C/D coverage level:</p>

<blockquote><pre>
#define ALWAYS(X)  (1)
#define NEVER(X)   (0)
</pre></blockquote>

<p>Another macro used in conjuction with test coverage measurement is
the <tt>testcase()</tt> macro.  The argument is a condition for which
we want test cases that evaluate to both true and false.
In non-coverage builds (that is to so, in release builds) the testcase()
macro is a no-op:</p>

<blockquote><pre>
#define testcase(X)
</pre></blockquote>

<p>But in a coverage measuring build, the testcase() macro generates code
that evaluates the condition in its argument.  Then during analysis, a check
is made to insure tests exists that evaluate the conditional to both true
and false.  Testcase() macros are used, for example, to help verify that
boundary values are tested.  For example:</p>

<blockquote><pre>
testcase( a==b );
testcase( a==b+1 );
if( a>b && c!=25 ){ d++; }
</pre></blockquote>

<p>Testcase macros are also used when two or more cases of a switch
statement go to the same block of code, to make sure that the code was
reached for all cases:</p>

<blockquote><pre>
switch( op ){
  case OP_Add:
  case OP_Subtract: {
    testcase( op==OP_Add );
    testcase( op==OP_Subtract );
    /* ... */
    break;
  }
  /* ... */
}
</pre></blockquote>

<p>For bitmask tests, testcase() macros are used to verify that every
bit of the bitmask effects the test.  For example, in the following block
of code, the condition is true if the mask contains either of two bits
indicating either a MAIN_DB or a TEMP_DB is being opened.  The testcase()
macros that preceed the if statement verify that both cases are tested:</p>

<blockquote><pre>
testcase( mask & SQLITE_OPEN_MAIN_DB );
testcase( mask & SQLITE_OPEN_TEMP_DB );
if( (mask & (SQLITE_OPEN_MAIN_DB|SQLITE_OPEN_TEMP_DB))!=0 ){ ... }
</pre></blockquote>

<p>The developers of SQLite have found that coverage testing is an
extremely productive method for finding bugs.  Because such a high
percentage of SQLite core code is covered by test cases, the developers
can have confident that changes they make in one part of the code
do not have unintended consequences in other parts of the code.
It would not be possible to maintain the quality of SQLite without
coverage testing.</p>

<h2>8.0 Dynamic Analysis</h2>

<p>Dynamic analysis refers to internal and external checks on the
SQLite code which are performed while the code is live and running.
Dynamic analysis has proven to be a great help in maintaining the
quality of SQLite.</p>

<h3>8.1 Assert</h3>

<p>The SQLite core contains 2450 <tt>assert()</tt>
statements that verify function preconditions and postconditions and
loop invariants.  Assert() is a macro which is a standard part of
ANSI-C.  The argument is a boolean value that is assumed to always be
true.  If the assertion is false, the program prints an error message
and halts.</p>

<p>Assert() macros are disabled by compiling with the NDEBUG macro defined.
In most systems, asserts are enabled by default.  But in SQLite, the
asserts are so numerous and are in such performance critical places, that
the database engine runs about three times slower when asserts are enabled.
Hence, the default (production) build of SQLite disables asserts.  
Assert statements are only enabled when SQLite is compiled with the
SQLITE_DEBUG preprocessor macro defined.</p>

<h3>8.2 Valgrind</h3>

<p><a href="http://valgrind.org/">Valgrind</a> is perhaps the most amazing
and useful developer tool in the world.  Valgrind is a simulator - it simulates
an x86 running a linux binary.  (Ports of valgrind for platforms other
than linux are in development, but as of this writing, valgrind only
works reliably on linux, which in the opinion of the SQLite developers 
means that linux should be preferred platform for all software development.)
As valgrind runs a linux binary, it looks for all kinds of interesting
errors such as array overruns, reading from uninitialized memory,
stack overflows, memory leaks, and so forth.  Valgrind finds problems
that can easily slip through all of the other tests run against SQLite.
And, when valgrind does find an error, it can dump the developer directly
into a symbolic debugger at the exact point where the error occur, to
facilitate a quick fix.</p>

<p>Because it is a simulator, running a binary in valgrind is slower than 
running it on native hardware.  So it is impractical to run the full
SQLite test suite through valgrind.  However, the veryquick tests and
a subset of the TH3 tests are run through valgrind prior to every release.</p>

<h3>8.3 Memsys2</h3>

<p>SQLite contains a pluggable memory allocation subsystem.
The default implementation uses system malloc() and free(). 
However, if SQLite is compiled with SQLITE_MEMDEBUG, an alternative
memory allocation wrapper (<a href="malloc.html#memdebug">memsys2</a>)
is inserted that looks for memory allocation
errors at run-time.  The memsys2 wrapper checks for memory leaks, of
course, but also looks for buffer overruns, uses of uninitialized memory,
and attempts to use memory after it has been freed.  These same checks
are also done by valgrind (and, indeed, valgrind does them better)
but memsys2 has the advantage of being much faster than valgrind, which
means the checks can be done more often and for longer tests.</p>

<h3>8.4 Mutex Asserts</h3>

<p>SQLite contains a pluggable mutex subsystem.  Depending on 
compile-time options, the default mutex system contains interfaces
<a href="c3ref/mutex_held.html">sqlite3_mutex_held()</a> and <a href="c3ref/mutex_held.html">sqlite3_mutex_notheld()</a> that detect
whether or not a particular mutex is held by the calling thread.
These two interfaces are used extensively within assert() statements
in SQLite to verify mutexes are held and released at all the right
moments, in order to double-check that SQLite does work correctly
in multi-threaded applications.</p>

<h3>8.5 Journal Tests</h3>

<p>One of the things that SQLite does to insure that transactions
are atomic across system crashes and power failures is to write
all changes into the rollback journal file prior to changing the
database.  The TCL test harness contains an alternative
<a href="c3ref/vfs.html">Virtual File System</a> implementation that helps to
verify this is occurring correctly.  The "journal-test VFS" monitors
all disk I/O traffic between the database file and rollback journal,
checking to make sure that nothing is written into the database
file which has not first by written and synced to the rollback journal.
If any discrepancies are found, an assertion fault is raised.</p>

<p>The journal tests are an additional double-check over and above
the crash tests to make sure that SQLite transactions will be atomic
across system crashes and power failures.</p>

<h2>9.0 Static Analysis</h2>

<p>Static analysis means analyzing code at or before compile-time to
check for correctness.  Static analysis consists mostly of making
sure SQLite compiles without warnings, even when all warnings are
enabled.  SQLite is developed primarily using GCC and it does
compile without warnings on GCC using the -Wall and -Wextra flags.
There are occasional reports of warnings coming from VC++, however.</p>

<p>Static analysis has not proven to be helpful in finding
bugs.  We cannot call to mind a single problem in SQLite that
was detected by static analysis that was not first seen by one
of the other testing methods described above.  On the other hand,
we have on occasion introduced new bugs in our efforts to get SQLite
to compile without warnings.</p>

<p>Our experience, then, is that static analysis is counter-productive
to quality.  In other words, focusing on static analysis (being
concerned with compiler warnings) actually reduces the quality of the
code.  Nevertheless, we developers have capitulated to pressure from
users and actively work to eliminate compiler warnings.  We are
willing to do this because the other tests described above do an
excellent job of finding the bugs that are often introduced when
removing compiler warnings, so that product quality is probably not
decreased as a result.</p>

<h2>10.0 Summary</h2>

<p>SQLite is open source.  That give many people with the idea that
it is not well tested and is perhaps unreliable.  But that impression is
false.  SQLite has exhibited very high reliability in the field and
a very low defect rate, especially considering how rapidly it is evolving.
The quality of SQLite is achieved in part by careful code design and
implementation.  But extensive testing also plays a vital role in
maintaining and improving the quality of SQLite.  This document has
summarized the testing procedures that every release of SQLite undergoes
with the hopes of inspiring the reader to understand that SQLite is
suitable for use in mission-critical applications.</p>
<hr><small><i>
This page last modified 2009/01/15 15:44:09 UTC
</i></small></div></body></html>