📄 testing.html
字号:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><html><head><title>How SQLite Is Tested</title><style type="text/css">body { margin: auto; font-family: "Verdana" "sans-serif"; padding: 8px 1%;}a { color: #45735f }a:visited { color: #734559 }.logo { position:absolute; margin:3px; }.tagline { float:right; text-align:right; font-style:italic; width:240px; margin:12px; margin-top:58px;}.toolbar { font-variant: small-caps; text-align: center; line-height: 1.6em; margin: 0; padding:1px 8px;}.toolbar a { color: white; text-decoration: none; padding: 6px 12px; }.toolbar a:visited { color: white; }.toolbar a:hover { color: #80a796; background: white; }.content { margin: 5%; }.content dt { font-weight:bold; }.content dd { margin-bottom: 25px; margin-left:20%; }.content ul { padding:0px; padding-left: 15px; margin:0px; }/* rounded corners */.se { background: url(images/se.png) 100% 100% no-repeat #80a796}.sw { background: url(images/sw.png) 0% 100% no-repeat }.ne { background: url(images/ne.png) 100% 0% no-repeat }.nw { background: url(images/nw.png) 0% 0% no-repeat }</style><meta http-equiv="content-type" content="text/html; charset=UTF-8"> </head><body><div><!-- container div to satisfy validator --><a href="index.html"><img class="logo" src="images/SQLite.gif" alt="SQLite Logo" border="0"></a><div><!-- IE hack to prevent disappearing logo--></div><div class="tagline">Small. Fast. Reliable.<br>Choose any three.</div><table width=100% style="clear:both"><tr><td> <div class="se"><div class="sw"><div class="ne"><div class="nw"> <div class="toolbar"> <a href="about.html">About</a> <a href="sitemap.html">Sitemap</a> <a href="docs.html">Documentation</a> <a href="download.html">Download</a> <a href="copyright.html">License</a> <a href="news.html">News</a> <a href="http://www.sqlite.org/cvstrac/index">Developers</a> <a href="support.html">Support</a> </div></div></div></div></div></td></tr></table> <h1 align="center">How SQLite Is Tested</h1><h2>1.0 Introduction</h2><p>The reliability and robustness of SQLite is achieved in large partby thorough and careful testing.</p><p>As of <a href="releaselog/3_6_10.html">version 3.6.10</a> (all statistics in the report are against thatrelease of SQLite),the SQLite library consists of approximately61.9 KSLOC of C code.(KSLOC means thousands of "Source Lines Of Code" or, in other words,lines of code excluding blank lines and comments.)By comparison, the project has381 times as muchtest code and test scripts - 23619.9 KSLOC.</p><h2>2.0 Test Harnesses</h2><p>There are three independent test harnesses used for testing the core SQLite library.Each test harness is designed, maintained, and managed separatelyfrom the others.</p><ol><li><p>The <b>TCL Tests</b> are the oldest and most complete set of testsfor SQLite. The TCL tests are contained in the same source tree as theSQLite core and like the SQLite core are in the public domain. TheTCL tests are the primary tests used during development.The TCL tests are written using the <a href="http://www.tcl.tk/">TCL scripting language</a>.The TCL test harness itself consists of 15.5 KSLOC of C code use to create the TCL interface. The test scripts are containedin 449 files totaling 7.5MB in size. There are23602 distinct test cases, but many of the testcases are parameterized and run multiple times (with different parameters)so that on a full test run, about 1.1 million separate tests are performed.</p></li><li><p>The <b>TH3</b> test harness is a set of proprietary tests, written inC. The impetus for TH3 was the need to have a set of tests that ranon embedded and specialized platforms that would not easily supportTCL or other workstation services. TH3 tests use only the published SQLite interfaces. These tests are free to <a href="consortium.html">SQLite Consortium</a> members and are available by license to others. TH3 consists of about9.6 MB or 141.8 KSLOCof C code implementing 2137 distinct test cases.TH3 tests are heavily parameterized, though, so a full test runsabout 8.0 million different testinstances.</p></li><li><p>The <b>SQL Logic Test</b> or SLT test harness is used to run huge numbersof SQL statements against both SQLite and several other SQL database enginesand verify that they all get the same answers. SLT currently comparesSQLite against PostgreSQL, MySQL, and Microsoft SQL Server.SLT runs 5.3 million queries comprising1.02GB of test data.</p></li></ol><p>All of the tests above must run successfully, on multiple platformsand under multiple compile-time configurations,before each release of SQLite.</p><p>Prior to each check-in to the SQLite source tree, developerstypically run a subset (called "veryquick") of the Tcl testsconsisting of about 39.6 thousand test casesand covering95.85% of the core SQLite source code.The veryquick tests covers everything except the anomaly, fuzz, and soak tests. The idea behind the veryquick tests are that they aresufficient to catch most errors, but also run in only a few minutesinstead of a few hours.</p><h2>3.0 Anomaly Testing</h2><p>Anomaly tests are tests designed to verify the correct behaviorof SQLite when something goes wrong. It is (relatively) easy to buildan SQL database engine that behaves correctly on well-formed inputson a fully functional computer. It is more difficult to build a systemthat responses sanely to invalid inputs and continues to function followingsystem malfunctions. The anomaly tests are designed to verify the latterbehavior.</p><h3>3.1 Out-Of-Memory Testing</h3><p>SQLite, like all SQL database engines, makes extensive use ofmalloc() (See the separate report on<a href="malloc.html">dynamic memory allocation in SQLite</a> foradditional detail.)On workstations, malloc() never fails in practice and so correcthandling of out-of-memory (OOM) errors is not particularly important.But on embedded devices, OOM errors are frightenly common and sinceSQLite is frequently used on embedded devices, it is important thatSQLite be able to gracefully handle OOM errors.</p><p>OOM testing is accomplished by simulating OOM errors.SQLite allows an application to substitute an alternative malloc()implementation using the <a href="c3ref/config.html">sqlite3_config</a>(<a href="c3ref/c_config_getmalloc.html">SQLITE_CONFIG_MALLOC</a>,...)interface. The TCL and TH3 test harnesses are both capable ofinserting a modified version of malloc() that can be rigged fail after a certain number of allocations. These instrumented mallocscan be set to fail only once and then start working again, or tocontinue failing after the first failure. OOM tests are done in aloop. On the first iteration of the loop, the instrumented mallocis rigged to fail on the first allocation. Then some SQLite operationis carried out and checks or done to make sure SQLite handled theOOM error correctly. Then the time-to-failure counteron the instrumented malloc is increased by one and the test isrepeated. The loop continues until the entire operation runs tocompletion without ever encountering a simulated OOM failure.Tests like this are run twice, once with the instrumented mallocset to fail only once, and again with the instrumented malloc setto fail continuously after the first failure.</p><h3>3.2 I/O Error Testing</h3><p>I/O error testing seeks to verify that SQLite responds sanelyto failed I/O operations. I/O errors might result from a full disk drive,malfunctioning disk hardware, network outages when using a networkfile system, system configuration or permission changes that occur in the middle of an SQL operation, or other hardware or operating system malfunctions. Whatever the cause, it is important that SQLite be ableto respond correctly to these errors and I/O error testing seeks toverify that it does.</p><p>I/O error testing is similar in concept to OOM testing; I/O errorsare simulated and checks are made to verify that SQLite respondscorrectly to the simulated errors. I/O errors are simulated in boththe TCL and TH3 test harnesses by inserting a new<a href="c3ref/vfs.html">Virtual File System object</a> that is specially riggedto simulate an I/O error after a set number of I/O operations.As with OOM error testing, the I/O error simulators can be set tofail just once, or to fail continuously after the first failure.Tests are run in a loop, slowly increasing the point of failure untilthe test case runs to completion without error. The loop is run twice,once with the I/O error simulator set to simulate only a single failureand a second time with it set to fail all I/O operations after the firstfailure.</p><p>In I/O error tests, after the I/O error simulation failure mechanismis disabled, the database is examined using<a href="pragma.html#pragma_integrity_check">PRAGMA integrity_check</a> to make sure that the I/O error has notintroduced database corruption.</p><h3>3.2 Crash Testing</h3><p>Crash testing seeks to demonstrate that an SQLite database will notgo corrupt if the application or operating system crashes or if thereis a power failure in the middle of a database update. A separatereported called<a href="atomiccommit.html">Atomic Commit in SQLite</a> describes thedefensive measure SQLite takes to prevent database corruption followinga crash. Crash tests strive to verify that those defensive measuresare working correctly.</p><p>It is impractical to do crash testing using real power failures, ofcourse, and so crash testing is done in simulation. An alternative<a href="c3ref/vfs.html">Virtual File System</a> is inserted that allows the testharness to simulate the state of the database file following a crash.</p><p>In the TCL test harness, the crash simulation is done in a separateprocess. The main testing process spawns a child process which runssome SQLite operation and randomly crashes somewhere in the middle ofa write operation. A special VFS randomly reorders and corruptsthe unsynchronizedwrite operations to simulate the effect of buffered filesystems. Afterthe child dies, the original test process opens and reads the testdatabase and verifies that the changes attempted by the child eithercompleted successfully or else were completely rolled back. The<a href="pragma.html#pragma_integrity_check">integrity_check</a> <a href="pragma.html#syntax">PRAGMA</a> is used to make sure no database corruptionoccurs.</p><p>The TH3 test harness needs to run on embedded systems that do notnecessarily have the ability to spawn child processes, so it usesan in-memory VFS to simulate crashes. The in-memory VFS can be riggedto make snapshot of the entire filesystem after a set number of I/Ooperations. Crash tests run in a loop. On each iteration of the loop,the point at which a snapshot is made is advanced until the SQLiteoperations being tested run to completion without ever hitting asnapshot. Within the loop, after the SQLite operation under test hascompleted, the filesystem is reverted to the snapshot and random filedamage is introduced that is characteristic of the kinds of damageone expects to see following a power loss. Then the database is openedand checks are made to ensure that it is well-formed and that thetransaction either ran to completion or was completely rolled back.The interior of the loop is repeated multiple times for eachsnapshot with different random damage each time.</p><h2>4.0 Fuzz Testing</h2><p><a href="http://en.wikipedia.org/wiki/Fuzz_testing">Fuzz testing</a>seeks to establish that SQLite response correctly to invalid, out-of-range,or malformed inputs.</p><h3>4.1 SQL Fuzz</h3><p>SQL fuzz testing consists of creating syntactically correct yetwildly nonsensical SQL statements and feeding them to SQLite to seewhat it will do with them. Usually some kind of error is returned(such as "no such table"). Sometimes, purely by chance, the SQLstatement also happens to be semantically correct. In that case, theresulting prepared statement is run to make sure it gives a reasonableresult.</p><p>The SQL fuzz generator tests are part of the TCL test suite.During a full test run, about 35.0 thousand fuzz SQL statements aregenerated and tested.</p><h3>4.2 Malformed Database Files</h3><p>There are numerous test cases that verify that SQLite is able todeal with malformed database files.These tests first build a well-formed database file, then addcorruption by changing one or more bytes in the file by some meansother than SQLite. Then SQLite is used to read the database.In some cases, the bytes changes are in the middle of data files.This causes the content to change, but does not otherwise impactthe operation of SQLite. In other cases, unused bytes of the fileare modified. The interesting cases are when bytes of the file thatdefine database structure get changed. The malformed database testsverify that SQLite finds the file format errors and reports themusing the SQLITE_CORRUPT return code without overflowing⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -