group__clienttest.tex

This SDK allows to integrate a syncml stack in a C++ application on a variety of platforms. Current

\section{Client Testing Framework}
\label{group__ClientTest}\index{Client Testing Framework@{Client Testing Framework}}
The testing framework of the C++ client library is based on CPPUnit.  
\subsection*{Classes}
\begin{CompactItemize}
\item 
class \textbf{ClientOutputter}
\item 
class \textbf{ClientListener}
\item 
class {\bf TestFileSource}
\begin{CompactList}\small\item\em This code uses the \doxyref{ClientTest}{p.}{classClientTest} and \doxyref{RawFileSyncSource}{p.}{classRawFileSyncSource} to test real synchronization against a server. \item\end{CompactList}\item 
class \textbf{RegisterTest}
\begin{CompactList}\small\item\em the only purpose of this class is to own the first \doxyref{TestFileSource}{p.}{classTestFileSource} and to register its tests at program startup \item\end{CompactList}\item 
class {\bf CreateSource}
\begin{CompactList}\small\item\em helper class to encapsulate \doxyref{ClientTest::Config::createsource\_\-t}{p.}{structClientTest_1_1Config_f7d2815b26c8007d45eac2f5cfb3a232} pointer and the corresponding parameters \item\end{CompactList}\item 
class {\bf LocalTests}
\begin{CompactList}\small\item\em local test of one sync source and utility functions also used by sync tests \item\end{CompactList}\item 
class {\bf SyncTests}
\begin{CompactList}\small\item\em Tests synchronization with one or more sync sources enabled. \item\end{CompactList}\item 
class {\bf ClientTestFactory}
\begin{CompactList}\small\item\em generates tests on demand based on what the client supports \item\end{CompactList}\end{CompactItemize}
\subsection*{Defines}
\begin{CompactItemize}
\item 
\#define {\bf SOURCE\_\-ASSERT\_\-NO\_\-FAILURE}(\_\-source, \_\-x)
\begin{CompactList}\small\item\em execute \_\-x and then check the status of the \_\-source pointer \item\end{CompactList}\item 
\#define {\bf SOURCE\_\-ASSERT}(\_\-source, \_\-x)
\begin{CompactList}\small\item\em check \_\-x for true and then the status of the \_\-source pointer \item\end{CompactList}\item 
\#define {\bf SOURCE\_\-ASSERT\_\-EQUAL}(\_\-source, \_\-value, \_\-x)
\begin{CompactList}\small\item\em check that \_\-x evaluates to a specific value and then the status of the \_\-source pointer \item\end{CompactList}\item 
\#define {\bf SOURCE\_\-ASSERT\_\-MESSAGE}(\_\-message, \_\-source, \_\-x)
\begin{CompactList}\small\item\em same as \doxyref{SOURCE\_\-ASSERT()}{p.}{group__ClientTest_g59fc0b73d4abeacf0cc61dc6668ef382} with a specific failure message \item\end{CompactList}\item 
\#define {\bf ADD\_\-TEST}(\_\-class, \_\-function)~addTest(new CppUnit::TestCaller$<$\_\-class$>$(getName() + \char`\"{}::\char`\"{} \#\_\-function, \&\_\-class::\_\-function, $\ast$this))
\begin{CompactList}\small\item\em convenience macro for adding a test name like a function, to be used inside an instance of that class \item\end{CompactList}\end{CompactItemize}
\subsection*{Functions}
\begin{CompactItemize}
\item 
void \textbf{simplifyFilename} (string \&filename)\label{group__ClientTest_g0e303fd8fed6073ee862d6078a6369fd}

\item 
const string \& \textbf{getCurrentTest} ()\label{group__ClientTest_g7cfc20a326e05788893bfeb6fb252100}

\item 
int \textbf{main} (int argc, char $\ast$argv[$\,$])\label{group__ClientTest_g0ddf1224851353fc92bfbff6f499fa97}

\item 
virtual void {\bf ClientTest::registerTests} ()
\begin{CompactList}\small\item\em This is the only function provided by \doxyref{ClientTest}{p.}{classClientTest} itself: it registers tests using this instance of \doxyref{ClientTest}{p.}{classClientTest} for later use during a test run. \item\end{CompactList}\item 
\textbf{ClientTest::ClientTest} (int serverSleepSec=0, const std::string \&serverLog=\char`\"{}\char`\"{})\label{group__ClientTest_gf569060e4a730e4f48c44076a32a496a}

\item 
static int {\bf ClientTest::dump} ({\bf ClientTest} \&client, {\bf SyncSource} \&source, const char $\ast$file)\label{group__ClientTest_g21f82c79cc49a242a9ddb28ccc4ae4cf}

\begin{CompactList}\small\item\em utility function for dumping items which are C strings with blank lines as separator \item\end{CompactList}\item 
static int {\bf ClientTest::import} ({\bf ClientTest} \&client, {\bf SyncSource} \&source, const char $\ast$file)\label{group__ClientTest_g6f8c5e62da7b5da62d4bf5e5f9f8938e}

\begin{CompactList}\small\item\em utility function for importing items with blank lines as separator \item\end{CompactList}\item 
static bool {\bf ClientTest::compare} ({\bf ClientTest} \&client, const char $\ast$fileA, const char $\ast$fileB)\label{group__ClientTest_g489f26d3ca7ec7dec96b14bc81b89dc0}

\begin{CompactList}\small\item\em utility function for comparing vCard and iCal files with the external synccompare.pl Perl script \item\end{CompactList}\item 
virtual void {\bf ClientTest::postSync} (int res, const std::string \&logname)
\begin{CompactList}\small\item\em This is called after successful \doxyref{sync()}{p.}{classClientTest_5e5a95277a8c953307c6c4c9415fe627} calls (res == 0) as well as after unsuccessful ones (res != 1). \item\end{CompactList}\item 
static void {\bf ClientTest::getTestData} (const char $\ast$type, Config \&config)
\begin{CompactList}\small\item\em A derived class can use this call to get default test cases, but still has to add callbacks which create sources and execute a sync session. \item\end{CompactList}\item 
virtual void {\bf CheckSyncReport::check} (int res, {\bf SyncReport} \&report) const 
\begin{CompactList}\small\item\em checks that the sync completed as expected and throws CPPUnit exceptions if something is wrong \item\end{CompactList}\end{CompactItemize}
\subsection*{Variables}
\begin{CompactItemize}
\item 
ClientListener \textbf{syncListener}\label{group__ClientTest_gec894e9ea4a001e35dd206b4ff4a5273}

\end{CompactItemize}


\subsection{Detailed Description}
The testing framework of the C++ client library is based on CPPUnit. 

It fulfills several different goals:\begin{itemize}
\item testing of the library itself\begin{itemize}
\item unit testing of individual classes\item full end-to-end tests of synchronization in different configurations\end{itemize}
\item regular testing of a SyncML server\item testing of SyncML clients built on top of the library\end{itemize}


Some of the design goals were:\begin{itemize}
\item a simple, unified test runner on all platforms\item it should be possible to add tests either directly in the .cpp which implements a class or in a separate file; if done in the class .cpp file then the extra code should be protected by an ifdef so that it is possible to choose whether it is in the released library or not\item adding tests should not require changes in a central file\end{itemize}


The test runner \char`\"{}client-test\char`\"{} is implemented in client-test-main.cpp. It automatically assembles all CPPUnit tests it was linked against and runs them all if started without parameters. Output is ASCII. The return code indicates success or failure of any test. \char`\"{}-h$|$--help\char`\"{} print some usage information and a full list of all available tests. Tests or test groups as printed in that list can be given as command line parameters to run just these tests. Because tests do not communicate directly with the command line front-end, they must be passed parameters via environment variables. See the description of \doxyref{TestFileSource}{p.}{classTestFileSource} for further information about running those end-to-end synchronization tests.

The test runner itself also understands some environment variables: CLIENT\_\-TEST\_\-FAILURES can be set to a comma separated list of tests which are allowed to fail without affecting the return code of the test runner.

On Linux the test runner supports setting a timeout which is triggered if a test runs for more than the number of seconds set in the environment variable CLIENT\_\-TEST\_\-ALARM. This relies on signals and therefore is disabled in compilations on Windows with an ifdef HAVE\_\-SIGNALS\_\-H.

Because of the dependency on CPPUnit the testing needs to be enabled explicitly. This can be done separately for tests embedded in the object files of the library (unit tests) and the end-to-end synchronization (integration tests) which can be used with release versions of the library.

On POSIX systems the configure --enable-integration-test and --enable-unit-tests switches enable the testing. \char`\"{}make check\char`\"{} automatically executes the unit tests.

On Windows the Visual Studio project file build/win32/cppunit.sln was prepared to build the \char`\"{}client-test\char`\"{} executable. Because it uses the standard win32.vcproj, some changes are necessary to enable unit tests:\begin{itemize}
\item replace DISABLE\_\-UNIT\_\-TESTS with ENABLE\_\-UNIT\_\-TESTS in the preprocessor settings\item put cppunit-1.12.0 into the same directory as the 3x directory that contains the client-api/native directory (other layouts are of course possible, just change the include and library search paths)\end{itemize}


Unit tests are only found if they are pulled into the executable. On Linux this is done by searching for variables with a special name, created by the FUNAMBOL\_\-TEST\_\-SUITE\_\-REGISTRATION() macro in \doxyref{test.h}{p.}{test_8h-source}, and setting link flags on the fly. On Windows an additional C file with references to these variables is used instead. This is done by a Python script, thus python.exe must be installed and found in the search path of Visual Studio. Any suggestion how to avoid the dependency on external scripting tools without having to manually register tests in a central file is welcome...

The integration tests depend on auxiliary files (test/synccompare.pl and test/testcases) which must be found in the directory where the test is run. On Linux the makefile rules automatically copy these, but the Visual Studio project file does not. Note that test/synccompare.pl is written in Perl (and thus creates another external dependency) because a) C/C++ do not have a standard library to implement the complex regular pattern matching/replace done in that script and b) the script is also meant to be shipped with clients (SyncEvolution does that) and thus needs to run on as many systems as possible. Perl is still the most universally available scripting language for that purpose.

Adding new unit tests is best done by following the example in src/c++/common/base/util/BasicTime.cpp.

Regular testing of a server can be done by using the reference data provided together with \doxyref{ClientTest}{p.}{classClientTest} and synchronizing the server against simple files on the client side via a \doxyref{TestFileSource}{p.}{classTestFileSource}. This is what enabling the integration tests adds to the \char`\"{}client-test\char`\"{} binary. By keeping the client side fixed regressions in the server can be detected. Regular testing of the client library can be done by testing the current client library source against stable servers.

For testing a client based on the C++ library the client developer needs to implement the \doxyref{ClientTest}{p.}{classClientTest} interface. There are some additional requirements for the client's \doxyref{SyncSource}{p.}{classSyncSource} implementations:\begin{itemize}
\item they must support more than one local database so that the same client host running the \char`\"{}client-test\char`\"{} binary can act as two different clients\item they must support change tracking for more than just the server and accessing the databases via \doxyref{SyncSource}{p.}{classSyncSource} instances so that the testing frame work can synchronize, then add/remove/update items and/or list changes without affecting the state of the client with respect to its server\end{itemize}


The \doxyref{TestFileSource}{p.}{classTestFileSource} is an example how the \doxyref{ClientTest}{p.}{classClientTest} interface could be implemented. 

\subsection{Define Documentation}
\index{ClientTest@{ClientTest}!ADD_TEST@{ADD\_\-TEST}}