Page Menu
Home
c4science
Search
Configure Global Search
Log In
Files
F80653770
testsuite.html.wml
No One
Temporary
Actions
Download File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Subscribers
None
File Metadata
Details
File Info
Storage
Attached
Created
Sun, Sep 1, 22:05
Size
15 KB
Mime Type
text/x-c
Expires
Tue, Sep 3, 22:05 (2 d)
Engine
blob
Format
Raw Data
Handle
20411609
Attached To
R3600 invenio-infoscience
testsuite.html.wml
View Options
## $Id$
## This file is part of the CERN Document Server Software (CDSware).
## Copyright (C) 2002, 2003, 2004, 2005 CERN.
##
## The CDSware is free software; you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation; either version 2 of the
## License, or (at your option) any later version.
##
## The CDSware is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with CDSware; if not, write to the Free Software Foundation, Inc.,
## 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
#include "cdspage.wml" \
title="Test Suite Strategy" \
navbar_name="hacking-test-suite-strategy" \
navtrail_previous_links="<a class=navtrail href=<WEBURL>/hacking/>Hacking CDSware</a> " \
navbar_select="hacking-test-suite-strategy"
<p>Version <: print generate_pretty_revision_date_string('$Id$'); :>
<h2>Contents</h2>
<strong>1. <a href="#1">General considerations</a></strong><br>
<strong>2. <a href="#2">Running test suite</a></strong><br>
<strong>3. <a href="#3">Writing test code</a></strong><br>
3.1 <a href="#3.1">Testing core code logic (Unit Testing)</a><br>
3.2 <a href="#3.2">Testing overall functionality (Regression Testing)</a><br>
3.3 <a href="#3.3">Testing user interface pages</a><br>
3.4 <a href="#3.4">File organization</a><br>
<strong>7. <a href="#4">Conclusions</a></strong><br>
<strong>8. <a href="#5">Additional information</a></strong><br>
<a name="1"></a><h2>1. General considerations</h2>
<p>This documents presents guidelines for unit testing and regression
testing homogenisation throughout all CDSware modules.
<p>Testing is an important coding activity. Most authors believe that
writing test cases should take between 10% and 30% of the project
time. But, even with such a large fraction, don't put too much belief
on such a testing. It cannot find bugs that aren't tested for. So,
while testing is an important activity inherent to safe software
development practices, it cannot become a substitute for pro-active
bug hunting, source code inspection, and bugfree-driven development
approach from the start.
<p>Testing should happen alongside with coding. If you write a
function, immediately load it into your toplevel, evaluate its
definition, and call it for a couple of arguments to make sure the
function works as expected. If not, then change the function
definition, re-evaluate it, re-call it, etc. Dynamic languages with
interactive toplevel such as Common Lisp or Python makes this easy for
you. Dynamic redefinition capabilities (full in Common Lisp, partial
in Python) are very programmer-friendly in this respect. If your test
cases are interesting to be kept, then keep them in a test file.
(It's almost all the time a good idea to store them in the test file,
since you cannot predict whether you won't want to change something in
the future.) We'll see below how to store your tests in a test file.
<p>When testing, it is nice to know some rules of thumb, like: check
your edge cases (e.g. null array), check atypical input values
(e.g. laaarge array instead of typically 5-6 elements only), check
your termination conditions, ask whether your arguments have already
been safe-proofed or whether it is in your mandate to check them,
write a test case for each `if-else' branch of the code to explore all
the possibilites, etc. Another interesting rule of thumb is the bug
frequency distribution. Experience has shown that the bugs tend to
cluster. If you discover a bug, there are chances that other bugs are
in the neighborhood. The famous 80/20 rule of thumb applies here too:
about 80% of bugs are located in about 20% of the code. Another rule
of thumb: if you find a bug caused by some coding practice pattern
thay may be used elsewhere too, look and fix other pattern instances.
<p>In a nutshell, the best advice to write bug-free code is: <em>think
ahead</em>. Try to prepare in advance for unusual usage scenarios, to
foresee problems before they happen. Don't rely on typical input and
typical usage scenarios. Things have a tendency to become atypical
one day. Recall that testing is necessary, but not sufficient, to
write good code. Therefore, think ahead!
<a name="2"></a><h2>2. Running test suite</h2>
<p>The CDSware test suite can be run in the source directory:
<blockquote>
<pre>
$ make test
</pre>
</blockquote>
or anytime after the installation:
<blockquote>
<pre>
$ /usr/local/cdsware-DEMO/bin/testsuite
</pre>
</blockquote>
The ``testsuite'' executable will run all available unit tests,
regression tests, and all the other kind of tests provided with
CDSware.
<p>The informative output is of the form:
<blockquote>
<pre>
$ make test
CDSware v0.3.2.20040519 test suite results:
===========================================
search engine washing of query patterns ... ok
search engine washing of URL arguments ... ok
search engine stripping of accented letters ... ok
bibindex engine list union ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.121s
OK
</pre>
</blockquote>
In case of problems you will see failures like:
<blockquote>
<pre>
CDSware v0.3.2.20040519 test suite results:
===========================================
search engine washing of query patterns ... FAIL
search engine washing of URL arguments ... ok
search engine stripping of accented letters ... ok
bibindex engine list union ... ok
======================================================================
FAIL: search engine washing of query patterns
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/cdsware-DEMO/lib/python/cdsware/search_engine_tests.py", line 25, in test_wash_pattern
self.assertEqual("ell*", search_engine.wash_pattern('ell*'))
File "/usr/lib/python2.3/unittest.py", line 302, in failUnlessEqual
raise self.failureException, \
AssertionError: 'ell*' != 'ell'
----------------------------------------------------------------------
Ran 4 tests in 0.091s
FAILED (failures=1)
</pre>
</blockquote>
The test suite compliance should be checked before each CVS commit.
(And, obviously, double-checked before each CDSware release.)
<a name="3"></a><h2>3. Writing test code</h2>
In this section, we shall firstly briefly touch ideas behind various
kinds of test code. Then we'll discuss more earthly matters, like how
to write the test code, where to put it, etc.
<a name="3.1"></a><h3>3.1 Testing core code logic (Unit Testing)</h3>
<p>Core functionality, such as the hit set intersection for the search
engine, or the text input manipulating functions of the BibConvert
language, should come with a couple of test cases to assure proper
behaviour of the core functionality. The test cases should cover
typical input (e.g. hit set corresponding to the query for ``ellis''),
as well as the edge cases (e.g. empty/full hit set) and other unusual
situations (e.g. non-UTF-8 accented input for BibConvert functions to
test a situation of different number of bytes per char).
<p>The test cases should be written for most important core
functionality. Not every function or class in the code is to be
throughly tested. Common sense will tell.
<p>For more information on Pythonic unit testing, see the
documentation to the unittest module at <a
href="http://docs.python.org/lib/module-unittest.html">http://docs.python.org/lib/module-unittest.html</a>.
For a tutorial, see for example <a
href="http://diveintopython.org/unit_testing/">http://diveintopython.org/unit_testing/</a>.
<a name="3.2"></a><h3>3.2 Testing overall functionality (Regression Testing)</h3>
<p>In addition to the above-mentioned unit testing of important
functions, a regression testing should ensure that the existing
program's overall functionality is not altered by new changes. This
is especially important if a bug had been previously found. Then a
regression test case should be written to assure that it will never
reapper. (It also helps to scan the neighborhood of the bug, see the
80/20 thumb rule cited above.)
<p>As an example of a regression test, we have several BibConvert
example configurations (*.cfg) in CDSware already. For each
configuration, we could write more or less complex input examples
(*.dat) -- some of them are already in the distribution -- that would
trigger a use of many or all BibConvert functionality. In addition we
would provide in the distribution the expected results file (*.res or
*.out or whatever) for that particular config file and that particular
input file. The test suite would then simply run a shell process to
run bibconvert with the sample configuration on the sample data and
would compare the output to the known one.
<p>For more information on regression testing, see for example <a
href="http://c2.com/cgi/wiki?RegressionTesting">http://c2.com/cgi/wiki?RegressionTesting</a>.
<a name="3.3"></a><h3>3.3 Testing user interface pages</h3>
<p>Web interfaces are an important part of our application, so that we
should possibly test the most important user interface pages too.
<p>It may be hard to test GUI stuff, though there are ways, see for
example <a
href="http://www.xp123.com/xplor/xp0001/index.shtml">http://www.xp123.com/xplor/xp0001/index.shtml</a>.
In our case mod_python is very unfriendly in this direction, it's
impossible to run code from within Emacs or command line. But, if the
GUI code is structured in the usual CDSware way, then it may be done
via string comparisons. That is, for functionality like printing of a
basket we typically have a web-seen function
<code>print_basket()</code> that looks, schematically:
<blockquote>
<pre>
$ cat /path/to/apache/htdocs/yourbaskets.py
[...]
from cdsware.webbasketlib import perform_print_basket
def print_basket(req, basket_name)
uid = getUid(req)
return perform_print_basket(uid, basket_name)
[...]
</pre>
</blockquote>
with all the logic being done inside the library
<code>perform_print_basket()</code>:
<blockquote>
<pre>
$ cat /path/to/cdsware/lib/python/cdsware/webbasketlib.py
[...]
def perform_print_basket(uid, basket_name):
out = ""
## function body goes here
return out
[...]
</pre>
</blockquote>
The latter function can run outside of Apache, it accepts `normal'
input variables and returns `normal' text. The function can therefore
be easily tested via unit testing technique described above.
<p>Usually, when testing GUIs, we don't want to go as far as to test
sessions, logins, redirections and stuff. In case it is interesting
to automate these tests too, we could look at, for example, WebUnit
that could be used for these purposes. <a
href="http://webunit.sourceforge.net/">http://webunit.sourceforge.net/</a>.
<a name="3.4"></a><h3>3.4 File organization</h3>
<p>Each core file that is located in the lib directory (such as the
<code>webbasketlib.py</code> in the example above) should come with a
testing file where the test cases are stored. The test file is to be
named identically as the lib file it tests, but with the suffix
<code>_tests</code> (in our example,
<code>webbasketlib_tests.py</code>).
<p>The test cases are written using Pythonic unittest TestCase class.
An example for testing search engine query parameter washing function:
<blockquote>
<pre>
$ cat /path/to/cdsware/lib/python/cdsware/search_engine_tests.py
[...]
import search_engine
import unittest
class TestWashQueryParameters(unittest.TestCase):
"""Test for washing of search query parameters."""
def test_wash_url_argument(self):
"""search engine washing of URL arguments"""
self.assertEqual(1, search_engine.wash_url_argument(['1'],'int'))
self.assertEqual("1", search_engine.wash_url_argument(['1'],'str'))
self.assertEqual(['1'], search_engine.wash_url_argument(['1'],'list'))
self.assertEqual(0, search_engine.wash_url_argument('ellis','int'))
self.assertEqual("ellis", search_engine.wash_url_argument('ellis','str'))
self.assertEqual(["ellis"], search_engine.wash_url_argument('ellis','list'))
self.assertEqual(0, search_engine.wash_url_argument(['ellis'],'int'))
self.assertEqual("ellis", search_engine.wash_url_argument(['ellis'],'str'))
self.assertEqual(["ellis"], search_engine.wash_url_argument(['ellis'],'list'))
[...]
</pre>
</blockquote>
<p>In addition, each test file is supposed to define a
<code>create_test_suite()</code> function that will return test suite
with all the tests available in this file:
<blockquote>
<pre>
$ cat /path/to/cdsware/lib/python/cdsware/search_engine_tests.py
[...]
def create_test_suite():
"""Return test suite for the search engine."""
return unittest.TestSuite((unittest.makeSuite(TestWashQueryParameters,'test'),
unittest.makeSuite(TestStripAccents,'test')))
[...]
</pre>
</blockquote>
<p>This will enable us to later include this file into
<code>testsuite</code> executable:
<blockquote>
<pre>
$ cat /path/to/cdsware/source/modules/miscutil/bin/testsuite.wml
[...]
from cdsware import search_engine_tests
from cdsware import bibindex_engine_tests
def create_all_test_suites():
"""Return all tests suites for all CDSware modules."""
return unittest.TestSuite((search_engine_tests.create_test_suite(),
bibindex_engine_tests.create_test_suite()))
[...]
</pre>
</blockquote>
<p>In this way, all the test cases defined in the file
<code>search_engine_tests.py</code> will be executed when the global
<code>testcase</code> executable is called.
<p>Note that it may be time-consuming to run all the tests in one go.
If you are interested in running tests only on a certain file (say
<code>search_engine_tests.py</code>), then launch:
<blockquote>
<pre>
$ python /path/to/cdsware/lib/python/cdsware/search_engine_tests.py
</pre>
</blockquote>
<p>For feature-full examples, consult the following files in the
source distribution:
<pre>
./modules/websearch/lib/search_engine.py.wml
./modules/websearch/lib/search_engine_tests.py.wml
./modules/bibindex/lib/bibindex_engine.py.wml
./modules/bibindex/lib/bibindex_engine_tests.py.wml
./modules/miscutil/bin/testsuite.wml
</pre>
<a name="4"></a><h2>4. Conclusions</h2>
A uniform testing technique and a test suite for CDSware is
introduced. Python unittest module is used for both unit testing and
regression testing, with proper calling of external scripts. The
functional core code is the most obvious candidate to start the
testing process with, but we should also remeber to test the most
important web page elements too. Each programmer should plan to write
the test code alongside the core code development. The guidelines
were given as to how to do it in a uniform way.
<a name="5"></a><h2>5. Additional information</h2>
<dl>
<dt>More information can be found on the URLs mentioned above:
<dd>
<pre>
<a href="http://c2.com/cgi/wiki?UnitTest">http://c2.com/cgi/wiki?UnitTest</a>
<a href="http://c2.com/cgi/wiki?RegressionTesting">http://c2.com/cgi/wiki?RegressionTesting</a>
<a href="http://docs.python.org/lib/module-unittest.html">http://docs.python.org/lib/module-unittest.html</a>
<a href="http://diveintopython.org/unit_testing/">http://diveintopython.org/unit_testing/</a>
<a href="http://www.xp123.com/xplor/xp0001/index.shtml">http://www.xp123.com/xplor/xp0001/index.shtml</a>
<a href="http://webunit.sourceforge.net/">http://webunit.sourceforge.net/</a>
</pre>
</dl>
<dl>
<dt>and elsewhere:
<dd>
<pre>
Steve McConnell: "Code Complete"
FIXME: list of other interesting references, like Kent Beck papers, etc
</pre>
</dl>
Event Timeline
Log In to Comment