I tried to create a simple, orthogonal and thus easy to extend and adapt testing framework that stays easy to use. I avoided some complexity for users of CUTE by exploiting modern C++ library features in the Boost library that is part of the
Note that all classes presented below are in namespace cute, which is omitted for the sake of brevity.
The core class stores test functions using
std::function, any parameterless function or functor can be a test. In addition, each
cute::test has a name for easier identification. That name is given either during construction or derived from a functor’s
typeid. The GNU g++ compiler requires that you demangle the name given by the
type_info object, while VC++ provides a human readable
type_info::name() result directly.
As you can see, there is no need to inherit from class
For simple functions, or when you want to name your tests differently from the functor’s type, you can use the
CUTE is a function-like macro that takes the name of a test function and instantiates the test class with the address of that test function and its name.
Using a template constructor allows you to use any kind of functor that can be stored in a
std::function<void()>, but this means that the functor can take no parameters. To construct with functions, functors or member functions with parameters, use
std::bind() as shown below.
Running a single test with
cute::runner is not very interesting. You might as well just call that function directly and check the results. The power of unit testing is realized when you have a larger collection of test cases that run after every compile and on a build server after every check-in. Thus there is a need for running many tests at once.
In contrast to other unit testing frameworks (including JUnit) I refrained from applying the Composite design pattern [GoF] for implementing the test case container. I love Composite and it is handy in many situations for tree structures, but it comes at the price of strong coupling by inheritance and lower cohesion in the base class, because of the need to support the composite class’ interface. The simplest solution I came up with is to simply represent the test suite as a
std::vector<cute::test>. Instead of a hierarchy of suites, you just run a sequence of tests. When the tests run, the hierarchy plays no role. You still can arrange your many tests into separate suites, but before you run them, you either concatenate the vectors or you run the suites individually in your
main() function using the runner.
Tests can be added to the suite using
vector::push_back(), but to make it really easy to fill your suite with tests, CUTE also provides an overloaded
operator+= that will append a test object to a suite:
This idea is blatantly stolen from
So this is all it takes to build a test suite:
If you really want to organize your test as a sequence of test suites, CUTE provides a
suite_test functor that will take a test suite and run it through its call operator. However, if any test in a
suite_test fails, the remaining tests will not be run.
CUTE’s Eclipse plug-in eases the construction of test suites by providing automatic code generation and adjustment for registering test functions in suites. You can have standalone CUTE executables for a single suite, or test multiple suites, each in a separate library project.
A unit testing framework would not be complete without a way to actually check something in a convenient way. One principle of testing is to fail fast, so any failed test assertion will abort the current test and signal the failure to the top-level runner. You might have already guessed that this is done by throwing an exception. Later on, we will want to know where that test failed, so I introduced an exception class
test_failure that takes the source file name and line number in the source file. Java does this automatically for exceptions, but as C++ programmers we must obtain and store this information ourselves. We rely on the preprocessor to actually know where we are in the code. Another
std::string allows sending additional information from the test programmer to the debugger of a failing test.
This is how
cute_base.h looks without the necessary
#include guards and
For actually writing test assertions, I provided macros that will throw if a test fails:
This is all you need to get started. However, some convenience is popular in testing frameworks. Unfortunately, convenience often tends to be over-engineered and I am not yet sure if the convenience functionality I provided is yet simple enough. Therefore I ask for your feedback on how to make things simpler or confirmation that it is already simple enough.
Testing two values for equality is probably the most popular test. Therefore, all testing frameworks provide a means to test for equality. JUnit, for example, provides a complete set of overloaded equality tests. C++ templates can do that as well with less code. For more complex data types, such as strings, it can be difficult to see the difference between two values, when they are simply printed in the error message.
One means to implement
ASSERT_EQUAL would be to just
#define it to map to
ASSERT((expected)==(actual)). However, from my personal experience of C++ unit testing since 1998, this gives too little information when the comparison fails. This is especially true for strings or domain objects, where seeing the two unequal values is often essential for correcting the programming mistake. In my former life, we had custom error messages for a failed string comparison that allowed us to spot the difference easily. Therefore, CUTE provides a template implementation of
ASSERT_EQUAL. This is of course called by a macro to enable file position reporting.
I speculated (perhaps wrongly) that it would be useful to specify your own mechanism to create the message if two values differ, which is implemented as a to-be-overloaded interface in the namespace
to_string function is then called in
diff_values which composes the standard message for your failed test case…
…and which is called in case your
ASSERT throws a
As of version 1.5, CUTE allows all kinds of types to be compared by
ASSERT_EQUAL. While earlier versions allowed only types where
operator<<(ostream &,TYPE) was defined, some template meta-programming tricks now allow also other types, as long as
operator==(expected,actual) is defined and delivers a bool compatible result. For integer types, meta-programming ensures that no signed-unsigned comparison warning is issued anymore. Comparing two floating point values without specifying a delta, automatically selects a delta that masks the least significant decimal digit, based on the size of expected. Floating point comparison subtracts actual and expected and sees if the absolute value of the difference is less than delta, by using
Another good unit testing practice is to verify that things go wrong as intended.
To embed a piece of code (an expression, or anyhting that can be passed as a macro parameter) that should throw a specific exception type, you can use the macro…
…within your test function. For example:
This test will fail if
should_throw_std_exception() does not throw an exception of type
std::exception. Any other exception will lead to an error, in contrast to failure.
There is no need to implement the try-catch again by hand to test error conditions. What is missing is the ability to expect a runtime error recognized by the operating system such as an invalid memory access. Those are usually signaled instead of thrown as a nice C++ exception.
You might need parenthesis around the code in the macro parameter to disambiguate commas, particularly commas in a parameter list.
You have already seen that the runner class template can be specialized by providing a listener. The
runner class is an inverted application of the Template Method design pattern [GoF]. Instead of implementing the methods called dynamically in a subclass, you provide a template parameter that acts as a base class to the class @runner@, which holds the template methods
If you look back to
runner::runit(), you will recognize that if any reasonable exception is thrown, it would be hard to diagnose the reason for the error. Therefore, I included catch clauses for
std::exception, string and char pointers to get information required for diagnosis. The demangling is required for GNU g++ to get a human-readable information from the exception’s class name.
Again I ask you for feedback if doing this seems over-engineered. Are you throwing strings as error indicators?
As you can see, there are a bunch of methods delegated to the base class given as
runner’s template parameter
(begin, end, start, success, failure, error). The default template parameter
null_listener applies the Null Object design pattern and provides the concept all fitting Listener base classes.
Whenever you need to collect the test results or you want to have a nice GUI showing progress with the tests, you can create your own custom listener.
Again you can stack listeners using an inverted version of a Decorator design pattern [GoF]. Here is an example of an inverted Decorator using C++ templates that counts the number of tests by category:
From the above schema, you can derive your own stackable listener classes, such as a listener that displays in a GUI the progress and results of tests as they run. If you do so, please share your solution.
std::bind() at your disposal, it is easy to construct a functor object from a class and its member function. Again this is canned in a macro that can be used like this:
The first version uses object
testobject, an instance of
TestClass, as the target for the member function
test1. The second version creates a new instance of
TestClass to then call its member function
test2 when the test is executed. The last macro provides a means to pass an additional object to
TestClass’ constructor when it is incarnated. The idea of incarnating the test object and thus have its constructor and destructor run as part of the test comes from Kevlin Henney and is implemented in Paul Grenyer’s testing framework Aeryn.
CUTE_MEMFUN delegates its work to a template function as follows:
When the template function
makeMemberFunctionTest is called, it employs
std::bind to create a functor object that will call the member function fun on object
t. Again we can employ C++ reflection using
typeid to derive part of the test object’s name. We need to derive the member function name again using the preprocessor with a macro. In order to also allow const member functions, the template function comes in two overloads, one using a reference (as shown) and the other using a const reference for the testing object.
I will spare you the details, and just present the mechanism of object incarnation and then calling a member function for the case where you can supply a context object:
This allows you to use test classes with a constructor to set up a test fixture and a destructor for cleaning up after the test. This eliminates need to for explicit
tearDown() methods, as in JUnit.