Data-driven test cases

Datasets

Dataset interface
Dataset creation and delayed creation

Declaring and registering test cases with datasets

Operations on dataset

Joins
Zips
Grid (Cartesian products)

Datasets generators

Some tests are required to be repeated for a series of different input parameters. One way to achieve this is manually register a test case for each parameter. You can also invoke a test function with all parameters manually from within your test case, like this:

void single_test( int i )
{
  BOOST_TEST( /* test assertion */ );
}

void combined_test()
{
  int params[] = { 1, 2, 3, 4, 5 };
  std::for_each( params, params+5, &single_test );
}

The approach above has several drawbacks:

the logic for running the tests is inside a test itself: single_test in the above example is run from the test case combined_test while its execution would be better handled by the Unit Test Framework
in case of fatal failure for one of the values in param array above (say a failure in BOOST_TEST_REQUIRE), the test combined_test is aborted and the next test-case in the test tree is executed.
in case of failure, the reporting is not accurate enough: the test should certainly be reran during debugging sessions by a human or additional logic for reporting should be implemented in the test itself.

Parameter generation, scalability and composition

In some circumstance, one would like to run a parametrized test over an arbitrary large set of values. Enumerating the parameters by hand is not a solution that scales well, especially when these parameters can be described in another function that generates these values. However, this solution has also limitations

Generating functions: suppose we have a function func(float f), where f is any number in [0, 1]. We are not interested that much in the exact value, but we would like to test func. What about, instead of writing the f for which func will be tested against, we choose randomly f in [0, 1]? And also what about instead of having only one value for f, we run the test on arbitrarily many numbers? We easily understand from this small example that tests requiring parameters are more powerful when, instead of writing down constant values in the test, a generating function is provided.
Scalability: suppose we have a test case on func1, on which we test N values written as constant in the test file. What does the test ensure? We have the guaranty that func1 is working on these N values. Yet in this setting N is necessarily finite and usually small. How would we extend or scale N easily? One solution is to be able to generate new values, and to be able to define a test on the class of possible inputs for func1 on which the function should have a defined behavior. To some extent, N constant written down in the test are just an excerpt of the possible inputs of func1, and working on the class of inputs gives more flexibility and power to the test.
Composition: suppose we already have test cases for two functions func1 and func2, taking as argument the types T1 and T2 respectively. Now we would like to test a new functions func3 that takes as argument a type T3 containing T1 and T2, and calling func1 and func2 through a known algorithm. An example of such a setting would be
```
// Returns the log of x
// Precondition: x strictly positive.
double fast_log(double x);

// Returns 1/(x-1)
// Precondition: x != 1
double fast_inv(double x);

struct dummy {
  unsigned int field1;
  unsigned int field2;
};

double func3(dummy value)
{
  return 0.5 * (exp(fast_log(value.field1))/value.field1 + value.field2/fast_inv(value.field2));
}
```
In this example,
- func3 inherits from the preconditions of fast_log and fast_inv: it is defined in (0, +infinity) and in [-C, +C] - {1} for field1 and field2 respectively (C being a constant arbitrarily big).
- as defined above, func3 should be close to 1 everywhere on its definition domain.
- we would like to reuse the properties of fast_log and fast_inv in the compound function func3 and assert that func3 is well defined over an arbitrary large definition domain.
Having parametrized tests on func3 hardly tells us about the possible numerical properties or instabilities close to the point {field1 = 0, field2 = 1}. Indeed, the parametrized test may test for some points around (0,1), but will fail to provide an asymptotic behavior of the function close to this point.

Data driven tests in the Boost.Test framework

The facilities provided by the Unit Test Framework addressed the issues described above:

the notion of datasets eases the description of the class of inputs for test cases. The datasets also implement several operations that enable their combinations to create new, more complex datasets,
two macros, BOOST_DATA_TEST_CASE and BOOST_DATA_TEST_CASE_F, respectively without and with fixture support, are used for the declaration and registration of a test case over a collection of values (samples),
each test case, associated to a unique value, is executed independently from others. These tests are guarded in the same way regular test cases are, which makes the execution of the tests over each sample of a dataset isolated, robust, repeatable and ease the debugging,
several datasets generating functions are provided by the Unit Test Framework

The remainder of this section covers the notions and feature provided by the Unit Test Framework about the data-driven test cases, in particular:

the notion of dataset and sample is introduced
the declaration and registration of the data-driven test cases are explained,
the operations on datasets are detailed
and finally the built-in dataset generators are introduced.

Boost C++ Libraries

Data-driven test cases

Why data-driven test cases?

Parameter generation, scalability and composition

Data driven tests in the Boost.Test framework