The C++ logo, by Jeremy Kratz, licensed under CC0 1.0 Universal Public Domain Dedication

A C++ test suite in 100 lines of code

Notes published the 29 Sep, 2024
Notes updated the 18 Oct, 2024
18 - 22 minutes to read, 4381 words
Categories: c++
Keywords: c++, compile-time, meta-programming, reflection, test

Table of Contents

Why?
The simplest test suite
Test runner
- Improved error messages
- Disable at compile time
Missing improvements
Conclusion

Why?

Why would you want to write a C++ test suite?

Aren’t there already enough?

Google Test, Catch / Catch2, CppUnit, doctest, UT / μt and many others.

Like many projects, there are different reasons, like for fun, to learn something new, and to see how much can be achieved with a minimal amount of code.

If you think that this is just reinventing the wheel, that does not have to be a bad thing. Wheels have been reinvented and improved a lot of times.

One reason why existing test suites might not be good enough is that they lack a feature fundamental to your project. While many test suites are suites are open source, for technical reasons it might still not be possible, or easy, to add the desired features.

In particular, I wanted to write a test suite with the following features:

no need to define tests, and register the tests separately
it should be possible to remove the tests at compile time (with the preprocessor)
test function should report the source file, line, and expression that failed
should not trigger a compiler error if a type is not printable

Since the complete test suite is less than 100 lines of code, changing how it works and adding any desired feature should not be too difficult.

The simplest test suite

For the whole notes, I’m going to test a buggy add function.

#include <iostream>

int add(int a, int b) {return a + b + 1;}

void test(){
  if(add(1,2) != 3){
    std::cerr << "add(1,2) != 3 failed\n";
  }
}

int main(){
  test();
}

That’s right, there is nothing special, just "normal" C++ code.

Here I have used

use the if statement to verify the correctness of the tested function
use std::cerr for logging errors
call the test function manually

For terminating tests, one would throw exceptions, call exit, or use assert instead of if-statements

Replace `if` statement with a check statement

The if statement followed by logging is going to be a very repetitive pattern. Plus, it does not offer any chance to automatically generate a decent error message.

Thus this is the first thing that should be part of a test suite: a way to verify if something meets the expectations of the programmer.

A `check` function

The first approach would be to replace the if statement with a function:

#include <iostream>
#include <string_view>
#include <source_location>

void check(bool condition, std::string_view additional_message, std::source_location sl = std::source_location::current()){
  if(not(condition)){
    std::cerr << "condition failed " << additional_message << " " << sl.file_name() << ":" << sl.line();
  }
}

int add(int a, int b) {return a + b + 1;}

void test(){
  check(add(1, 2) == 3, "add(1, 2) == 3");
}

int main(){
  test();
}

With check, all the boilerplate from void test has been removed, and we have added information about the location automatically.

The major drawback is that we need to generate manually a string containing the expression that failed.

Every test suite worth its name would try to generate such a message automatically.

A `CHECK` macro

Yes, a macro.

Although I do not like them, in this situation, the advantages are too many to be ignored.

#include <iostream>
#include <string_view>
#include <source_location>

void check_internal(bool condition, std::string_view expression_literal, std::source_location sl, std::string_view additional_info = ""){
  if(not(condition)){
    std::cerr << "  " << sl.file_name() << ":" << sl.line() << "   " << expression_literal << " " << additional_info << "\n";
  }
}

#define CHECK(e, ...) \
  check_internal(e, #e, std::source_location::current() __VA_OPT__(,) __VA_ARGS__ )

int add(int a, int b) {return a + b + 1;}
void test(){
  CHECK(add(1, 2) == 3);
}

int main(){
  test();
}

With this approach, we are on par with most test suites.

If the test fails, the failing expression, line number, and file name are logged automatically.

A more advanced `CHECK` macro

Since we are already using the preprocessor, an even better approach would be to decompose an expression.

The resulting code looks more or less like the following:

#include <string>
#include <source_location>
#include <iostream>

struct info_t {
  std::string_view expr = "";
  std::source_location sl = std::source_location::current();
  bool result = false;
  std::string opt_lhs = "";
  std::string opt_rhs = "";
};

template <class A>
struct check_t {
 A a;
  info_t& i;
#define CHECK_T_COMP_OP(OP) \
  template <class B> \
  info_t operator OP(B&& b) && {\
    i.opt_lhs = std::to_string(a);\
    i.opt_rhs = std::to_string(b);\
    i.result = (a OP b);\
    return std::move(i);\
  } static_assert(true)
  CHECK_T_COMP_OP(==);
  CHECK_T_COMP_OP(!=);
  CHECK_T_COMP_OP(> );
  CHECK_T_COMP_OP(>=);
  CHECK_T_COMP_OP(< );
  CHECK_T_COMP_OP(<=);
#undef CHECK_T_COMP_OP
  operator info_t() && {
    i.opt_lhs = std::to_string(a);
    i.result = a;
    return std::move(i);
 }
};
struct destruct_t {
  info_t i;
  template <class A>
  check_t<A> operator<(A&& a) && {
    return check_t<A>{std::forward<A>(a), i};
  }
};
#define DECOMPOSE_CMP_OP(expr, lit) destruct_t{info_t{lit ""}} < expr

void check_internal(info_t i, std::string_view additional_info = ""){
  if(not i.result){
    std::cerr << "  " << i.sl.file_name() << ":" << i.sl.line() << "   " << i.expr << " with values:  " << i.opt_lhs << ", and " << i.opt_rhs << additional_info << "\n";
  }
}

#define CHECK(e, ...) check_internal(DECOMPOSE_CMP_OP(e, #e) __VA_OPT__(,) __VA_ARGS__)

// example usage
int add(int a, int b) {return a + b + 1;}
void test(){
  int i = 1;
  CHECK(add(i, 2) == 3);
}

int main() {
  test();
}

Note the addition of operator info_t, without it, CHECK(true) would fail to compile, as there is no comparison, and thus check_t would not convert to info_t. An alternate approach would be to overload check_internal, and maybe it would be better to prevent possible unintended conversions, even if the types should not be used outside of CHECK.

Another important difference is the macro DECOMPOSE_CMP_OP, which takes two values instead of one. The main reason is that DECOMPOSE_CMP_OP is used inside CHECK, and if the argument of check would be a macro, then it would be expanded, and the literal would not be equal to what is written in the source code.

To give a more concrete example

#define DECOMPOSE_CMP_OP(expr, lit) destruct_t{info_t{lit ""}} < expr
#define CHECK(e, ...) check_internal(DECOMPOSE_CMP_OP(e, #e) __VA_OPT__(,) __VA_ARGS__)

int main(){
  int i = 1;
  CHECK((CHECK(true),false));
}

This macro would print something like

/app/example.cpp:57   (CHECK(true),false) with values:  0, and

The currently proposed version:

#define DECOMPOSE_CMP_OP(expr) destruct_t{info_t{#expr}} < expr
#define CHECK(e, ...) check_internal(DECOMPOSE_CMP_OP(e) __VA_OPT__(,) __VA_ARGS__)

int main(){
  int i = 1;
  CHECK((CHECK(true),false));
}

Prints something like

/app/example.cpp:57   (check_internal(destruct_t{info_t{"true"}} < true ),false) with values:  0, and

With approximately 55 lines of code (example excluded), we have a more advanced CHECK macro, even more advanced compared to some test suites. Similarly one should implement an ASSERT macro that stops the current test, but for brevity, I’m leaving it out.

Note 📝

Why did I write info_t{lit ""} and not simply info_t{lit}? If lit is not literal, the first construct will trigger a compile error. In the case of a literal, it is thus completely optional but ensures that the second argument of DECOMPOSE_CMP_OP is a literal.

Improve support for printable and non-printable types

The current struct check_t uses std::to_string to convert numerical values to string.

But what if someone writes CHECK(std::string("abc") == std::string("abc"))?

The expected result would be a successful test, but in practice, the code does not compile, because there are no std::to_string overloads that take a string.

This is, currently, a big limitation of the more advanced macro compared to the simple one.

A better approach would be to test, at compile-time, if a given class can be printed, and if not use a placeholder value; for example <obj>.

Thanks to if constexpr and requires this can be done with few lines of code, otherwise the proposed test suite would have had a great handicap:

#include <sstream>
#include <string>

template <typename T>
constexpr std::string try_tostr(const T& t){
  if constexpr(requires{ (std::ostringstream() << t).str(); }){
    return (std::ostringstream() << std::boolalpha << t).str();
  } else {
    return "<obj>";
  }
}

The function can still be improved; for example, if the input is a std::string, a null-terminated char* or std::string_view, then it is possible to avoid using a stream altogether and just copy the content.

Also, some types from the standard library do not have a stream operator; for example std::vector<int>, but for the test suite, it would make sense to print the size and all the elements one by one.

Pointers to other objects, if not nullptr, could be first dereferenced and then streamed, to provide a more useful result.

Other types, like std::wstring, have the stream operator, but they need to work with a wide stream. It also makes sense to support those types, somehow, out of the box.

Either way, even without those improvements, try_tostr is good enough for now, and even more powerful than the facilities provided by some other test suites (some will not compile if they do not know how to print a type).

Test runner

Another feature that I consider very important is how to register a test.

Until now, I had to call manually the tests from main.

This is cumbersome, especially if you have a lot of tests, as it means (unless you write all the tests inline or in the same file) to write the same name two or three times: a declaration in the header file, a definition, and calling it from main.

This is a lot of boilerplate code; unlike "normal" functions, which are generally used in more places, tests are called in only one place.

Also, what if I want to execute only a subset of the tests?

Or randomize the order of execution to ensure there are no hidden dependencies?

Or execute them from different threads?s

A test runner is responsible for those tasks, but I want one that knows "automatically" what are the tests to execute. I do not want to write the name of every test three times!

In other languages, tests are annotated or follow a specific naming convention that can be queried at runtime; in C++, there is unfortunately no such mechanism.

This is the simplest solution (approximately 30 lines of code) that came to my mind:

#include <iostream>
#include <vector>
#include <cassert>

using function_sig = void();

constinit struct {
  std::vector<function_sig*> tests = {};
  int add(function_sig* test) {
    tests.push_back(test);
    return 0;
  }
  void execute_tests(){
    for(const auto& t : tests){
      try{
        t();
      } catch(...){
        std::cerr << "test failed\n";
      }
    }
  }
} runner;

#define CONCAT_IMPL(x, y) x##y
#define CONCAT(x, y) CONCAT_IMPL(x, y)
#define TEST_CASE(name) static void name(); int CONCAT(impl, __LINE__) = runner.add(name); static void name()

// example usage
int add(int a, int b) {return a + b + 1;}

TEST_CASE(test1) {
  assert(add(1, 2) == 3);
  assert(add(4, 5) == 9);
};

TEST_CASE(test2) {
  assert(1 == 1);
};

int main() {
  runner.execute_tests();
}

The test runner runner executes functions; for brevity, I’ve used assert here.

Improved error messages

If an exception is thrown, the test runner reports that the test failed, but it does not report the function name.

The solution is to store in the runner also the name, not only a pointer:

#include <iostream>
#include <vector>

struct testfunction{
  using function_sig = void();
  function_sig& function;
  std::string_view name;
};

constinit struct {
  std::vector<testfunction> tests = {};
  int add(testfunction test) {
    tests.push_back(test);
    return 0;
  }
  void execute_tests(){
    for(const auto& t : tests){
      try{
        t.function();
      } catch(...){
        std::cerr << "test " << t.name << " failed\n";
      }
    }
  }
} runner;

#define CONCAT_IMPL(x, y) x##y
#define CONCAT(x, y) CONCAT_IMPL(x, y)
#define TEST_CASE(name) static void name(); int CONCAT(impl, __LINE__) = runner.add({name, #name}); static void name()

// example usage

TEST_CASE(test2) {
  throw 42;
};

int main() {
  runner.execute_tests();
}

Disable at compile time

If you want to embed the test suite in your normal application, then you might want to be able to disable it at compile time.

It is sufficient to replace the current TEST_CASE macro with

#define TEST_CASE(name) static void name()

The compiler/linker will remove the unused function.

It is also possible to remove other parts of the test suite, but this is the minimum amount of necessary work.

Missing improvements

Some possible improvements that are not proposed here, as I did not manage to implement them in 100 lines of code.

I guess that most of them, taken singularly, would not increase dramatically the total amount of code. However, implementing them all will blow up the implementation.

Better error messages

The error message are very simple and unstructured, the only justification is that I had no lines to spare.

Better compiler support

This is the feature that could make the current implementation much more complex. It depends on the toolchain you are using.

I’ve used modern C++ features, like string_view, source_location, __VA_OPT__, constinit, if constexpr, and requires.

None of those features are required but they help to keep the code more short and easier to maintain.

If you need to support older compilers or an older standard, you might need to implement some of those features by yourself. In particular with the MSVC compiler, constinit and std::vector do not play well together ^🗄️.

This particular issue can be avoided with the singleton pattern; it is not as nice as constinit, but it is not a big issue.

There is also another known issue, potentially much more problematic.

The autoregistration functionality relies on global variables. A linker might decide to remove them, as unused. There are workarounds that depend on your toolchain, which is not ideal.

Fewer allocations

Currently, there are two places where an allocation occurs.

In some environments (embedded for example) allocation can be problematic.

The first one is inside info_t used in CHECK, for storing lhs and rhs.

A possible workaround would be to use buffers with a fixed maximum length, although the question is, which should be the maximum length?

info_t could also use two (thread-local) global buffers; one for lhs, and one for rhs.

Since it is used only as a parameter for check_internal, in one thread there should never be two instances overwriting the content of the buffer before it has been read out again, even when one CHECK function calls another CHECK function:

int add(int a, int b) {return a + b + 1;}

bool helper(int i){
  CHECK(add(i,i) == 2*i);
  return false;
}

TEST_CASE(test) {
  CHECK(helper(1));
  helper(2);
};

Thus CHECK is reentrant even if using global buffers.

The second allocation is in runner when collecting all tests to execute.

In theory, the amount of TEST_CASE is known at compile time.

In practice, TEST_CASE appear on different translation units that are compiled separately, thus this information is not available, and cannot be used with constexpr/consteval.

One could use a buffer that is "big enough". The buffer has to be global, and since the elements in struct testfunction are non-owning, they should not take too much space (one could reduce the size further by using const char* instead of string_view…).

UPDATE: A better, and still standard-compliant alternative, as explained here, is to use a linked list

#include <cstdio>

using test_signature = void();

struct node {
  private:
    inline static constinit const node* first = nullptr;
  public:
    static node const* start() noexcept {
      return first;
    }
    test_signature* const test;
    const node* const next;
    explicit node(test_signature* t) noexcept : test( t ), next(first)
    {
      first = this;
    }
};

#define CONCAT_IMPL(x, y) x##y
#define CONCAT(x, y) CONCAT_IMPL(x, y)
#define TEST_CASE(name) \
  static void name(); \
  const auto CONCAT(impl, __LINE__) = node(name); \
  static void name()

TEST_CASE(test1){ std::puts("test1"); }
TEST_CASE(test2){ std::puts("test2"); }

int main() {
  for(auto i = node::start(); i != nullptr; i = i->next){
    i->test();
  }
}

It does not need to allocate memory, and it does not require to set a maximum size.

Support JUnit XML reports

And eventually other report formats.

The advantage of the JUnit XML report is that it can be parsed by other tools, for example, a pipeline process.

The current CHECK macro and runner object simply write to stderr/std::cerr in free form. This approach also has the drawback that the output of the test suite might be mixed with the output of the tested code.

A better approach would be to have the output stream configurable.

Report failures directly to the runner

In the snippet, I’ve only shown CHECK, a function/macro for verifying the correctness of code that does not stop the test that is running.

Normally test suites have also a similar terminating function; ASSERT.

Terminating means that something similar to the following happens: an exception is thrown if the tested value is not true, and the exception pops up until it is caught by the runner, which reports that the test suite failed.

As any piece of code can catch exceptions, even the code that is being tested, relying solely on exceptions for detecting a failed test is not a robust approach.

One could verify if some errors have been written to stderr, but the tested code could write to stderr too, and std::cerr could have been closed.

A more robust test suite should implement a separate "communication channel" between CHECH/ASSERT and the runner, to ensure that failed tests are reported correctly. A global boolean variable could do the job (should be thread-local if the test suite supports parallel execution), although a more complex data structure mechanism (a counter, exception_ptr, separate stream channel, …) could be used to gather even more information.

bool test_failed = false;
constinit struct {
  // ...
  void execute_tests(){
    for(const auto& t : tests){
      try{
        test_failed = false;
        t.function();
        if(test_failed){
          std::cerr << "test " << t.name << " failed\n";
        }
      } catch(...){
        std::cerr << "test " << t.name << " failed\n";
      }
    }
  }
} runner;

void check_internal(info_t i, std::string_view additional_info = ""){
  if(not i.result){
    test_failed = true;
    std::cerr << "  " << i.sl.file_name() << ":" << i.sl.line() << " epxr: " << i.expr << " , with values:  " << i.opt_lhs << " and " << i.opt_rhs << additional_info << "\n";
 }
}

Improvements for the test runner

Filter functionality

I mentioned that it is the responsibility of a test runner to be able to execute only a subset of tests, but the current implementation always executes them all.

The execute_tests function could be extended to take something like a predicate or regex and use the function name as an argument for deciding whether to execute the test. Probably the method with the biggest ROI would be to use an std::regex, and let the user set it from the command line when invoking the binary.

Random execution order

To ensure that there is no hidden dependency between tests, the test runner should

randomize the execution order if not instructed otherwise
print the seed used for determining the execution order
accept the seed as a parameter for replaying a given execution order

Parallel tests

less execution time
use more of the available resources on your machine
global stateful data that is not easy to reset
write different test suites, and execute them independently
threads
fork processes

Death tests

Support for death tests.

Ensure that a process ends when a specific error condition arises.

Most test suites do not support it because they do not support fork/multiple processes, as they are OS-specific.

Additional information

Something like…

execution time of every single test
execution time of all tests
how many tests were executed
how many tests did fail

It should be possible to enable and disable such reports through a flag.

Improvements for the test `CHECK` macro

Report test name on failure

In case of errors, CHECK shows the current function and line, but it might not be enough, as a function can be called from different tests.

int add(int a, int b) {return a + b + 1;}

void helper(int i){
  CHECK(add(i,i) == 2*i);
}

TEST_CASE(test2) {
  helper(1);
  helper(2);
};

TEST_CASE(test1) {
  helper(-1);
};

A possible improvement is thus to print on failure the current test name too.

Capture exceptions

If something inside CHECK throws, then the exception escapes the scope.

It is possible to extend the comparison operator of check_t to store an exception in an exception_ptr, which can be queried from check_internal, but this will not be enough, as

bool foo(int i){
  throw 42;
}

TEST_CASE(test2) {
  CHECK(helper());
  helper(2);
};

does not invoke the comparison operator.

A more complete approach might be to use an anonymous lambda with an embedded try-catch clause and log if an exception is thrown.

User-provided print functions

Some types cannot be printed, but at least they will not trigger a compile error.

Users of the test suite should have the possibility to define how to print custom types without changing them, either because they cannot, or because they do not want to.

Debugger support

If someone is running the test suite with a debugger, the CHECK macro should add a breakpoint.

With C++26 this should be doable in the 100 line test suite, in the meantime, one can look at the external references and implement it manually.

Stacktrace support

C++23 added apparently support for stacktraces.

This could replace or enhance the current std::source_location mechanism, as it provides more informations.

Conclusion

Put everything together

#include <iostream>
#include <vector>
#include <cassert>
#include <string>
#include <source_location>
#include <sstream>

template <typename T>
constexpr std::string try_tostr(const T& t){
  if constexpr(requires{ (std::ostringstream() << t).str(); }){
    return (std::ostringstream() << std::boolalpha << t).str();
  } else {
    return "<obj>";
  }
}

struct info_t {
  std::string_view expr = "";
  std::source_location sl = std::source_location::current();
  bool result = false;
  std::string opt_lhs = "";
  std::string opt_rhs = "";
};

template <class A>
struct check_t {
 A a;
  info_t& i;
#define CHECK_T_COMP_OP(OP) \
  template <class B> \
  info_t operator OP(B&& b) && { \
    i.opt_lhs = try_tostr(a); \
    i.opt_rhs = try_tostr(b); \
    i.result = (a OP b);\
    return std::move(i);\
  } static_assert(true)
  CHECK_T_COMP_OP(==);
  CHECK_T_COMP_OP(!=);
  CHECK_T_COMP_OP(> );
  CHECK_T_COMP_OP(>=);
  CHECK_T_COMP_OP(< );
  CHECK_T_COMP_OP(<=);
#undef CHECK_T_COMP_OP
  operator info_t() && {
    i.opt_lhs = try_tostr(a);
    i.result = a;
    return std::move(i);
  }
};

struct destruct_t {
  info_t i;
  template <class A>
  check_t<A> operator<(A&& a) && {
    return check_t<A>{std::forward<A>(a), i};
  }
};
#define DECOMPOSE_CMP_OP(expr, lit) destruct_t{info_t{lit ""}} < expr

void check_internal(info_t i, std::string_view additional_info = ""){
  if(not i.result){
    std::cerr << "  " << i.sl.file_name() << ":" << i.sl.line() << " epxr: " << i.expr << " , with values:  " << i.opt_lhs << " and " << i.opt_rhs << additional_info << "\n";
  }
}
#define CHECK(e, ...) check_internal(DECOMPOSE_CMP_OP(e,#e) __VA_OPT__(,) __VA_ARGS__)

struct testfunction{
  using function_sig = void();
  function_sig& function;
  std::string_view name;
};

#ifdef DISABLE_TESTS
#define TEST_CASE(name) static void name()
constinit struct {
  void execute_tests(){}
} runner;
#else
constinit struct {
  std::vector<testfunction> tests = {};
  int add(testfunction test) {
    tests.push_back(test);
    return 0;
  }
  void execute_tests(){
    for(const auto& t : tests){
      try{
        t.function();
      } catch(...){
        std::cerr << "test " << t.name << " failed\n";
      }
    }
  }
} runner;
#define CONCAT_IMPL(x, y) x##y
#define CONCAT(x, y) CONCAT_IMPL(x, y)
#define TEST_CASE(name) static void name(); int CONCAT(impl, __LINE__) = runner.add({name, #name}); static void name()
#endif

which should be 98 lines of code! (phew 😌).

Example usage:

int add(int a, int b) {return a + b + 1;}

TEST_CASE(test1) {
  CHECK(add(1, 2) == 3);
  CHECK(add(4, 5) == 0);
};

TEST_CASE(test2) {
  CHECK(add(1, 2) == 3);
  CHECK(false);
  CHECK(not true);
};

int main() {
  runner.execute_tests();
}

These notes (hopefully) can inspire how to extend an existing test suite (or verification framework) if any of the following features are desired, but missing:

no need to define tests, and register them separately
being able to embed tests in the binary, and remove them through the preprocessor
report the source file, line, and destructured expression
support both printable and non-printable types

Do you want to share your opinion? Or is there an error, some parts that are not clear enough?

You can contact me anytime.