C++ logo
The C++ logo, by Jeremy Kratz, licensed under CC0 1.0 Universal Public Domain Dedication

Compile-time testing in C++

Notes published the
6 - 8 minutes to read, 1526 words
Categories: c++
Keywords: API, c++, constexpr, test

Testing code is not always easy; in particular, testing internal methods (for good or bad) might require changes in the C++ code, and/or the build system.

To avoid such changes, I’ve been using static_assert in some places as a "poor man test suite". Counterintuitively, adding constexpr makes it easier to test existing code!

Test at compile-time

There is a simple way for testing an expression at compile-time: with static_assert.s

static_assert(std::string_view("123").size() == 3);

For such simple cases, there is not much to add.

But most things I want to test cannot be expressed in a single expression.

The "no test suite approach"

To overcome the fact that static_assert accepts only one expression, it is possible to write a lambda and invoke it immediately.

constexpr void my_algorithm(std::string&){ /**/ }

static_assert([]{
   std::string data = "some test data";
   my_algorithm(data);
   assert(data == "expected output");
   return true;
}());

With no external dependencies and no complex setup, it is possible to write some tests that are "executed" at compile-time.

Unfortunately, the test I just wrote down is error-prone to write correctly. In particular, at the beginning, my example looked like the following:

constexpr void my_algorithm(std::string&){ /**/ }

static_assert([]{
  std::string data = "some test data";
  my_algorithm(data);
  assert(data == "expected output");
  return true;
});

I was extremely confused when I noticed that my_algorithm did not perform the expected transformation at runtime! Since the lambda is implicitly convertible to true and I forgot the () at the end, the static_assert did not "execute" the function body, but its conversion operator!

Another issue is that assert depends on NDEBUG, and at least once I’ve implemented the tests without realizing that my assert where not testing the expression I wrote. Again, the test seemed to report success, but in practice my_algorithm was still buggy.

The easiest way to fix it is by throwing an exception in case of error:

constexpr void my_algorithm(std::string&){ /**/ }

static_assert([]{
  std::string data = "some test data";
  my_algorithm(data);
  (data == "expected output") ? void() : throw "abort compile-time test";
  return true;
}());

Given those pitfalls, I decided that for my tests, it would be better to provide some "primitives" for making testing at compile-time more developer-friendly

A minimal test suite

Excluding the documentation, it is just three lines of code:

//!
//! Test suite for executing tests at compile-time
//!
//! Example usage:
//!  FEK_CTEST_BEGIN();
//!    std::string var = "hello";
//!    ctest_check(var.begins_with("h"));
//!  FEK_CTEST_END();
//!

#define FEK_CTEST_BEGIN() static_assert( ([]() -> void { static_assert(true)
#define FEK_CTEST_END()   }(), true) )
consteval void ctest_check(bool p){ p  ? void() : throw "abort compile-time test";}

It does more or less what one did before with static_assert, except that there is no risk of having the lambda convert to true by accident.

Also contrary to assert, ctest_check is always executed, and only works at compile-time, as you should never want to throw a const char* around.

If you try this "test suite" out, you’ll notice that it will not work with the current version of MSVC; the workaround is to no use a function for testing the condition:

//!
//! Test suite for executing tests at compile-time
//!
//! Example usage:
//!  FEK_CTEST_BEGIN();
//!    std::string var = "hello";
//!    FEK_CTEST_END(var.begins_with("h"));
//!  FEK_CTEST_END();
//!

#define FEK_CTEST_BEGIN() static_assert( ([]() -> void { static_assert(true)
#define FEK_CTEST_END()   }(), true) )

#if defined( _MSC_VER )
FEK_CTEST_CHECK(p) (p) ? void() : throw "abort compile-time test"
#else
consteval void ctest_check(bool p){ p  ? void() : throw "abort compile-time test";}
FEK_CTEST_CHECK(p) ctest_check(p)
#endif

At this point, one could have one implementation without the consteval function, but the error messages are shorter when using the consteval function if the test fails. I also hope that I’ll be able to use ctest_check directly with newer compilers and drop the macro FEK_CTEST_CHECK altogether.

Better API

I do not like the API of my minimal test suite:

#define FEK_CTEST_BEGIN() static_assert( ([]() -> void { static_assert(true)
#define FEK_CTEST_END()   }(), true) )

#if defined( _MSC_VER )
#define FEK_CTEST_CHECK(p) (p) ? void() : throw "abort compile-time test"
#else
consteval void ctest_check(bool p){ p  ? void() : throw "abort compile-time test";}
#define FEK_CTEST_CHECK(p) ctest_check(p)
#endif

// code to test
constexpr int square(int num) {
    return num * num;
}

FEK_CTEST_BEGIN();
  FEK_CTEST_CHECK(1!=square(1));
FEK_CTEST_END();

Similarly to the test suite in 100 lines of code, I would love to be able to write something more similar too

FEK_CTEST() {
  std::string data = "some test data";
  my_algorithm(data);
  FEK_CTEST_CHECK(data == "expected output");
}

Since it is not possible to declare a constexpr function, static_assert it, and then define the constexpr function, I currently see no way how to implement such macro.

One could write and use a macro that looks like the following:

#define FEK_CTEST(...) static_assert( ( __VA_ARGS__(), true) )

FEK_CTEST( []{
  std::string data = "some test data";
  my_algorithm(data);
  FEK_CTEST_CHECK(data == "expected output");
});

It works, but since macros are expanded as one-liners, good luck understanding at which line an error happens.

So at the moment the pair FEK_CTEST_BEGIN() and FEK_CTEST_END() is the best interface I could come up with.

Test failures

If a compile-time test fails, it’s game over.

When refactoring a piece of code, I am grateful when I can execute the test suite and see all failing tests at once, instead of knowing only the next one that fails.

With compile-time tests, at least those that are on the same file, tests are "executed" sequentially, and after the first failure, the successive tests are not executed.

If you were thinking more about ensuring that certain constructs are not valid, then this "test suite" does not replace the approach presented here.

On the contrary, you cannot even test that a functions correctly throws an exception on invalid input, as throw will fail the test, even if there is a catch block to handle it.

Error messages

Compiler error messages are far from ideal. The compiler will tell you that the code is not a constant expression, and depending on the compiler you might get some other information.

There is currently no way to inspect values, like printing to the console what the expected and actual values are and seeing the difference.

This makes finding errors in your code or test suite much harder.

Code coverage

There is no code coverage, because there is no code executed at runtime. Unfortunately, it does not play well if you have a code coverage report, as it will be incomplete.

Most of my code is not constexpr, and I do not want to move everything in header files

There is an ugly, but perfectly functional, workaround: leave the declaration as-is, move the implementation to a constexpr function, forward all parameters to the constexpr implementation:

// file.hpp
namespace fek{
  void my_algorithm(std::string&);
}

// file.cpp
namespace{
  constexpr void my_algorithm_impl(std::string&){ /**/ }
  FEK_CTEST_BEGIN();
    std::string data = "some test data";
    my_algorithm(data);
    FEK_CTEST_CHECK(data == "expected output");
  FEK_CTEST_END();
}
namespace fek{
  void my_algorithm(std::string& str){
    return my_algorithm_impl(str);
 }
}

Most of the standard library is constexpr, since C++20 containers like std::string and std::vector too, which makes it possible to test many components at compile-time by simply adding constexpr in front of them.

This pattern obviously does not work once your implementation interacts with libraries that do not have a constexpr API.

Conclusion

If you look at it feature-wise, any runtime test suite can provide many features to the presented compile time test suite:

  • destructuring

  • printing incorrect values

  • statistic about the amount of executed tests

  • support for test failures

  • not breaking tools that analyze code coverage

  • not breaking debugging tools

  • test code that throws exception

  • test code that is not constexpr/consteval

This makes, at least for me, testing at compile time usable only in very particular use cases:

Relatively simple non-exported functions that are usable at compile time.

  • Relatively simple, as otherwise finding out the errors gets too difficult

  • Non-exported, as otherwise using the regular test suite is more practical

Unfortunately, this leaves out most internal functions. While with never standards it is possible to implement more and more internal functions as constexpr, and I’m inclined to do so, doing tests at compile-time might also affect the build times negatively; another reason for not writing too many and too complex compile-time tests.

Do you want to share your opinion? Or is there an error, some parts that are not clear enough?

You can contact me anytime.