Test failures in C++
When designing a library, or more generically an interface, we need to decide what are valid and invalid parameters.
In compiled languages, it is possible to take advantage of the compiler to do some checks at compile-time, while others need to be done at runtime.
For example, consider a function that generates the fibonacci sequence:
int fibonacci(int);How should the code react if the input is not an integer, for example:
const char* mynumber = "hello 1234";
fibonacci(mynumber);In interpreted languages, it would probably generate a runtime error. It might be an exception, setting a global variable, or simply crashing.
In strong-typed compiled languages we do not normally even care for those cases, as the compiler would catch those errors for us.
While it is standard practice to test valid and invalid inputs at runtime, it is not that common to write failing compile-time tests.
This cannot be implemented directly in the language (at least in C++, other languages might be powerful enough to verify if a snippet can compile or not without hard failing), but can be implemented outside the language pretty easily.
Just try to compile the executable and verify that it fails.
This method seems clunky even if it works. For every method we want to test, we need a separate file (or reuse the same file and take advantage of the preprocessor).
It is important to verify that the compiler error is the expected one, not that the code does not compile for another reason, for example, because of a typo. Thus it is better to move everything except the very line you want to test in another file and compile this another file with the runtime tests to ensure there are no accidental errors.
The next step is integrating this logic into your build system (or test suite), which removes some of the clunkiness.
With CMake, it is not that hard:
add_executable(test.ct test.ct.cpp)
target_link_libraries(test.format.ct foo::foo)
add_test(NAME ct
COMMAND ${CMAKE_COMMAND} --build ${CMAKE_BINARY_DIR} --target test.ct
)
set_target_properties(
test.ct
PROPERTIES
EXCLUDE_FROM_ALL TRUE
EXCLUDE_FROM_DEFAULT_BUILD TRUE
)
set_tests_properties(
ct
PROPERTIES
WILL_FAIL TRUE
)Check for specific compiler errors
Even when minimizing the test code, a typo is just a misplaced or missing letter that could invalidate the tests without anyone noticing.
To avoid such errors, it is possible to grep for the expected error.
g++ test.ct.cpp 2>&1 | grep "compiler message or message inside static assertion"
CMake offers a similar feature: PASS_REGULAR_EXPRESSION 🗄️.
The big advantage is that it will also work on Windows, where grep is normally not available out of the box. The strange side-effect is that we need to drop WILL_FAIL TRUE, as the test succeeds if the regular expression has any match.
if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
set(TEST_CONVERSION_ERROR ".* error: could not convert .*")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang")
set(TEST_CONVERSION_ERROR ".* no known conversion from .*")
elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
set(TEST_CONVERSION_ERROR ".* error C2664: .* cannot convert .*")
else()
message(FAIL "Compile test-suite not implemented for ${CMAKE_CXX_COMPILER_ID}")
endif()
set_tests_properties(
ct
PROPERTIES
PASS_REGULAR_EXPRESSION "${TEST_CONVERSION_ERROR}"
)Notice that error messages of the compiler are not guaranteed to be stable across compiler versions. Since PASS_REGULAR_EXPRESSION expects a list of expressions, it is trivial to cover more and more compiler and compiler versions without too many issues. Another method for making compiler errors more stable is using static_assert with a known message, as this message is part of the diagnostic, but it is not always possible (for example in an SFINAE context).
While I focused on compile-time errors, the same technique can be used for other types of non-recoverable failures.
A common example is checking that when compiling a debug/checked version of the library, a given assert is triggered with certain invalid inputs.
It is difficult to test this in a test suite, as a failing assert calls std::abort, which normally terminates the program. Thus those tests cannot live with others (or at least get executed with others), and need, similarly to a failing compile-time-test, a separate target.
As the test code should compile correctly, there is no need to exclude it from the other targets (thus using EXCLUDE_FROM_ALL and EXCLUDE_FROM_DEFAULT_BUILD). Just like with the compiler errors, it is important to ensure that the test is failing for exactly the reason it was written for.
When testing a simple assert, in case of failure, the expression is printed on stderr. This behavior is mandated by the standard, thus it is possible to add a message and check explicitly for it, for example:
#include <cassert>
int main(){
assert(false && "custom error message");
}It is possible to test for the failure with:
g++ main.cpp && ./a.out | grep "custom error message";And similarly with CMake and PASS_REGULAR_EXPRESSION.
The approach with CMake is not perfect, when using PASS_REGULAR_EXPRESSION it does not take into account the exit code anymore, Another issue is that Ctest treats a crash always as a Failing test, even with WILL_FAIL TRUE. It is possible, instead of executing directly the target, to create a script that would check the error message and that the process did crash or exit with a specific error code. The disadvantage would be that this script is probably platform-specific while invoking the executable from CTest directly had the advantage to be platform-neutral.
And because generally, recovering from some type of errors (programming errors like logic errors), makes no sense.
In an ideal world, we would catch all those unrecoverable errors at compile-time, or during a separate static analysis. In the meantime, tools like assertions and contracts (unfortunately not in C++20) might help us to design and test better interfaces.
Do you want to share your opinion? Or is there an error, some parts that are not clear enough?
You can contact me anytime.