Testing a C++ project: CTest, fixtures and golden images

In an interview for a C++ role you'll be asked "how would you test this?", and in your first week you'll be asked to fix a bug with a test that proves it. Both are about the same habits: tests at the right level, run by one command (ctest), that give the same answer on every machine and, when they fail, say what went wrong. This skill builds those habits for a rendering library — the splat::core library of the scene-viewer project — but nothing here is specific to graphics.

The cpp-project-build skill set up the targets and presets. Here we add the tests: what to test, the framework (doctest, with GoogleTest alongside), CTest, deterministic fixtures, golden images with documented tolerances, regression tests, test seams and malformed input. The build task is the piece every rendering team has and few write well: a golden-image comparator and a reproducible fixture generator.

1. What to test, and at which level

A healthy suite is a pyramid: many small, fast tests at the bottom, a few slow, broad ones at the top.

Rules of thumb that interviewers like to hear, and that are true:

• Test behaviour through the public interface, not private details: a test that breaks on every refactoring gets deleted.
• One reason to fail per test, with a name that says what is promised: "depth sort is stable for equal depths", not "test3".
• Every bug fix comes with a test that failed before the fix (§6).
• Tests are code: reviewed, warning-free, run under the sanitizers (the asan and tsan presets).

2. A test framework: doctest (and GoogleTest)

The project uses doctest: one header, very fast to compile, and its macros read like plain C++. You'll meet GoogleTest in most company codebases, so know both. The same test in each:

// doctest
#include <doctest/doctest.h>

TEST_CASE("depth sort orders splats back to front") {
    std::vector<Splat> s = make_splats({3.0f, 1.0f, 2.0f});
    sort_by_depth(s);
    REQUIRE(s.size() == 3);                 // stop this test case if false
    CHECK(s[0].depth == 3.0f);              // record and continue
    CHECK(s[2].depth == doctest::Approx(1.0f).epsilon(1e-6));

    SUBCASE("equal depths keep input order") {   // runs the case again per SUBCASE
        auto t = make_splats({1.0f, 1.0f});
        t[0].id = 7; t[1].id = 8;
        sort_by_depth(t);
        CHECK(t[0].id == 7);
    }
}

// GoogleTest
#include <gtest/gtest.h>

TEST(DepthSort, OrdersBackToFront) {
    std::vector<Splat> s = make_splats({3.0f, 1.0f, 2.0f});
    sort_by_depth(s);
    ASSERT_EQ(s.size(), 3u);                // ASSERT_* stops the test
    EXPECT_EQ(s[0].depth, 3.0f);            // EXPECT_* records and continues
    EXPECT_NEAR(s[2].depth, 1.0f, 1e-6);
}

class SceneTest : public ::testing::Test {  // a fixture: SetUp() before each TEST_F
protected:
    void SetUp() override { scene = load_scene("fixtures/room.ply"); }
    Scene scene;
};
TEST_F(SceneTest, HasSplats) { EXPECT_GT(scene.size(), 0u); }

Neither is magic. A TEST_CASE macro defines a function and a static object whose constructor registers it before main runs; CHECK records a failure with the file and line and carries on. Here is the whole idea in 40 lines (the harness that grades this course works the same way):

#include <functional>
#include <print>
#include <source_location>
#include <string_view>
#include <vector>

struct TestCase { std::string_view name; void (*fn)(); };
std::vector<TestCase>& registry() { static std::vector<TestCase> r; return r; }
struct Registrar { Registrar(std::string_view n, void (*f)()) { registry().push_back({n, f}); } };
int failures = 0;

void check(bool ok, std::string_view expr, std::source_location loc = std::source_location::current()) {
    if (!ok) { ++failures; std::println("  line {}: CHECK({}) failed", loc.line(), expr); }
}
#define CAT2(a, b) a##b
#define CAT(a, b) CAT2(a, b)
#define TEST_CASE(name)                                         \
    static void CAT(test_, __LINE__)();                          \
    static Registrar CAT(reg_, __LINE__)(name, &CAT(test_, __LINE__)); \
    static void CAT(test_, __LINE__)()
#define CHECK(...) check(static_cast<bool>(__VA_ARGS__), #__VA_ARGS__)

int clamp255(int v) { return v < 0 ? 0 : v > 255 ? 255 : v; }

TEST_CASE("clamp keeps values in range") {
    CHECK(clamp255(-4) == 0);
    CHECK(clamp255(300) == 255);
}
TEST_CASE("clamp is the identity inside the range") {
    CHECK(clamp255(0) == 0);
    CHECK(clamp255(128) == 129);   // a deliberately wrong expectation
}

int main() {
    for (const TestCase& t : registry()) {
        int before = failures;
        t.fn();
        std::println("{} {}", failures == before ? "PASS" : "FAIL", t.name);
    }
    std::println("{} failed check(s)", failures);   // a real runner would return 1 here
}

The project builds one test executable per library (core_tests), with doctest's main defined once (#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN in tests/main.cpp). A single binary links fast, and CTest can still run its cases separately (§3).

3. CTest: one command runs everything

CTest is CMake's test driver. enable_testing() (or include(CTest), which also adds a BUILD_TESTING option) in the top-level CMakeLists.txt turns it on; each add_test registers a command whose exit code is the verdict (0 passes).

# tests/CMakeLists.txt
find_package(doctest 2.4 CONFIG REQUIRED)          # or FetchContent, see cpp-project-build
add_executable(core_tests main.cpp test_sort.cpp test_compare.cpp test_ply.cpp)
target_link_libraries(core_tests PRIVATE splat::core doctest::doctest)

add_test(NAME core_tests COMMAND core_tests)       # a target name runs that target's binary
set_tests_properties(core_tests PROPERTIES
    LABELS "unit"
    TIMEOUT 60
    ENVIRONMENT "SPLAT_FIXTURES=${PROJECT_SOURCE_DIR}/fixtures")

add_test(NAME cli_golden_room
         COMMAND splat-render --scene ${PROJECT_SOURCE_DIR}/fixtures/room.ply
                 --compare ${PROJECT_SOURCE_DIR}/fixtures/golden/room.png --tolerance 2)
set_tests_properties(cli_golden_room PROPERTIES LABELS "e2e;golden")

Instead of one CTest test per executable you can register every TEST_CASE as its own CTest test with doctest_discover_tests(core_tests) (from doctest's doctest.cmake script) or gtest_discover_tests(...); then -R selects single cases and -j runs them in parallel.

The commands you'll use daily:

ctest --test-dir build/debug --output-on-failure        # everything; print output of failures only
ctest --preset asan                                     # a testPreset from CMakePresets.json
ctest --test-dir build/debug -R sort                    # names matching a regex  (-E: exclude)
ctest --test-dir build/debug -L unit -j 8               # by label (-LE: exclude), 8 in parallel
ctest --test-dir build/debug --rerun-failed --output-on-failure
ctest --test-dir build/debug -R tile_binning --repeat until-fail:200 --output-on-failure
ctest --test-dir build/debug -N                         # list tests without running them

--repeat until-fail:N is the tool for a flaky test: run it up to N times and stop at the first failure, with its output. Its mirror image, until-pass:N, retries failures — it hides flakiness, so use it only as an explicit, tracked workaround. --schedule-random shuffles the order to flush out tests that depend on each other.

A test preset keeps those flags in one place, next to the configure presets:

{ "name": "asan", "configurePreset": "asan",
  "output": { "outputOnFailure": true },
  "execution": { "noTestsAction": "error", "jobs": 8 } }

Fixtures: setup and cleanup that run when needed

Some tests need something to exist first: generated scenes, a temp directory, a server. CTest fixtures declare that, so ctest -R golden_render alone still works:

add_test(NAME make_scenes COMMAND gen_fixtures ${CMAKE_BINARY_DIR}/scenes)
add_test(NAME clean_scenes COMMAND ${CMAKE_COMMAND} -E rm -rf ${CMAKE_BINARY_DIR}/scenes)
set_tests_properties(make_scenes   PROPERTIES FIXTURES_SETUP    scenes)
set_tests_properties(golden_render PROPERTIES FIXTURES_REQUIRED scenes)
set_tests_properties(clean_scenes  PROPERTIES FIXTURES_CLEANUP  scenes)

Tests that share a resource without needing each other (one GPU, one port) get RESOURCE_LOCK instead, so -j never runs them together.

4. Deterministic tests

A test that passes on your laptop and fails in CI "sometimes" is worse than no test: people learn to ignore red. Most flakiness in C++ suites comes from four sources, and each has a standard fix.

Random data without a fixed, portable sequence. Never seed from std::random_device or the clock in a test. Fixed-seed std::mt19937 is fine — its output sequence is specified by the standard — but std::uniform_int_distribution, uniform_real_distribution and normal_distribution are not: libstdc++ and libc++ turn the same engine output into different numbers. For fixtures whose bytes end up in golden files, use a generator whose whole algorithm you control. SplitMix64 is ten lines:

#include <cstdint>
#include <print>

struct SplitMix64 {
    std::uint64_t state;
    std::uint64_t next() {
        std::uint64_t z = (state += 0x9e3779b97f4a7c15ULL);
        z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9ULL;
        z = (z ^ (z >> 27)) * 0x94d049bb133111ebULL;
        return z ^ (z >> 31);
    }
    double unit() { return static_cast<double>(next() >> 11) * 0x1.0p-53; }   // [0, 1)
};

int main() {
    SplitMix64 a{1234567}, b{1234567};
    std::uint64_t first = a.next();
    std::uint64_t second = a.next();                  // (two statements: argument order is unspecified)
    std::println("{} {}", first, second);             // the published reference values
    std::println("{}", b.next() == 6457827717110365317ULL ? "reproducible" : "broken");
    SplitMix64 c{42};
    std::println("{:.6f} {:.6f}", c.unit(), c.unit());
}

Pin the first outputs in a test (the build task does). Then a change to the generator fails one obvious test instead of turning every golden image red. When you want varied data (property tests), pick a fresh seed per run, print it on failure, and accept a seed from the environment (SPLAT_TEST_SEED=…) so anyone can replay the failure.

Thread counts that follow the machine. hardware_concurrency() is 4 on a laptop and 64 on a CI runner. If the work split changes, the order of float additions changes — and float addition is not associative:

#include <print>
#include <vector>

// Sums `v` the way a parallel reduction with `chunks` workers would: one partial sum per chunk, then the partials.
float chunked_sum(const std::vector<float>& v, std::size_t chunks) {
    std::vector<float> partial(chunks, 0.0f);
    for (std::size_t i = 0; i < v.size(); ++i) partial[i * chunks / v.size()] += v[i];
    float total = 0.0f;
    for (float p : partial) total += p;
    return total;
}

int main() {
    std::vector<float> v;
    for (int i = 0; i < 100000; ++i) v.push_back(1.0f / static_cast<float>(i % 97 + 1));
    for (std::size_t chunks : {1, 4, 16, 64})
        std::println("{:>2} chunks: {:.6f}", chunks, chunked_sum(v, chunks));
}

Fix the thread count in tests (pass it in; don't read the machine), or make the reduction order independent of it (fixed-size chunks combined in a fixed order), and compare floats with a documented tolerance.

Unstable ordering. std::unordered_map/unordered_set iteration order, std::filesystem::directory_iterator order and the order threads finish are all unspecified. Sort before you print, compare or write a golden file; use std::stable_sort when equal keys must keep their order.

Time and the environment. Sleeps "long enough for the worker", timeouts that assume a fast machine, the current date, the locale, $HOME: replace each with something the test controls (§7). Synchronise with latches and condition variables, never with sleep_for.

5. Golden files with documented tolerances

A golden test runs the code on a fixed input and compares the output with a reviewed reference — an image, a text report, a serialised scene. For images, exact equality is the wrong default: compilers, -ffp-contract (FMA), SIMD paths and library std::exp implementations differ in the last bits, and an 8-bit image rounds those differences to ±1. So every golden test states its tolerance and why:

Two numbers express that: a per-channel absolute tolerance (how far one sample may be off) and a maximum fraction of differing pixels (how many may exceed it). The comparator then reports what a reviewer needs:

• count and fraction of bad pixels — noise or a real change?
• max error — a 1/255 rounding difference or a wrong colour?
• PSNR — one number for dashboards: 10 · log10(peak² / MSE); +∞ for identical images, ~30–40 dB for "visibly similar", 0 dB when the error is as large as the signal.
• first mismatch and bounding box — where to look in the diff image.
• shape mismatch as its own verdict — "rendered 1919 × 1080" is a different bug from "wrong colours", and must never read pixels at all.

Why per channel? Summing the channels' errors before comparing makes the tolerance depend on the channel count and lets one badly wrong channel hide:

#include <array>
#include <cstdint>
#include <cstdlib>
#include <print>

using Rgb = std::array<std::uint8_t, 3>;

bool bad_per_channel(Rgb a, Rgb b, int tol) {
    for (int c = 0; c < 3; ++c)
        if (std::abs(int(a[c]) - int(b[c])) > tol) return true;
    return false;
}
bool bad_summed(Rgb a, Rgb b, int tol) {
    int sum = 0;
    for (int c = 0; c < 3; ++c) sum += std::abs(int(a[c]) - int(b[c]));
    return sum > tol;
}

int main() {
    Rgb golden{100, 100, 100};
    Rgb noise{102, 98, 102};    // anti-aliasing: every channel off by 2
    Rgb wrong{100, 100, 104};   // blue channel off by 4
    std::println("tolerance 3 per channel: noise bad? {:5}  wrong bad? {}",
                 bad_per_channel(golden, noise, 3), bad_per_channel(golden, wrong, 3));
    std::println("tolerance 3 on the sum:  noise bad? {:5}  wrong bad? {}",
                 bad_summed(golden, noise, 3), bad_summed(golden, wrong, 3));
    std::println("tolerance 9 on the sum:  noise bad? {:5}  wrong bad? {}",
                 bad_summed(golden, noise, 9), bad_summed(golden, wrong, 9));
}

Per channel, the noise passes and the wrong blue fails — what you want. Summed with the same 3, the noise fails too; "fix" that by scaling the tolerance to 3 × 3 and the wrong pixel now passes.

The workflow around the comparator matters as much as the comparator:

• On failure, the test writes the actual image and a diff image next to the build (e.g. build/test-artifacts/) and prints their paths with the report; CI keeps them as artifacts.
• Goldens are updated deliberately: an explicit command or environment variable (SPLAT_UPDATE_GOLDEN=1 ctest -R golden) rewrites them, and the change is reviewed in the pull request with the diff images. A test that regenerates its own expectation can never fail.
• Goldens are small: 64 × 64 or 128 × 128 renders of fixture scenes, not 4K frames. Small images keep the suite fast and the repository light.
• Text outputs (reports, scene dumps) are goldens too: sorted, with fixed float formatting (std::format("{:.6f}")), compared line by line.

6. Regression tests from an observed failure

Every bug report is a test case someone else found for you. The order matters:

1. Reproduce the failure exactly (same input, same build type, under the sanitizer preset if it smells like memory).
2. Minimise the input: delete half, check it still fails, repeat. A 2 GB file usually shrinks to a few lines; that small file goes into fixtures/.
3. Write the test first and watch it fail. If it passes, you haven't reproduced the bug — you've written a test of something else.
4. Fix, watch it pass, and keep it with a name that states the promise:

// Regression: issue #41 — loader crashed on a PLY with zero vertices.
TEST_CASE("ply: a header with zero vertices loads as an empty scene") {
    auto scene = splat::load_ply(fixture("ply/zero-vertices.ply"));
    REQUIRE(scene.has_value());
    CHECK(scene->empty());
}

In an interview, "I'd write a failing test from the minimal reproduction first" is the answer to most "how would you fix this bug?" questions.

7. Test seams: inject what the test must control

Code that calls steady_clock::now(), reads files or spawns threads directly is hard to test: you can't make "ten minutes later" happen, or make a file disappear between two reads. A seam is a place where a test can substitute its own implementation without changing production behaviour: a constructor parameter, a template parameter, a function object.

A function object is the lightest seam, and costs nothing in production:

#include <chrono>
#include <functional>
#include <print>
#include <string>
#include <unordered_map>

using Clock = std::chrono::steady_clock;

class ShaderCache {
public:
    explicit ShaderCache(std::function<Clock::time_point()> now = Clock::now) : now_(std::move(now)) {}

    // Returns true if the shader had to be (re)compiled.
    bool get(const std::string& name) {
        auto t = now_();
        auto it = compiled_at_.find(name);
        if (it != compiled_at_.end() && t - it->second < std::chrono::minutes(10)) return false;
        compiled_at_[name] = t;   // "compile"
        return true;
    }

private:
    std::function<Clock::time_point()> now_;
    std::unordered_map<std::string, Clock::time_point> compiled_at_;
};

int main() {
    Clock::time_point fake{};                      // the test owns time
    ShaderCache cache([&] { return fake; });
    std::println("first use compiles: {}", cache.get("splat.frag"));
    fake += std::chrono::minutes(9);
    std::println("9 min later:       {}", cache.get("splat.frag"));
    fake += std::chrono::minutes(2);
    std::println("11 min later:      {}", cache.get("splat.frag"));
}

The same move works for the filesystem (pass a std::function<std::optional< std::string>(const std::filesystem::path&)> reader, or a small interface), randomness (pass the seed or the generator), thread count (a parameter) and the GPU (an interface with a CPU implementation for tests — the project's optional CUDA backend works that way). Don't add seams everywhere: add one where a test needs control it can't otherwise get.

8. Testing malformed input

Production code reads files other people made: truncated downloads, hand-edited headers, files from a newer exporter. Malformed input must give a clear error, never undefined behaviour. For each parser, write tests for:

• empty input; truncated in the header and in the body;
• counts that don't match the data (element vertex 1000 with 3 vertices);
• numbers out of range, negative sizes, sizes whose product overflows;
• wrong types or unknown properties; NaN and infinity in float fields;
• the largest legal input and one more.

Return errors as values (std::expected), and test every branch:

#include <charconv>
#include <cstdint>
#include <expected>
#include <print>
#include <string>
#include <string_view>

struct Header { int width, height, channels; };

// Parses "P <width> <height> <channels>" — a toy image header.
std::expected<Header, std::string> parse_header(std::string_view s) {
    if (!s.starts_with("P ")) return std::unexpected("bad magic");
    s.remove_prefix(2);
    int v[3];
    for (int i = 0; i < 3; ++i) {
        auto [p, ec] = std::from_chars(s.data(), s.data() + s.size(), v[i]);
        if (ec != std::errc{}) return std::unexpected("field " + std::to_string(i) + ": not a valid int");
        s.remove_prefix(static_cast<std::size_t>(p - s.data()));
        if (i < 2) {
            if (!s.starts_with(' ')) return std::unexpected("truncated");
            s.remove_prefix(1);
        }
    }
    if (!s.empty()) return std::unexpected("trailing data");
    if (v[0] <= 0 || v[1] <= 0 || v[2] < 1 || v[2] > 4) return std::unexpected("dimensions out of range");
    if (std::int64_t{v[0]} * v[1] > (std::int64_t{1} << 28)) return std::unexpected("image too large");
    return Header{v[0], v[1], v[2]};
}

int main() {
    for (std::string_view in : {"P 64 32 3", "", "P 64", "P 64 x 3", "P 64 32 3 9",
                                "P -1 32 3", "P 64 32 7", "P 99999 99999 1", "P 99999999999 1 1"}) {
        auto h = parse_header(in);
        if (h) std::println("{:<20} ok {}x{}x{}", in, h->width, h->height, h->channels);
        else   std::println("{:<20} error: {}", in, h.error());
    }
}

Two habits make this cheap. Keep tiny malformed files in fixtures/ (each named after what's wrong: truncated-body.ply, vertex-count-too-big.ply), and run the whole suite under the asan preset, where an out-of-bounds read on bad input crashes loudly instead of passing by luck. Past the hand-written cases, a fuzzer generates malformed inputs for you; a libFuzzer target is one function, built with -fsanitize=fuzzer,address on Clang:

extern "C" int LLVMFuzzerTestOneInput(const std::uint8_t* data, std::size_t size) {
    (void)splat::parse_ply({reinterpret_cast<const char*>(data), size});   // must never crash
    return 0;
}

Every crash a fuzzer finds becomes a minimised fixture and a regression test (§6).

Summary

• Pyramid: many unit tests, some integration tests on small fixtures, a few end-to-end golden tests. Test behaviour through public interfaces.
• doctest CHECK/REQUIRE ≈ GoogleTest EXPECT_*/ASSERT_*. A test macro is a function plus a self-registering static object.
• CTest: add_test + LABELS/TIMEOUT/ENVIRONMENT; run with ctest --output-on-failure, select with -R/-E/-L/-LE, hunt flakes with --repeat until-fail:N; FIXTURES_SETUP/REQUIRED/CLEANUP for setup that must exist.
• Determinism: fixed seeds with a fully specified generator (not std::*_distribution for golden data), fixed thread counts, sorted output, no sleeps or clocks.
• Golden images: per-channel tolerance + max bad fraction, documented per test; a report with count, max error, PSNR, first mismatch and bounding box; updates are explicit and reviewed.
• Bug fixes start with a failing test from a minimal reproduction.
• Seams (injected clock, filesystem, seed, thread count) make the hard cases testable.
• Malformed input returns errors; test each one, run under ASan, fuzz the parsers.

In the project

Ungraded practice in projects/scene-viewer/ (stage 1); the evidence is your own build and test output.

1. Read tests/: how cmake/DoctestTests.cmake turns each doctest case into its own CTest test, how tests/run_cli.cmake checks the CLI's exit status, and how tests/test_render.cpp compares against fixtures/tiny-160x120.ppm with a documented tolerance.
2. Run ctest --preset debug -L unit, -R ply, and the flaky hunt ctest --preset tsan --repeat until-fail:50; note what each selects.
3. Add a test of your own for one splat::core function (e.g. orbit_camera at the poles, or compare with NaN). Watch it fail against a deliberately wrong implementation first.
4. Exercise 01-dangling-scene asks for a CLI golden-image test: write it with your build task's comparator ideas (bounding box and first mismatch in the failure message), and write the actual image next to the build directory on failure.
5. Explain why make_synthetic uses its own splitmix64 instead of std::uniform_real_distribution, and what tests/test_scene.cpp's pinned values protect.