Debugging failures: sanitizer reports, core dumps, minimal repros

Sooner or later a test fails in CI with forty lines of hex addresses, or the renderer segfaults on a customer's scene and leaves nothing but a core file. What separates a junior from someone you trust with production C++ is not avoiding these failures; it's getting from "it crashed" to "line 6 of splat_store.cpp reallocates the vector while first still refers to it, and here is the test that pins it" quickly and methodically.

This skill is that method. It builds on smart-pointers-ownership (most crashes are ownership mistakes) and threads-jthread-stop-token (the rest are races and deadlocks), and on the presets and CTest workflow of cpp-project-testing.

1. Build so that failures explain themselves

A crash in a release build tells you almost nothing. The same crash under a sanitizer tells you what went wrong, where, and where the memory came from. Three things make reports readable:

• -g — debug info, so frames carry file:line instead of +0x1f3.
• -fno-omit-frame-pointer — the sanitizers' fast unwinder walks frame pointers; without them stacks lose frames.
• -O1 or -O0 — optimised code still works, but inlining merges frames.

ASan and UBSan share one build (-fsanitize=address,undefined); TSan needs its own (its shadow memory is incompatible with ASan's). The flag goes on the link line too — that's what pulls in the runtime. In a target-based CMake project the clean way is presets that set cache variables, so the same CMakeLists.txt builds every flavour in its own directory:

{
  "version": 6,
  "configurePresets": [
    { "name": "asan", "binaryDir": "build/asan",
      "cacheVariables": {
        "CMAKE_BUILD_TYPE": "Debug",
        "CMAKE_CXX_FLAGS": "-fsanitize=address,undefined -fno-omit-frame-pointer -fno-sanitize-recover=undefined",
        "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" } },
    { "name": "tsan", "binaryDir": "build/tsan",
      "cacheVariables": {
        "CMAKE_BUILD_TYPE": "Debug",
        "CMAKE_CXX_FLAGS": "-fsanitize=thread -fno-omit-frame-pointer",
        "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=thread" } }
  ],
  "testPresets": [
    { "name": "asan", "configurePreset": "asan", "output": { "outputOnFailure": true },
      "environment": { "ASAN_OPTIONS": "detect_stack_use_after_return=1:abort_on_error=1",
                       "UBSAN_OPTIONS": "print_stacktrace=1:halt_on_error=1" } },
    { "name": "tsan", "configurePreset": "tsan", "output": { "outputOnFailure": true },
      "environment": { "TSAN_OPTIONS": "second_deadlock_stack=1:halt_on_error=1" } }
  ]
}

Runtime options matter as much as flags:

If frames show (prog+0x1a2b) but no file:line, the symbolizer is missing: install llvm-symbolizer (or point ASAN_SYMBOLIZER_PATH at it), or symbolize by hand with addr2line -e prog -f -C 0x1a2b.

2. Reading an ASan report

Here is a real report (GCC 16, -g -fno-omit-frame-pointer, the build directory renamed). The program kept a pointer to a Scene, deleted it, then used it:

==1819473==ERROR: AddressSanitizer: heap-use-after-free on address 0x7bb4571e0048 at pc 0x55b58e12729e bp 0x7ffe5018c400 sp 0x7ffe5018c3f0
READ of size 8 at 0x7bb4571e0048 thread T0
    #0 0x55b58e12729d in std::vector<int, std::allocator<int> >::size() const /usr/include/c++/16/bits/stl_vector.h:1109
    #1 0x55b58e12534c in count_ids(Scene const*) /home/dev/scene-viewer/core/src/scene_stats.cpp:4
    #2 0x55b58e1256d8 in main /home/dev/scene-viewer/core/src/scene_stats.cpp:9
    #3 0x7f8458027780  (/usr/lib/libc.so.6+0x27780) (BuildId: d320029ad1a23be2f4bce117fd321cbd23a2014b)
    #4 0x7f84580278b8 in __libc_start_main (/usr/lib/libc.so.6+0x278b8) (BuildId: d320029ad1a23be2f4bce117fd321cbd23a2014b)
    #5 0x55b58e1251e4 in _start (/home/dev/scene-viewer/build/asan/scene_stats+0x61e4)

0x7bb4571e0048 is located 8 bytes inside of 24-byte region [0x7bb4571e0040,0x7bb4571e0058)
freed by thread T0 here:
    #0 0x7f845912e421 in operator delete(void*, unsigned long) (/usr/lib/libasan.so.8+0x12e421)
    #1 0x55b58e1256cc in main /home/dev/scene-viewer/core/src/scene_stats.cpp:8
    ...
previously allocated by thread T0 here:
    #0 0x7f845912d2a1 in operator new(unsigned long) (/usr/lib/libasan.so.8+0x12d2a1)
    #1 0x55b58e125509 in load() /home/dev/scene-viewer/core/src/scene_stats.cpp:5
    #2 0x55b58e125685 in main /home/dev/scene-viewer/core/src/scene_stats.cpp:7
    ...
SUMMARY: AddressSanitizer: heap-use-after-free /home/dev/scene-viewer/core/src/scene_stats.cpp:4 in count_ids(Scene const*)
Shadow bytes around the buggy address:
=>0x7bb4571e0000: fa fa 00 00 00 fa fa fa fd[fd]fd fa fa fa fa fa

Read it in this order:

1. The header: the kind (heap-use-after-free) and that it was a READ of size 8 in thread T0.
2. The access stack, from the top, skipping library frames. #0 is inside std::vector::size() — the vector is innocent, it was handed a dead object. The first frame with a source file of yours, #1 count_ids at scene_stats.cpp:4, is where you were using it. Skip frames with no file (interceptors, libc, _start), frames under /usr/, and std::/__ names.
3. The related stack: freed by — the first user frame (main:8) is what ended the object's lifetime. For an overflow there's no free; the allocated by stack tells you which buffer you ran off.
4. The region line: "8 bytes inside of 24-byte region" — the offset tells you which member (ids, at offset 8 inside Scene, is a vector of three pointers: 24 bytes).

The shadow bytes show how ASan knew. Every 8 bytes of application memory have one shadow byte: 00 addressable, fa heap redzone (padding ASan puts around every allocation), fd freed. Freed memory is kept in a quarantine instead of being reused immediately, so a later access lands on fd and is caught:

The kinds you'll see most, and what each asks you:

Most of these are ownership bugs, and the fix is an ownership fix. The classic one — a reference into a vector across a push_back — has its free deep inside libstdc++, but the first user frame on the freed stack points straight at the push_back:

#include <cstddef>
#include <print>
#include <vector>

struct Splat { float x, y, z, opacity; };

// Buggy version: `Splat& first = splats[0]; splats.push_back(...); return first.opacity;`
// push_back may reallocate and free the buffer `first` points into.
float brighten_first(std::vector<Splat>& splats) {
    std::size_t first = 0;                 // an index survives reallocation
    splats.push_back({0, 0, 0, 1});
    return splats[first].opacity * 2;
}

int main() {
    std::vector<Splat> s(1, Splat{1, 2, 3, 0.5f});
    std::println("{} (size {})", brighten_first(s), s.size());
}

3. LSan and UBSan

LeakSanitizer runs at exit and reports every block nothing points to any more:

==1850794==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f23cd32d2a1 in operator new(unsigned long) (/usr/lib/libasan.so.8+0x12d2a1)
    #1 0x562e0c9d1393 in make_node(char const*) /home/dev/scene-viewer/core/src/scene_graph.cpp:6
    #2 0x562e0c9d18bb in main /home/dev/scene-viewer/core/src/scene_graph.cpp:9

Indirect leak of 48 byte(s) in 1 object(s) allocated from:
    ...
SUMMARY: AddressSanitizer: 96 byte(s) leaked in 2 allocation(s).

A direct leak is unreachable from anywhere; an indirect leak is reachable only from leaked blocks. Fix the direct ones — usually by giving the allocation a unique_ptr owner — and the indirect ones disappear with them.

UndefinedBehaviorSanitizer reports UB that isn't about memory ownership: signed overflow, shifts out of range, out-of-bounds indexing of real arrays, null member access, misaligned loads, invalid bool/enum values. Its report is one line, file:line:col: runtime error: …, plus a stack with print_stacktrace=1:

/home/dev/scene-viewer/core/src/raster.cpp:3:56: runtime error: signed integer overflow: 50001 * 50001 cannot be represented in type 'int'
    #0 0x563eafd581e2 in tile_offset(int, int, int) /home/dev/scene-viewer/core/src/raster.cpp:3
    #1 0x563eafd582b2 in main /home/dev/scene-viewer/core/src/raster.cpp:7

Two things make UBSan easy to miss: without halt_on_error it prints and keeps going (the test passes), and in an ASan+UBSan build it often fires just before ASan does, so the UBSan line sits above the big ASan report. Read the whole log. For arithmetic that can legitimately overflow — pixel offsets on a 16k × 16k image — check it instead of hoping:

#include <cstdint>
#include <print>
#include <stdckdint.h>

// C++26 <stdckdint.h>: returns true if the exact result didn't fit.
bool tile_offset(int ty, int pitch, int tx, int& out) {
    int row = 0;
    if (ckd_mul(&row, ty, pitch)) return false;
    return !ckd_add(&out, row, tx);
}

int main() {
    int off = 0;
    std::println("small: {} {}", tile_offset(3, 512, 7, off), off);
    bool ok = tile_offset(50001, 50001, 3, off);
    std::println("large: {}", ok ? "fits" : "overflow detected");
    std::int64_t wide = std::int64_t{50001} * 50001 + 3;   // or widen first
    std::println("as int64: {}", wide);
}

4. Reading a TSan report

ThreadSanitizer tracks, for every memory location, which threads accessed it and whether those accesses are ordered by a happens-before relation (a lock released then acquired, a thread join, an atomic release/acquire pair). When two accesses from different threads, at least one a write, are not ordered, it reports a data race:

WARNING: ThreadSanitizer: data race (pid=1822914)
  Read of size 8 at 0x7fff84b67ff8 by main thread:
    #0 read_frames(Stats const&) /home/dev/scene-viewer/core/src/render_stats.cpp:5 (render_stats+0x12a2)
    #1 main /home/dev/scene-viewer/core/src/render_stats.cpp:6 (render_stats+0x1338)

  Previous write of size 8 at 0x7fff84b67ff8 by thread T1:
    #0 render_loop(Stats&) /home/dev/scene-viewer/core/src/render_stats.cpp:4 (render_stats+0x125d)
    #1 void std::__invoke_impl<void, void (*)(Stats&), std::reference_wrapper<Stats> >(...) /usr/include/c++/16/bits/invoke.h:63
    ...
  Location is stack of main thread.

  Thread T1 (tid=1822917, finished) created by main thread at:
    #0 pthread_create <null> (libtsan.so.2+0x616c9)
    #2 main /home/dev/scene-viewer/core/src/render_stats.cpp:6 (render_stats+0x1324)

SUMMARY: ThreadSanitizer: data race /home/dev/scene-viewer/core/src/render_stats.cpp:5 in read_frames(Stats const&)

Note the format differences from ASan: no addresses in frames, the module comes after the location, unknowns are <null>. The blocks:

• The first access block is the access TSan was checking when it noticed (Read … by main thread).
• Previous write is the earlier, conflicting access in the other thread. "Previous" is about TSan's history, not a guarantee of order — that's the bug.
• Location is … says what the memory is (a heap block with its allocation stack, a global, a stack variable).
• Thread T1 … created by identifies T1; it's not part of the race.

The other report you'll meet is lock-order inversion: TSan records "held M0 while taking M1" edges for every thread, and reports a cycle — a deadlock that may not have happened this run:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1823009)
  Cycle in lock order graph: M0 (0x558c6a7360c0) => M1 (0x558c6a736100) => M0

  Mutex M1 acquired here while holding mutex M0 in thread T1:
    #2 std::mutex::lock() /usr/include/c++/16/bits/std_mutex.h:116
    #4 upload() /home/dev/scene-viewer/core/src/scene_cache.cpp:4

  Mutex M0 acquired here while holding mutex M1 in thread T2:
    #2 std::mutex::lock() /usr/include/c++/16/bits/std_mutex.h:116
    #4 evict() /home/dev/scene-viewer/core/src/scene_cache.cpp:5

TSan finds races only on code paths the test actually runs concurrently, so give it real concurrency: several worker threads, a std::latch so they all start together, and ctest --preset tsan --repeat until-fail:50 for the test that flakes.

5. Lifetime bug or concurrency bug?

The two look alike from a distance — "it crashes sometimes" — and the fix is very different, so decide early.

The hard ones are both: ASan reports heap-use-after-free, freed by thread T3, used in thread T1, on one run in fifty. That's a concurrency bug whose symptom is a lifetime bug — one thread ended an object's life while another was using it. The usual production fix is to make the user a co-owner for the duration:

#include <map>
#include <memory>
#include <mutex>
#include <print>
#include <string>
#include <thread>

struct Tile { std::string name; int splats = 0; };

class TileCache {
public:
    // The caller gets shared ownership: eviction can drop the cache's reference,
    // but the tile lives until the last user lets go.
    std::shared_ptr<const Tile> lookup(int id) const {
        std::lock_guard lock(m_);
        auto it = entries_.find(id);
        return it == entries_.end() ? nullptr : it->second;
    }
    void insert(int id, Tile t) {
        std::lock_guard lock(m_);
        entries_[id] = std::make_shared<const Tile>(std::move(t));
    }
    void evict(int id) {
        std::lock_guard lock(m_);
        entries_.erase(id);
    }

private:
    mutable std::mutex m_;
    std::map<int, std::shared_ptr<const Tile>> entries_;
};

int main() {
    TileCache cache;
    cache.insert(7, {"garden/7", 1200});
    auto tile = cache.lookup(7);          // a co-owner, not a raw pointer
    { std::jthread evictor([&] { cache.evict(7); }); }
    std::println("{} still has {} splats; cache has it: {}", tile->name, tile->splats,
                 cache.lookup(7) != nullptr);
}

Returning const Tile* from lookup would be the same code with a race-shaped use-after-free in it — and both ASan and TSan would be needed to see the two halves.

6. Core dumps and gdb

Not every failure happens under a sanitizer. A release binary that crashes should leave a core dump: the process's memory and registers at the moment of death, which gdb can open later.

ulimit -c unlimited                 # allow core files in this shell (the default is often 0)
cat /proc/sys/kernel/core_pattern   # where they go; on systemd: |/usr/lib/systemd/systemd-coredump …
./build/debug/cli/splat-render fixtures/garden.ply -o out.png
# Segmentation fault (core dumped)
coredumpctl list splat-render       # systemd keeps them in the journal
coredumpctl info splat-render       # signal, command line, a first backtrace
coredumpctl gdb splat-render        # open the newest one in gdb
# without systemd-coredump:  gdb ./build/debug/cli/splat-render core

A core is only as useful as the binary's debug info: keep -g builds (or separate debug files) of what you ship. Inside gdb, the handful of commands that matter:

(gdb) bt                        # backtrace of the crashing thread
#0  0x000055555555a291 in fov_of (v=...) at core/src/camera.cpp:4
#1  0x000055555555a34d in render_frame (scene=0x55555556f2b0, view=...) at core/src/render.cpp:41
#2  0x000055555555a5e1 in main (argc=3, argv=0x7fffffffe0b8) at cli/main.cpp:27
(gdb) frame 1                   # select a frame (also: up, down)
(gdb) info args                 # its parameters
(gdb) info locals               # its local variables
(gdb) p view.cam                # print an expression
$1 = (Camera *) 0x0
(gdb) p *scene                  # print what a pointer points to (pretty-printers show std:: types)
(gdb) bt full                   # every frame with its locals
(gdb) info threads              # all threads, * marks the current one
(gdb) thread apply all bt       # every thread's stack: the command for hangs and deadlocks

For a hang, don't kill the process: gdb -p PID (or gcore PID to save a core) and thread apply all bt. A deadlock shows two threads parked in __lll_lock_wait/futex_wait under frames of yours that each hold the lock the other wants. In CI, gdb -batch -ex "thread apply all bt" -p PID from a timeout handler captures it non-interactively. For schedule-dependent bugs, rr record (then rr replay, with reverse-continue) records one failing run so you can replay it deterministically as often as you need.

7. Minimise the reproduction

A failure on a 200 MB scene after 40 seconds is a fact; a failure on 3 splats in 2 ms is a test. Shrink every dimension:

• Input: delete halves of the file (or of the splat list) while it still fails; tools like creduce/cvise do this for source code, and ddmin-style shrinkers for data.
• Configuration: one thread instead of eight (if it still fails, it's not a race), one tile, smallest image size, defaults for every flag.
• Code: inline the call chain into one test function, delete what doesn't matter.
• History: if it used to work, git bisect run ctest --preset asan -R tile_cache finds the commit.
• Randomness: if a fuzz-style test found it, print and keep the seed.

#include <cstdint>
#include <print>
#include <vector>

// splitmix64: a tiny seeded generator, so a failing case is reproducible from one number.
std::uint64_t splitmix64(std::uint64_t& s) {
    std::uint64_t z = (s += 0x9e3779b97f4a7c15ULL);
    z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9ULL;
    z = (z ^ (z >> 27)) * 0x94d049bb133111ebULL;
    return z ^ (z >> 31);
}

// The code under test: which of `count` bins a splat's x in [0, width) falls into.
// Bug: it rounds up instead of down, so some x land one past the last bin.
int bin_of(int x, int width, int count) { return (x * count + width - 1) / width; }

int main() {
    for (std::uint64_t seed = 1; seed <= 200; ++seed) {
        std::uint64_t s = seed;
        int width = 1 + static_cast<int>(splitmix64(s) % 64);
        int count = 1 + static_cast<int>(splitmix64(s) % 8);
        int x = static_cast<int>(splitmix64(s) % static_cast<std::uint64_t>(width));
        if (bin_of(x, width, count) >= count) {
            std::println("seed {} fails: width={} count={} x={} -> bin {}", seed, width, count, x,
                         bin_of(x, width, count));
            return 0;   // record the seed and the values: that's the regression test's input
        }
    }
    std::println("no failure in 200 seeds");
}

The printed values — not the seed loop — go into the regression test.

8. Turn it into a regression test

The last step is the one people skip. Before fixing, write the smallest test that fails for the reason you found, run it to see it fail, then fix:

// tests/test_tile_bins.cpp — doctest, run by CTest under every preset
#include <doctest/doctest.h>
#include <splat/tile_bins.hpp>

// Regression: bin_of rounded up, putting x one past the last bin (seed 4 of the bin fuzz
// test: width=11, count=1, x=1 gave bin 1). Seen by ASan as a heap-buffer-overflow when the
// bin index was used to write into the bin array.
TEST_CASE("bin_of never returns a bin past the end") {
    CHECK(splat::bin_of(1, 11, 1) == 0);          // the minimised failing case
    CHECK(splat::bin_of(10, 11, 1) == 0);
    for (int x = 0; x < 64; ++x) CHECK(splat::bin_of(x, 64, 8) == x / 8);
}

Good regression tests share a few habits:

• Named after the behaviour, commented with the evidence — the report kind, the input, the seed. Six months later someone will ask why it exists.
• Deterministic — fixed inputs or a recorded seed, fixed thread counts, no sleeps for correctness; a race test uses a latch to start threads together and runs under the TSan preset.
• Fails before the fix. A test you never saw fail may not test anything — check it against the old code (or git stash the fix).
• Runs where the bug shows. A use-after-free test is only reliable under ASan: label it so CI runs it in the asan preset (ctest --preset asan -L memory).

9. Your task: triage reports automatically

When a CI run fails across twenty tests, the first question is "how many bugs is that?" — and reading twenty reports by hand is slow. In the build task you'll write the triage step this lesson does by eye: parse ASan, LSan, UBSan and TSan text into structured reports — tool, kind, access, every stack with its role and thread — and pick the first user frame of the primary stack and of the related (freed / previous / second-lock) stack, then classify the report as lifetime, bounds, concurrency, undefined behaviour or leak. The tests feed it the real reports shown above, noise included.

In the project

Ungraded practice in projects/scene-viewer/ (stage 3); the evidence is your own build, test and debugger output.

1. The asan and tsan presets build everything with the sanitizer, and their test presets set ASAN_OPTIONS / TSAN_OPTIONS. Run both green.
2. Do the four exercises in exercises/ (exercises/start.sh NN-name). For each, before opening the source, write down from the report alone: the kind, the first frame in the project's code, the related stack, and your lifetime / concurrency / input-validation / build verdict.
3. Exercises 01 and 04 are the lifetime/concurrency pair: write two sentences on which signals told them apart (which tool saw each, -j 1 vs -j 16).
4. In exercise 04, open the core dump of the aborting splat-render with coredumpctl gdb (or gdb … core) and save bt, thread apply all bt and info locals with your note (docs/DESIGN-NOTE-TEMPLATE.md).
5. exercises/check.sh NN-name DIR checks each fix; confirm yourself that your regression test fails on the broken variant.

Summary

• Build to diagnose: -g -fno-omit-frame-pointer, ASan+UBSan in one preset, TSan in another, halt_on_error and print_stacktrace so UB fails tests.
• Read a report top-down: kind and access, then the first user frame of the access stack (skip runtime, libc, /usr/, std::), then the related stack — freed by for lifetime bugs, Previous access for races.
• Deterministic and same-thread → lifetime; schedule-dependent and cross-thread → concurrency, even when ASan is the one that noticed.
• Crashes without sanitizers: core dump, coredumpctl gdb, bt, frame, info locals, p; hangs: thread apply all bt.
• Minimise, then pin it with a deterministic regression test that fails first.