Atomics and the C++ memory model

A mutex makes threads take turns. Atomics let them cooperate without taking turns — but then you decide what each thread is guaranteed to see, and the language gives you very few guarantees for free. This lesson is about those guarantees: what a data race is, what atomic operations do, and what the memory orders (relaxed, acquire/release, seq_cst) promise.

Snippets here run under ThreadSanitizer (TSan), like the exercises. TSan tracks happens-before exactly as the standard defines it, so it reports a missing ordering even when the program happens to print the right answer.

1. Data races are undefined behaviour

A data race is two threads accessing the same memory location, at least one of them writing, with no happens-before relation between them and not both atomic. A data race is UB — not "might print a stale value", but "the compiler may assume it never happens":

bool done = false;              // plain bool, written by another thread
while (!done) { }               // the compiler may hoist the load: loop forever

The fixes are a mutex, or making the shared variable std::atomic. Atomic operations never race with each other; the question is what other data they make safe to touch.

2. std::atomic

std::atomic<T> works for trivially copyable T. Integers, pointers and bool are lock-free on every mainstream platform; check with is_always_lock_free.

#include <atomic>
#include <print>

int main() {
    std::atomic<int> n{10};
    int before = n.fetch_add(5);        // returns the OLD value
    int old = n.exchange(100);          // store and return the old value
    std::println("{} {} {}", before, old, n.load());
    std::println("lock-free int: {}", std::atomic<int>::is_always_lock_free);

    int plain = 0;
    std::atomic_ref<int> ref(plain);    // C++20: atomic ops on a plain object
    ref.fetch_add(1);                   // (while any atomic_ref is in use, only access it through atomic_refs)
    std::println("{}", plain);
}

Atomics aren't copyable (std::atomic<int> a = b; doesn't compile — copying would be a load plus a store, two operations, not one).

3. Read-modify-write and compare-exchange

fetch_add, fetch_sub, fetch_or, exchange are single indivisible read-modify-writes. Anything they can't express — a maximum, a saturating add, pushing onto a linked stack — is a compare-exchange loop:

T expected = a.load();
do {
    desired = f(expected);                       // compute from what you saw
} while (!a.compare_exchange_weak(expected, desired));
// on failure, expected was overwritten with the current value: just retry

compare_exchange_weak may fail spuriously (on LL/SC architectures like AArch64 without LSE), and of course it fails when another thread changed the value first. That's why it lives in a loop. compare_exchange_strong never fails spuriously; use it for one-shot attempts whose failure means something.

Worked example: atomic maximum

#include <atomic>
#include <cstdint>
#include <print>
#include <thread>
#include <vector>

void store_max(std::atomic<std::uint64_t>& m, std::uint64_t v) {
    std::uint64_t cur = m.load(std::memory_order_relaxed);
    // Stop as soon as the stored max is already >= v; otherwise try to install v.
    while (v > cur && !m.compare_exchange_weak(cur, v, std::memory_order_relaxed)) {
    }
}

int main() {
    std::atomic<std::uint64_t> m{0};
    {
        std::vector<std::jthread> ts;
        for (std::uint64_t t = 0; t < 4; ++t)
            ts.emplace_back([&, t] {
                for (std::uint64_t i = 0; i < 1000; ++i) store_max(m, i * 4 + t);
            });
    }   // jthreads join here
    std::println("max = {}", m.load());
}

relaxed is fine here: nobody uses the maximum to decide whether some other data is ready. The join at the end synchronizes the final read.

4. Memory orders

Every atomic operation takes a std::memory_order. It says nothing about the atomic itself (that is always consistent) — it says what other memory a thread is guaranteed to see.

Release/acquire: message passing

The pattern you'll use most: write the data, then release-store a flag; the reader acquire-loads the flag, then reads the data.

#include <atomic>
#include <print>
#include <string>
#include <thread>

std::string message;                // plain data
std::atomic<bool> ready{false};     // the hand-off

int main() {
    std::jthread producer([] {
        message = "hello from the producer, long enough to live on the heap";
        ready.store(true, std::memory_order_release);   // publishes `message`
    });
    std::jthread consumer([] {
        while (!ready.load(std::memory_order_acquire)) { }  // synchronizes-with the store
        std::println("{}", message);                        // guaranteed to see it
    });
}

In words: the release store synchronizes with the acquire load that reads true, so the write to message happens before the read. Change either side to relaxed and TSan reports a data race on message — the counts might still look right on x86, which is exactly why this is dangerous (§5).

Two rules people miss:

1. The acquire must read the value the release wrote (or a later value in that atomic's modification order). Acquire-loading some other atomic does nothing.
2. Release orders what comes before it in program order. Writing the data after the release store isn't published.

Played on a weak (Arm-like) machine, where the hardware really does reorder: first with relaxed operations, then with release/acquire.

seq_cst and the store-buffer test

// x = y = 0 initially
// Thread 1:            Thread 2:
x.store(1);             y.store(1);
r1 = y.load();          r2 = x.load();

With acquire/release (or relaxed) operations, r1 == 0 && r2 == 0 is allowed — each thread's store can still be in its core's store buffer when the other thread loads. With the default seq_cst on all four operations it is forbidden: there is a single order of all seq_cst operations that both threads agree on. You need this for "announce mine, then check yours" algorithms (Dekker/Peterson locks, some hazard-pointer schemes). Most code doesn't — but seq_cst is the default because it's the hardest to get wrong.

5. Why x86 hides your bugs

x86 is TSO (total store order): stores aren't reordered with other stores, loads aren't reordered with other loads. So a relaxed flag publishing data usually "works" on x86 — the hardware happens to give you release/acquire for free (only the store-buffer case above differs). AArch64 and POWER are weakly ordered: without acquire/release the reader really can see the flag before the data. Code that "works on my laptop" ships a bug to every ARM server and phone.

Two more reasons to get it right anyway: the compiler reorders relaxed operations on every architecture, and TSan (which you have here) reports the race regardless of hardware.

6. Waiting without spinning: wait / notify

C++20 atomics can block:

#include <atomic>
#include <print>
#include <thread>

int main() {
    std::atomic<int> state{0};
    std::jthread waiter([&] {
        state.wait(0, std::memory_order_acquire);   // sleeps while state == 0
        std::println("woke with state {}", state.load(std::memory_order_acquire));
    });
    state.store(1, std::memory_order_release);
    state.notify_one();                             // after the store
}

wait(old) returns when the value is no longer old — possibly spuriously, so loop on your condition: while (!ready.load(acquire)) ready.wait(false, acquire);.

7. Cache lines and false sharing

Atomics are coherent per cache line (64 bytes on x86-64 and most ARM cores). Two atomics written by different threads that share a line make the line bounce between cores on every write — false sharing — even though the threads never touch the same variable.

#include <atomic>
#include <new>

struct Shared {                       // two counters, one cache line: they fight
    std::atomic<long> a, b;
};

struct Padded {                       // one line each
    alignas(std::hardware_destructive_interference_size) std::atomic<long> a;
    alignas(std::hardware_destructive_interference_size) std::atomic<long> b;
};

static_assert(sizeof(Padded) >= 2 * std::hardware_destructive_interference_size);

int main() {}

Watch the coherence protocol do it. Two cores with private toy caches (16-byte lines) each run fetch_add — a read and a write — on their own counter. The badge in each slot is the line's MESI state: Modified, Exclusive, Shared, Invalid.

On a typical desktop, two threads incrementing Shared::a and Shared::b run several times slower than with Padded. Paste Padded into the Playground's Memory tab to see the gap. GCC warns about using hardware_destructive_interference_size in headers (its value can change with -mtune, which would change struct layouts across translation units), so library code often uses a named constant like cache_line = 64.

8. Spin locks done right — and why you rarely want one

A spin lock is an atomic flag. The good version is test-and-test-and-set: try to take it with an acquire exchange; while it's taken, spin on a plain relaxed load (which keeps the cache line shared instead of bouncing it with writes); release with a release store.

#include <atomic>
#include <mutex>
#include <print>
#include <thread>
#include <vector>

class SpinLock {
public:
    void lock() noexcept {
        while (locked_.exchange(true, std::memory_order_acquire))
            while (locked_.load(std::memory_order_relaxed)) { }
    }
    void unlock() noexcept { locked_.store(false, std::memory_order_release); }

private:
    std::atomic<bool> locked_{false};
};

int main() {
    SpinLock lock;
    long total = 0;   // plain: protected by the lock
    {
        std::vector<std::jthread> ts;
        for (int t = 0; t < 4; ++t)
            ts.emplace_back([&] {
                for (int i = 0; i < 1000; ++i) {
                    std::lock_guard g(lock);   // SpinLock is Lockable
                    ++total;
                }
            });
    }
    std::println("{}", total);
}

Why rarely: if the lock holder is preempted, every waiter burns a full time slice doing nothing. std::mutex already spins briefly before sleeping in the kernel. Spin locks pay off only for tiny critical sections on threads that won't be preempted (kernels, real-time code, per-core data).

Faded example: a single-producer/single-consumer ring

The producer owns tail_, the consumer owns head_; the slot contents ride along with each hand-off. Fill in the memory orders and the order of operations in try_pop (what must happen before the consumer hands the slot back?):

bool try_push(const T& v) {
    auto tail = tail_.load(std::memory_order_relaxed);         // our own index
    if (tail - head_.load(std::memory_order_/* ??? */) == Capacity) return false;
    slots_[tail & (Capacity - 1)] = v;
    tail_.store(tail + 1, std::memory_order_/* ??? */);
    return true;
}
std::optional<T> try_pop() {
    auto head = head_.load(std::memory_order_relaxed);
    if (head == tail_.load(std::memory_order_/* ??? */)) return std::nullopt;
    /* read the slot and publish the new head — in which order? */
}

Summary

• A data race is UB. Atomics never race; memory orders decide what other data becomes visible.
• relaxed: counters and anything nothing else depends on.
• release store → acquire load of the same atomic reading that store: everything before the release is visible after the acquire.
• seq_cst (default): one total order; needed for store-then-load-the-other patterns.
• Compare-exchange lives in a loop; weak may fail spuriously.
• x86 hides ordering bugs; ARM and the compiler don't; TSan finds them.
• Keep independently written atomics on separate cache lines.
• Spin locks: test-and-test-and-set, acquire/release — and usually just use a mutex.