The x86-64 memory model (TSO)

The C++ memory model (Level 8) tells you what a program may observe. This skill looks one level down: what the x86-64 hardware reorders, what each std::atomic order turns into, and why "it works on x86" is not the same as "it's correct".

1. Total Store Order in one picture

Every x86 core has a store buffer: stores retire into it and drain to the cache (and so become visible to other cores) in program order, a little later. Loads read the cache — but first check the core's own store buffer.

That gives TSO (total store order). Of the four ways two memory operations can be reordered, x86 allows exactly one:

Locked instructions (lock add, lock cmpxchg, xchg with memory) and mfence drain the store buffer first, so nothing passes them.

2. Seeing it: a litmus test

<crucible/litmus.hpp> runs two thread bodies many times, releasing them together each iteration, and counts the outcomes. Here is store buffering (SB) with relaxed atomics — compiled at -O2 without sanitizers, because the effect needs the store and the load to be a few cycles apart:

#include <atomic>
#include <crucible/litmus.hpp>
#include <format>

int main() {
    std::atomic<int> x, y;
    int r1 = 0, r2 = 0;
    constexpr auto rlx = std::memory_order_relaxed;
    auto hist = crucible::litmus::run(100'000,
        [&] { x.store(0, rlx); y.store(0, rlx); },
        [&] { x.store(1, rlx); r1 = y.load(rlx); },
        [&] { y.store(1, rlx); r2 = x.load(rlx); },
        [&] { return std::format("r1={} r2={}", r1, r2); });
    crucible::litmus::print(hist);
}

Run it a few times: r1=0 r2=0 usually shows up thousands of times. Both loads ran while both stores were still buffered. Release stores and acquire loads produce the same result — on x86 they are the same instructions.

Play it on a model of the machine: each core's store goes into its store buffer, the loads read memory, and only then do the buffers drain. Switch to SC to see that no interleaving in program order gets there.

Now make the stores seq_cst:

#include <atomic>
#include <crucible/litmus.hpp>
#include <format>

int main() {
    std::atomic<int> x, y;
    int r1 = 0, r2 = 0;
    auto hist = crucible::litmus::run(100'000,
        [&] { x.store(0); y.store(0); },
        [&] { x.store(1); r1 = y.load(); },   // seq_cst
        [&] { y.store(1); r2 = x.load(); },
        [&] { return std::format("r1={} r2={}", r1, r2); });
    crucible::litmus::print(hist);
}

r1=0 r2=0 is gone. A litmus run can prove an outcome possible (you saw it); not seeing an outcome proves nothing — always reason from the model.

On x86 a seq_cst store compiles to xchg (or mov + mfence): either way the core waits for its store buffer to drain before it goes on. Watch the fence hold the load back:

The other classic, message passing (MP: write data, then a flag; read the flag, then data), never shows the bad outcome on x86 hardware because stores stay in order and loads stay in order. It does on AArch64 with relaxed orders — and the compiler is allowed to reorder relaxed accesses on any target. That's the trap: "it passed on my x86 box" tests the hardware, not your program.

3. What each order compiles to

GCC 16 and Clang 23 at -O2 on x86-64:

#include <atomic>
std::atomic<int> a;
int  load_acq()       { return a.load(std::memory_order_acquire); }  // mov eax, a[rip]
void store_rel(int v) { a.store(v, std::memory_order_release); }     // mov a[rip], edi
void store_sc(int v)  { a.store(v); }                                // xchg edi, a[rip]
int  add(int v)       { return a.fetch_add(v, std::memory_order_relaxed); }  // lock xadd

Open the Compiler tab on that snippet and switch to AArch64 to see the contrast: ldar, stlr, ldaxr/stlxr loops or LSE ldadd — there the orders are real instructions.

Two consequences:

• Choosing acquire/release over relaxed costs nothing on x86 — use whatever is correct; x86 won't punish you.
• seq_cst stores and every RMW cost a buffer drain (roughly 20 cycles, much more under contention because the cache line has to be owned exclusively). In hot paths with one writer, prefer load + store over fetch_add, and release over seq_cst unless you need store→load ordering.

4. When you do need store→load ordering

Any algorithm where each thread writes its own variable and then reads another's is store buffering in disguise:

• Dekker / Peterson mutual exclusion: flag[me] = true; … while (flag[other] && …).
• Hazard pointers: publish the pointer, then re-check the source.
• "Last one out" checks: done[me] = 1; if (all others done) ….

These need seq_cst on both the store and the load (or a seq_cst fence between them). Acquire/release isn't enough — x86 proves it in the litmus test above.

5. The compiler reorders too

The hardware is only half the story. Given

int data;
bool ready;           // plain, or volatile
void producer() { data = 42; ready = true; }

the compiler may store ready first, keep data in a register, or merge two stores into one. volatile doesn't help: it only orders volatile accesses among themselves and still leaves a data race (undefined behaviour). An std::atomic<bool> with store(true, release) compiles to the same mov and forbids all of that. On x86, correct atomics are usually free; incorrect plain code is a time bomb.

6. Compare-and-swap at the machine level

lock cmpxchg [mem], new compares eax with memory: if equal it stores new and sets ZF; otherwise it loads the memory value into eax and clears ZF. That's why compare_exchange updates its expected argument on failure — and why a CAS loop needs no extra load:

#include <atomic>
#include <cstdio>

int fetch_max(std::atomic<int>& a, int v) {
    int cur = a.load(std::memory_order_relaxed);
    while (cur < v && !a.compare_exchange_weak(cur, v, std::memory_order_relaxed)) {
        // cur now holds the value another thread wrote; re-check and retry
    }
    return cur;
}

int main() {
    std::atomic<int> m{3};
    std::printf("%d ", fetch_max(m, 10));
    std::printf("%d ", fetch_max(m, 7));
    std::printf("%d\n", m.load());
}

On x86 compare_exchange_weak never fails spuriously (it's the same lock cmpxchg), but write the loop anyway — on AArch64 without LSE it's an LL/SC pair that can.

7. The build: a seqlock

A sequence lock lets one writer update a small record while readers copy it without ever blocking the writer: the writer makes a counter odd, writes, makes it even; a reader copies the data between two reads of the counter and retries if the counter was odd or changed. On x86 every order it needs is free — the codegen stage checks your writer contains no xchg, mfence or lock.