The System V calling convention

A function call is a contract between two pieces of machine code that were compiled separately — maybe by different compilers, years apart. The contract is the ABI (application binary interface). On x86-64 Linux, macOS and the BSDs it is the System V AMD64 ABI, extended for C++ by the Itanium C++ ABI. It answers four questions:

1. Where are the arguments?
2. Where does the result go?
3. Which registers may the callee destroy, and which must it give back?
4. What must the stack look like at a call?

Once you know the answers you can read any compiler output, call assembly from C++ and C++ from assembly, and understand why some innocent-looking signatures are slower than others.

Try it live: open the Compiler explorer pane on any snippet, set -O2, and hover a mnemonic for its description. In the Stepper, tick Instructions to watch the argument registers fill up as you step.

1. Integer arguments: rdi, rsi, rdx, rcx, r8, r9

The first six integer or pointer arguments travel in registers, in this order:

So long add3(long a, long b, long c) finds a in rdi, b in rsi, c in rdx, and leaves its result in rax:

#include <cstdio>

extern "C" long add3(long a, long b, long c);

asm(R"(
    .text
    .intel_syntax noprefix
    .globl add3
add3:
    lea rax, [rdi+rsi]      # a + b
    add rax, rdx            # + c
    ret
    .att_syntax prefix
)");

int main() { std::printf("%ld\n", add3(1, 20, 300)); }

extern "C" turns off C++ name mangling (§7), so the assembler label add3 is the symbol the C++ code calls.

Narrow arguments. For an int, short, char or bool, only the low 32/16/8 bits of the register are defined. GCC and Clang zero- or sign-extend them in practice, but the ABI doesn't promise it, and the extension they do is to 32 bits: -5 passed as an int usually arrives as rdi = 0x00000000FFFFFFFB. Use edi (32-bit) or sign-extend with movsxd rax, edi before doing 64-bit arithmetic.

2. Floating-point arguments: xmm0–xmm7

float and double arguments go in xmm0…xmm7, results come back in xmm0 (and xmm1). The crucial detail: integer and floating-point arguments are numbered separately.

d is the fourth parameter but only the second floating-point one, so it's in xmm1. GCC compiles return d; to a single register copy:

first_double:
    movapd  xmm0, xmm1
    ret

Mixing classes is free: a function with six longs and eight doubles still passes everything in registers.

3. Stack arguments and the 16-byte rule

The seventh integer argument (or ninth double) goes on the stack. The caller pushes stack arguments right to left, aligns the stack, and executes call, which pushes the return address. So on entry to a function:

The alignment rule: rsp must be a multiple of 16 at every call instruction. That's why rsp % 16 == 8 on entry: the 8-byte return address sits on top of an aligned stack. If your function calls another one, it must restore alignment first — each push moves rsp by 8, so an odd number of pushes needs an extra sub rsp, 8.

Breaking the rule is a classic bug in hand-written assembly: nothing fails until a callee (often deep in libc) uses movaps to save an SSE register to its stack frame, which faults on an address that isn't a multiple of 16.

The red zone. A leaf function (one that calls nothing) may use the 128 bytes below rsp as scratch space without moving rsp. Signal handlers are required to skip it. That's why -O0 code for small leaf functions writes to [rbp-8] without any sub rsp.

4. Who saves what

When you call a function, which of your registers survive?

Compilers use this directly. To keep x alive across a call, GCC parks it in a callee-saved register — and because the function now uses rbx, it must save its caller's rbx first:

long helper(long);
extern "C" long keep(long x) { return helper(x) + x; }

keep:
    push    rbx              ; save the caller's rbx (also realigns rsp to 16)
    mov     rbx, rdi         ; x survives the call in rbx
    call    helper
    add     rax, rbx
    pop     rbx              ; give rbx back
    ret

Note the free bonus: one push makes rsp a multiple of 16 again, so no sub rsp, 8 is needed.

Checking it: crucible/abi.hpp

The test harness has a helper that calls a function with known "sentinel" values in every callee-saved register and reports which ones changed. This function borrows rbx without saving it:

#include <crucible/abi.hpp>
#include <cstdio>

extern "C" long sneaky(long x);

asm(R"(
    .text
    .intel_syntax noprefix
    .globl sneaky
sneaky:
    mov rbx, rdi             # uses rbx …
    lea rax, [rbx+1]
    ret                      # … and never restores it
    .att_syntax prefix
)");

int main() {
    auto r = crucible::abi::call(sneaky, 41L);
    std::printf("returned %ld, clobbered:", r.ret);
    for (auto& name : r.clobbered) std::printf(" %s", name.c_str());
    std::printf("\n");
}

The answer is right, but the caller's rbx is gone. In real code that's a heisenbug: the caller's loop counter or this pointer silently changes.

5. Returning structs

The ABI classifies each 8-byte chunk ("eightbyte") of a small struct:

• Larger than 16 bytes, or not trivially copyable (user-defined copy/move constructor or destructor) → MEMORY: returned through a hidden pointer.
• Otherwise each eightbyte is INTEGER (returned in rax, then rdx) or SSE (returned in xmm0, then xmm1).

struct Pair   { long a, b; };          // INTEGER, INTEGER → rax, rdx
struct Vec2   { double x, y; };        // SSE, SSE         → xmm0, xmm1
struct Mixed  { long n; double d; };   // INTEGER, SSE     → rax, xmm0
struct Triple { long a, b, c; };       // 24 bytes         → memory (sret)

Pair costs nothing to return:

mkpair:                 ; Pair mkpair(long a, long b) { return {a, b}; }
    mov     rax, rdi
    mov     rdx, rsi
    ret

For Triple, the caller allocates the result and passes its address as a hidden first argument in rdi — every real argument shifts one register to the right — and the callee returns that same address in rax:

mktriple:               ; Triple mktriple(long a, long b, long c)
    mov     QWORD PTR [rdi], rsi       ; a  (a arrived in rsi!)
    mov     rax, rdi                   ; return the hidden pointer
    mov     QWORD PTR 8[rdi], rdx      ; b
    mov     QWORD PTR 16[rdi], rcx     ; c
    ret

This is also how C++'s guaranteed copy elision works at the machine level: the callee constructs the result directly in the caller's object.

Here's a round trip: divmod returns a Pair, and idiv happens to leave quotient and remainder in exactly the right registers:

#include <cstdio>

struct Pair { long quot, rem; };
extern "C" Pair divmod(long a, long b);

asm(R"(
    .text
    .intel_syntax noprefix
    .globl divmod
divmod:
    mov rax, rdi
    cqo                      # sign-extend rax into rdx:rax
    idiv rsi                 # rax = quotient, rdx = remainder
    ret
    .att_syntax prefix
)");

int main() {
    Pair p = divmod(-17, 5);
    std::printf("%ld %ld\n", p.quot, p.rem);   // -3 -2: truncation, like C++
}

6. Passing objects with destructors

The same "trivially copyable" test applies to arguments. A struct with a non-trivial destructor or copy constructor is never passed in registers: the caller creates a temporary, passes its address, and destroys it after the call.

struct Small { int x; };
struct Owned { int x; ~Owned(); };
extern "C" int get_small(Small s) { return s.x; }   // mov eax, edi
extern "C" int get_owned(Owned o) { return o.x; }   // mov eax, DWORD PTR [rdi]

This is why std::unique_ptr<T> passed by value costs a memory round trip while a raw pointer doesn't: unique_ptr has a destructor. (It's also why "trivially relocatable" types were proposed for C++26 — and then removed before the standard was finalised.)

Member functions are no different: this is just an implicit first argument in rdi.

7. Varargs and al

For a call to a variadic function like printf, the caller also sets al to an upper bound on the number of vector registers used for arguments. The callee's prologue uses it to decide whether to spill xmm0–xmm7:

show:                   ; void show(double d) { std::printf("%f\n", d); }
    lea     rdi, .LC0[rip]   ; the format string
    mov     eax, 1           ; one vector register (d in xmm0)
    jmp     printf           ; tail call

Calling a variadic function through a non-variadic prototype (or vice versa) breaks this contract — another reason to never cast function pointers.

8. Names: C++ mangling

The linker only knows symbol names. C++ encodes the namespace, class, parameter types and qualifiers into the name so overloads don't collide (Itanium mangling):

long add(long, long)            _Z3addll
double add(double, double)      _Z3adddd
int ns::f(int)                  _ZN2ns1fEi
void S::g() const               _ZNK1S1gEv

_Z + length-prefixed name + parameter codes (l long, d double, i int, v void). c++filt (and the explorer's Demangle filter) turn them back. extern "C" disables mangling — and therefore overloading — which is what you want for functions defined in assembly.

// Two overloads, two symbols: _Z3addll and _Z3adddd.
long add(long a, long b) { return a + b; }
double add(double a, double b) { return a + b; }
extern "C" long add_c(long a, long b) { return a + b; }   // symbol: add_c

Summary

 integer args   rdi rsi rdx rcx r8 r9, then [rsp+8], [rsp+16], …
 FP args        xmm0 … xmm7 (numbered separately from integers)
 results        rax (+rdx), xmm0 (+xmm1); > 16 bytes → hidden pointer in rdi
 callee-saved   rbx rbp r12 r13 r14 r15
 at a call      rsp % 16 == 0      on entry   rsp % 16 == 8
 red zone       128 bytes below rsp, leaf functions only
 varargs        al = number of vector registers used