x86-64 stack frames and calls

A crash backtrace, a "stack smashing detected" abort, a mysterious sub rsp, 8 — they all make sense once you can see the stack the way the CPU does. This lesson covers:

• what push, pop, call and ret actually do;
• prologues, epilogues and the frame pointer;
• where locals live: stack slots, spills, the red zone;
• the 16-byte alignment rule at calls, and stack arguments;
• tail calls, and how -O0 and -O2 frames differ.

1. Four instructions

The stack grows down (towards lower addresses); rsp points at the last thing pushed.

So call and ret are nothing more than a push/jump and a pop/jump. If a function leaves rsp anywhere other than where it found it, ret pops something that isn't a return address — usually a crash at a nonsense address.

2. Prologue and epilogue

At -O0, every function sets up a frame pointer:

extern "C" int inc(int x) { return x + 1; }

inc:                              ; GCC -O0
    push  rbp                     ; save the caller's frame pointer
    mov   rbp, rsp                ; rbp = this frame's base
    mov   DWORD PTR -4[rbp], edi  ; x lives in a stack slot
    mov   eax, DWORD PTR -4[rbp]
    add   eax, 1
    pop   rbp                     ; restore the caller's rbp
    ret

With rbp fixed for the whole function, every local is at a constant offset from it (-4[rbp]), and [rbp] holds the caller's rbp, [rbp+8] the return address. Following that chain of saved rbps is how a debugger (or a sampling profiler) walks the stack and prints a backtrace.

leave is the short epilogue: mov rsp, rbp; pop rbp.

At -O2 that function is just:

inc:
    lea   eax, 1[rdi]
    ret

No frame at all — the optimiser keeps x in a register, and -fomit-frame-pointer (the default at -O1+) frees rbp as one more general register. Debuggers then unwind with the DWARF .eh_frame tables instead. Some distributions build with -fno-omit-frame-pointer again because frame-pointer unwinding makes profiling cheap and robust.

3. Locals, spills and the red zone

Locals that don't fit in registers — or whose address is taken — need stack slots. A function that calls nothing (a leaf) can use the 128 bytes below rsp without moving rsp: the red zone, guaranteed by the System V ABI not to be clobbered by signal handlers. That's why inc at -O0 writes to -4[rbp] without any sub rsp.

A function that calls something must actually reserve its space, because the call itself pushes into that area:

extern "C" void h(long*);
extern "C" long local() { long x = 1; h(&x); return x; }

local:                                     ; GCC -O2
    sub   rsp, 24                          ; reserve: x + canary + alignment
    mov   rax, QWORD PTR fs:40             ; stack-protector canary
    mov   QWORD PTR 8[rsp], rax
    xor   eax, eax
    mov   rdi, rsp                         ; &x
    mov   QWORD PTR [rsp], 1
    call  h
    mov   rax, QWORD PTR [rsp]             ; x may have changed
    mov   rdx, QWORD PTR 8[rsp]
    sub   rdx, QWORD PTR fs:40             ; canary intact?
    jne   .L9
    add   rsp, 24
    ret
.L9:
    call  __stack_chk_fail                 ; "*** stack smashing detected ***"

Taking &x forces x into memory. The fs:40 lines are the stack protector (on by default on most Linux distributions): a random value stored next to local buffers and checked before returning, so a buffer overflow that overwrites the return address is caught.

A spill is the same idea forced by register pressure: the compiler runs out of registers and parks a value in a stack slot.

4. Alignment at calls

The ABI requires rsp % 16 == 0 at every call. So on entry to any function, rsp % 16 == 8 (the return address was just pushed). A function that calls others must adjust by an odd multiple of 8 before calling:

extern "C" long g(long);
extern "C" long notail(long x) { return g(x) + 1; }

notail:                 ; GCC -O2
    sub   rsp, 8        ; only for alignment
    call  g
    add   rsp, 8
    add   rax, 1
    ret

Why does it matter? Callees may store 16-byte SSE values on the stack with movaps, which faults on a misaligned address. A misaligned call often works for a while, then crashes deep inside a library — the worst kind of bug.

Counting rule: entry is 8 off; every push moves 8. An odd number of pushes leaves rsp aligned; an even number (including zero) needs one more sub rsp, 8.

5. Callee-saved registers in practice

Values needed after a call must live in rbx, rbp, r12–r15 (the callee keeps them) — and a function that uses those registers must save the caller's values first. That's what the pushes at the top of optimised functions are:

some_function:
    push  r12          ; save the caller's values…
    push  rbx
    sub   rsp, 8       ; 2 pushes: even → realign
    ...                ; …use rbx and r12 across calls…
    add   rsp, 8
    pop   rbx          ; …and restore them, in reverse order
    pop   r12
    ret

6. Stack arguments

The first six integer arguments go in registers. The 7th and later are pushed, last argument first, so the 7th sits at [rsp] at the call:

extern "C" long seven(long, long, long, long, long, long, long);
extern "C" long call7() { return seven(1, 2, 3, 4, 5, 6, 7); }

call7:                  ; GCC -O0
    push  rbp
    mov   rbp, rsp
    sub   rsp, 8        ; alignment for the coming push
    push  7             ; 7th argument
    mov   r9d, 6
    mov   r8d, 5
    mov   ecx, 4
    mov   edx, 3
    mov   esi, 2
    mov   edi, 1
    call  seven
    add   rsp, 16       ; the caller removes the stack arguments
    leave
    ret

Inside seven, the 7th argument is at [rsp+8] (just above the return address), or 16[rbp] after its prologue.

7. Tail calls

When the last thing a function does is return another call's result, there's nothing left to do afterwards — so the optimiser replaces call; ret with jmp, reusing the caller's frame. The callee returns straight to our caller:

extern "C" long g(long);
extern "C" long tail(long x) { return g(x + 1); }

tail:                   ; GCC -O2
    add   rdi, 1
    jmp   g             ; no call, no ret, no frame

That's also why tail-called functions are "missing" from backtraces of optimised code. return g(x) + 1; can't be a tail call: the + 1 happens after g returns.

8. Reading a backtrace

#0  0x0000555555555149 in parse (p=0x0) at main.cpp:7
#1  0x0000555555555180 in load (path=...) at main.cpp:15
#2  0x00005555555551a5 in main () at main.cpp:20

Each line is a frame. The debugger finds frame #1 from frame #0 by reading #0's return address (where #1 resumes), and the saved registers from the frame-pointer chain or unwind tables. A frame "missing" between two others is usually an inlined function or a tail call; a garbage frame address usually means the stack itself was overwritten.

Try it: open the Stepper with Instructions on, step into a function with Step insn, and watch rsp drop by 8 at the call and every push.

Key points

• call = push return address + jump; ret = pop + jump.
• -O0: push rbp / mov rbp, rsp frames, everything in stack slots. -O2: often no frame at all, values in registers.
• Leaf functions may use the 128-byte red zone below rsp.
• rsp % 16 == 0 at every call; entry is 8 off; count your pushes.
• Callee-saved: rbx, rbp, r12–r15 — save before use, restore in reverse.
• Args 7+ go on the stack; the caller cleans them up.
• return f(…); becomes jmp f (tail call) at -O2.