x86-64 control flow: flags, jumps, setcc and cmov

Every if, for, ?:, && and switch in your C++ becomes a comparison followed by something that reads the result. This lesson covers:

• how cmp and test set the flags;
• signed vs unsigned condition codes — the place where types live in asm;
• three ways to consume a comparison: jcc, setcc, cmovcc;
• how compilers shape loops (rotation) and switches (jump tables vs compare trees).

1. Flags

Most arithmetic instructions set four flags in rflags as a side effect:

Two instructions exist only to set flags:

• cmp a, b computes a − b and throws the result away;
• test a, b computes a & b and throws it away. test rdi, rdi is the idiom for "compare with zero" (ZF = is zero, SF = is negative).

mov, lea, cmov, setcc and jmp don't touch the flags, which is why compilers can schedule them between a cmp and the instruction that reads it.

2. Condition codes: the same bits, two readings

After cmp a, b, a condition code asks a question about a versus b:

The instruction doesn't know the types: cmp edi, esi is the same for int and unsigned. Signedness is decided by which letter follows. That's why this matters so much when you read or write assembly:

extern "C" int slt(int a, int b) { return a < b; }            // cmp edi, esi ; setl al
extern "C" int ult(unsigned a, unsigned b) { return a < b; }  // cmp edi, esi ; setb al

With a = −1, b = 1: signed, −1 < 1 is true. Unsigned, −1 is 4 294 967 295, which is not below 1. Same bits, opposite answers:

#include <cstdio>

extern "C" int lt_signed(long a, long b);
extern "C" int lt_unsigned(long a, long b);
asm(R"(
    .text
    .intel_syntax noprefix
    .globl lt_signed
lt_signed:
    xor eax, eax
    cmp rdi, rsi
    setl al
    ret
    .globl lt_unsigned
lt_unsigned:
    xor eax, eax
    cmp rdi, rsi
    setb al
    ret
    .att_syntax prefix
)");

int main() {
    std::printf("signed   -1 < 1: %d\n", lt_signed(-1, 1));
    std::printf("unsigned -1 < 1: %d\n", lt_unsigned(-1, 1));
}

3. Three ways to use a comparison

Branch: jcc label. Jump if the condition holds, otherwise fall through. Cheap when the branch predictor guesses right, expensive (~15–20 cycles) when it doesn't.

Materialise a bool: setcc r8. Writes 0 or 1 into an 8-bit register. Because it only writes the low byte, compilers zero the full register first — and before the cmp, since xor changes the flags:

pos:                       ; return x > 0;
    xor   eax, eax
    test  edi, edi
    setg  al
    ret

Select: cmovcc dst, src. Copies src into dst if the condition holds. No branch, so nothing to mispredict; both values must already be computed.

extern "C" long smax(long a, long b) { return a > b ? a : b; }

smax:
    cmp     rdi, rsi
    mov     rax, rsi
    cmovge  rax, rdi       ; signed >=
    ret

Compilers choose between a branch and a cmov heuristically: cmov when both sides are cheap and the condition is unpredictable; a branch when one side is expensive or the condition is predictable. You'll return to this in the branchless lesson.

4. The range-check trick

lo <= x && x <= hi looks like two comparisons and a branch. Subtract lo from everything, then compare unsigned: values below lo wrap around to huge numbers and fail the same single test.

extern "C" int is_digit(int c) { return c >= '0' && c <= '9'; }

is_digit:
    sub   edi, 48          ; c - '0'
    xor   eax, eax
    cmp   edi, 9
    setbe al               ; unsigned (c - '0') <= 9
    ret

5. Loops: rotation

A for loop tests its condition before every iteration. Compiled literally that's two jumps per iteration (one back to the test, one out). Compilers rotate the loop: one guard before it, and the test at the bottom.

extern "C" long sum(const long* a, long n) {
    long s = 0;
    for (long i = 0; i < n; ++i) s += a[i];
    return s;
}

sum:
    test  rsi, rsi            ; guard: n <= 0 → skip the loop
    jle   .L10
    lea   rdx, [rdi+rsi*8]    ; end pointer
    xor   eax, eax
.L9:                          ; body
    add   rax, QWORD PTR [rdi]
    add   rdi, 8
    cmp   rdi, rdx            ; test at the bottom
    jne   .L9                 ; one taken branch per iteration
    ret
.L10:
    xor   eax, eax
    ret

Notice also that the index disappeared: GCC walks a pointer to an end pointer (strength reduction). When you write a loop by hand, keep the guard — a bottom-tested loop without it runs once even when n == 0.

6. switch: jump tables and compare trees

A dense switch (cases close together) becomes a jump table: one unsigned bounds check, then an indirect jump through an array of addresses.

    cmp   edi, 5              ; one unsigned check covers x < 0 too
    ja    .Ldefault
    lea   rdx, .L18[rip]      ; table of 32-bit offsets
    mov   edi, edi
    movsxd rax, DWORD PTR [rdx+rdi*4]
    add   rax, rdx
    jmp   rax                 ; indirect jump

A switch that only returns constants may even become a plain table lookup (CSWTCH in GCC's output). A sparse switch (case 1, 100, 1000, 10000) becomes a binary tree of cmp/je/jg instead:

7. Worked example

extern "C" long count_below(const long* a, long n, long x) {
    long c = 0;
    for (long i = 0; i < n; ++i)
        if (a[i] < x) ++c;
    return c;
}

GCC turns the if into arithmetic — no branch inside the loop:

.L39:
    xor   edx, edx
    cmp   rcx, QWORD PTR [rdi]    ; x vs a[i]
    setg  dl                      ; x > a[i]  ⇔  a[i] < x   (signed)
    add   rdi, 8
    add   rax, rdx                ; c += (a[i] < x)
    cmp   rsi, rdi
    jne   .L39

Note the operands are swapped (x vs a[i]) so the condition flips from l to g. Always read the condition with the operand order of the cmp.

8. Faded example

extern "C" unsigned long umax(unsigned long a, unsigned long b) { return a > b ? a : b; }

umax:
    cmp     rdi, rsi
    mov     rax, rsi
    cmov??  rax, rdi
    ret

(Answer: an unsigned condition — cmovae or cmova; GCC emits cmovnb, which is the same as cmovae.)

Key points

• cmp a, b = flags of a − b; test a, b = flags of a & b.
• l/le/g/ge are signed; b/be/a/ae are unsigned. The types live in the condition code, not the instruction.
• jcc branches, setcc makes a 0/1 byte (zero the register first), cmovcc selects without branching.
• Loops are rotated: guard, body, bottom test. Dense switches use jump tables.