NEON intrinsics

On x86 you met SSE and AVX2 through <immintrin.h>. AArch64's counterpart is Advanced SIMD, universally called NEON: a 128-bit vector unit that every AArch64 application core has. This skill is the NEON version of the AVX2 skill — same ideas (lanes, masks, reductions, tails), different instruction set, and a few things x86 can't do in one instruction.

Everything in this skill builds with the AArch64 cross compiler and runs under qemu. Use the explorer's AArch64 target to read the code GCC produces.

1. Registers and arrangements

• v0–v31: 32 registers of 128 bits — twice as many as AVX2's 16 ymm, half as wide. The low 64 bits are d0–d31, the low 32 s0–s31: scalar float/double math lives in the same registers (fadd s0, s0, s1).
• In assembly the arrangement is a suffix on the register, not part of the mnemonic: add v0.4s, v1.4s, v2.4s adds four 32-bit lanes, add v0.16b, … sixteen bytes. (x86 puts the type in the mnemonic: vpaddd vs vpaddb.)
• AAPCS64 passes the first eight vector arguments in v0–v7 and returns in v0. v8–v15 are partly callee-saved (only their low 64 bits, d8–d15).

Intrinsic types carry the element type and count — much stricter than x86's __m256i:

uint8x16_t   int16x8_t   int32x4_t   float32x4_t   int64x2_t     (q = 128-bit)
uint8x8_t    int16x4_t   int32x2_t   float32x2_t                  (64-bit)
uint8x16x3_t   three registers, from vld3q_u8 (structure loads)

Intrinsic names follow v<op>[q]_<type>: vaddq_s32 (add, q = 128-bit, s32 lanes), vadd_u8 (64-bit register of bytes), vld1q_f32, vgetq_lane_s32. Reinterpreting between types is free and explicit: vreinterpretq_u16_u8(x).

2. No target attribute needed

AVX2 had to be enabled per function and guarded by a CPU check. NEON is part of the AArch64 baseline, so you just include <arm_neon.h>:

#include <arm_neon.h>
#include <cstdio>

int main() {
    int32_t a[4] = {1, 2, 3, 4}, b[4] = {10, 20, 30, 40};
    int32x4_t va = vld1q_s32(a);            // ld1 / ldr q: any alignment
    int32x4_t vb = vld1q_s32(b);
    int32x4_t s = vaddq_s32(va, vb);        // add v.4s
    int32x4_t p = vmlaq_s32(s, va, vb);     // s + a*b  (mla)
    std::printf("%d %d %d %d\n", vgetq_lane_s32(p, 0), vgetq_lane_s32(p, 1),
                vgetq_lane_s32(p, 2), vgetq_lane_s32(p, 3));
    std::printf("sum of lanes: %d\n", vaddvq_s32(p));   // addv: across-lane add
}

Loads and stores (vld1q, vst1q) take any alignment — there is no loadu vs load distinction to get wrong. vdupq_n_s32(x) broadcasts a scalar (dup v.4s, w0), vgetq_lane_s32(v, k) reads one lane (the index must be a compile-time constant).

3. Compares make masks

Like SSE, a compare sets every bit of a true lane: vceqq_u8 gives 0xFF per matching byte, vcgtq_s32 0xFFFFFFFF (-1) per greater lane. Masks drive bitwise operations:

#include <arm_neon.h>
#include <cstdio>

int main() {
    int32x4_t a = {1, 9, -3, 7}, lo = vdupq_n_s32(0), hi = vdupq_n_s32(5);
    uint32x4_t big = vcgtq_s32(a, hi);
    int32x4_t clamped = vbslq_s32(big, hi, a);            // select hi where a > 5
    clamped = vmaxq_s32(clamped, lo);                     // …and 0 below 0
    for (int i = 0; i < 4; ++i) std::printf("%d ", clamped[i]);
    std::printf("\nmask lane 1 = %d\n", (int)vgetq_lane_u32(big, 1));
}

• vbslq(mask, x, y): bitwise select — x's bits where mask bits are 1, y's elsewhere (bsl/bit/bif). The x86 equivalent is blendv or and/andnot/or.
• To count matches, AND the mask with 1s (vandq_u8(eq, vdupq_n_u8(1))), or subtract it (x - (-1) = x + 1). Summing raw 0xFF lanes adds 255 per match.

4. Across-lane reductions — and no movemask

AArch64 NEON can reduce a whole register in one instruction:

vaddvq_s32(v)   addv  s0, v.4s     sum of 4 lanes
vmaxvq_u8(v)    umaxv b0, v.16b    max of 16 bytes
vminvq_f32(v)   fminv s0, v.4s
vaddlvq_u8(v)   uaddlv h0, v.16b   widening sum (16 bytes → 16 bits)

x86 needs shuffle-and-add ladders for this. But x86 has pmovmskb, which packs one bit per byte lane into a general register, and NEON has nothing like it. Two replacements cover nearly every use:

• "Did any lane match?" — vmaxvq_u8(eq) != 0 (umaxv).
• "Which lane matched first?" — narrow the mask so each byte lane becomes a nibble of a 64-bit value, then count trailing zeros:

#include <arm_neon.h>
#include <cstdint>
#include <cstdio>

int main() {
    const char text[17] = "find the x here!";
    uint8x16_t v = vld1q_u8(reinterpret_cast<const uint8_t*>(text));
    uint8x16_t eq = vceqq_u8(v, vdupq_n_u8('x'));
    // shrn: shift each 16-bit pair right by 4 and keep the low byte →
    // byte lane k becomes bits 4k..4k+3 of a 64-bit value.
    uint64_t m = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(eq), 4)), 0);
    std::printf("mask = %016llx, first match at %d\n", (unsigned long long)m, __builtin_ctzll(m) / 4);
}

That three-instruction sequence (cmeq, shrn, umov/fmov) is what glibc's AArch64 memchr and strlen use.

5. Widening and narrowing

Multiplying bytes produces 16-bit results; NEON makes the width change explicit in the name:

vmull_u8(a8, b8)        umull   8B × 8B → 8H       (l = long: results twice as wide)
vmull_high_u8(a, b)     umull2  high halves of 16B registers
vmlal_u8(acc, a, b)     umlal   acc += a × b, widened
vpadalq_s32(acc64, v)   sadalp  add neighbouring pairs into wider lanes
vmovl_u8 / vaddl_u8     widen without multiplying
vmovn_u16(x)            xtn     keep the low half of each lane (truncate)
vqmovn_s32(x)           sqxtn   narrow with saturation   (q = saturating)
vshrn_n_u16(x, 8)       shrn    shift right, then narrow

Narrowing silently truncates unless you ask for saturation (vqmovn, vqshrn), and the _high forms work on the upper half of a 128-bit register so you don't need to extract it first.

6. Structure loads: ld2, ld3, ld4

Interleaved data — RGB pixels, complex numbers, (x, y) pairs — is awkward on x86 (a chain of shuffles). NEON loads and stores it directly:

#include <arm_neon.h>
#include <cstdint>
#include <cstdio>

int main() {
    uint8_t rgb[48];
    for (int i = 0; i < 16; ++i) { rgb[3 * i] = 200; rgb[3 * i + 1] = uint8_t(i * 10); rgb[3 * i + 2] = 0; }
    uint8x16x3_t px = vld3q_u8(rgb);
    // gray = (77 R + 150 G + 29 B) >> 8, computed in 16-bit lanes
    uint16x8_t lo = vmull_u8(vget_low_u8(px.val[0]), vdup_n_u8(77));
    lo = vmlal_u8(lo, vget_low_u8(px.val[1]), vdup_n_u8(150));
    lo = vmlal_u8(lo, vget_low_u8(px.val[2]), vdup_n_u8(29));
    uint8x8_t gray = vshrn_n_u16(lo, 8);
    for (int i = 0; i < 8; ++i) std::printf("%d ", gray[i]);
    std::printf("\n");
}

The weights add up to 256, so the largest sum is 255 × 256 = 65280: it fits 16 bits, and >> 8 brings it back to a byte.

7. Byte shuffles with tbl

vqtbl1q_u8(table, idx) returns table[idx[i]] for every byte, with 0 for any index ≥ 16; vqtbl2q–vqtbl4q use 32–64-byte tables. It is both a general shuffle and a 16-entry lookup table (nibble → hex digit, nibble → popcount…). x86's pshufb is similar but zeroes when the index's top bit is set and otherwise wraps mod 16 — and on AVX2 it works within each 128-bit half only. NEON has no 128-bit lane trap: registers are 128 bits.

Popcount has its own instruction: vcntq_u8 (cnt v.16b) counts bits per byte. Baseline AArch64 has no scalar popcount, so std::popcount(x) moves x into a vector register, runs cnt and addv, and moves the result back (Armv8.9/9.4's FEAT_CSSC added a scalar cnt).

8. The loop shape, again

Nothing changes from AVX2 except the width:

std::size_t i = 0;
for (; i + 16 <= n; i += 16) {         // whole 16-byte chunks only
    uint8x16_t v = vld1q_u8(p + i);
    // … lane-wise work, accumulate or test a mask …
}
for (; i < n; ++i) { /* scalar tail */ }

On this target the tests run without AddressSanitizer (it isn't available for the cross target), so reading past the end isn't reported — it just silently uses whatever follows the buffer. The tests put poison values there so that such a read changes the answer. On real hardware an over-read can cross into an unmapped page and crash.

The compiler vectorizes simple loops for you (at -O2 only when the trip count is friendly, at -O3 more), so intrinsics are for what it won't do: early-exit searches, de-interleaving tricks, table lookups, and exact control over widening.

9. NEON vs AVX2 at a glance

The next skill, SVE, drops the fixed 128-bit width altogether.