CI, reproducible builds and releases

cpp-project-build got one machine to build the project. A job needs more: every commit is built and tested the same way on several compilers and configurations before it is merged (continuous integration), and what you hand to users is a versioned, reproducible, runnable artifact made by a scripted, boring release process. Interviewers for production C++ roles ask about exactly this — "how would you set up CI for this library?", "a test fails only on Clang with TSan, what do you do?", "how do you know which commit a crash report came from?" — and it's what your first weeks on a team look like: you open a pull request and a matrix of jobs decides whether it can land.

This lesson covers the pipeline from the inside out: the matrix and the local script that runs it, the hosted-CI config, warnings policy, reproducible builds, versions (SemVer and git describe — the build task), install rules, RPATH and a CPack tarball, and the release checklist.

1. The CI matrix

One build proves little: GCC and Clang disagree about warnings, extensions and template corner cases; Debug hides bugs that only -O2 exposes; the sanitizers each catch a different family of bugs, and AddressSanitizer and ThreadSanitizer can't run in the same binary. So CI builds a matrix:

What each column buys you:

Two compilers × four configurations is eight jobs. On a real team you might add an old supported compiler, a different OS or a static-analysis job (clang-tidy) — but every job must be deterministic: a job that fails one time in twenty trains everyone to press "re-run", and then real failures get re-run too.

Write the matrix as presets, then a local script

The source of truth is CMakePresets.json (from cpp-project-build): one configure preset per cell, plus build and test presets with the same names. The CI system — any CI system — then only runs three commands per cell:

cmake --preset gcc-asan                 # configure into build/gcc-asan
cmake --build --preset gcc-asan -j "$(nproc)"
ctest --preset gcc-asan --output-on-failure

CMake ≥ 3.25 can chain the three as a workflow preset (cmake --workflow --preset gcc-asan). Then put the whole matrix in a script in the repository, so that "does it pass CI?" is a question you can answer on your own machine, before pushing, and without the CI service:

#!/usr/bin/env bash
# ci/local-ci.sh — run the CI matrix locally. Exits non-zero on the first failure.
set -euo pipefail
if [ "$#" -gt 0 ]; then presets=("$@")
else presets=(gcc-debug gcc-asan gcc-tsan gcc-release clang-debug clang-asan clang-tsan clang-release); fi
for preset in "${presets[@]}"; do
    echo "::: $preset"
    cmake --preset "$preset" -DCMAKE_COMPILE_WARNING_AS_ERROR=ON
    cmake --build --preset "$preset" -j "$(nproc)"
    ctest --preset "$preset" --output-on-failure
done
echo "all presets passed"

set -euo pipefail is not decoration: without -e, a failing ctest in the middle of the loop is ignored and the script ends with "all presets passed". Arguments let you run one cell (ci/local-ci.sh clang-tsan) while you chase a failure.

The same matrix on a hosted CI service

A hosted service (GitHub Actions, GitLab CI, Buildkite, Jenkins…) adds a clean machine per job, runs on every push and pull request, and blocks merging until the jobs pass. Here is the matrix as a GitHub Actions workflow — an example file; the structure carries over to any service, and the hosted config should stay a thin wrapper around the presets so it never drifts from the local script:

# .github/workflows/ci.yml
name: ci
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-24.04
    strategy:
      fail-fast: false            # let every cell finish: the pattern of failures is information
      matrix:
        compiler: [gcc, clang]
        config: [debug, asan, tsan, release]
    steps:
      - uses: actions/checkout@v4 # pin third-party actions to a full commit SHA in real projects
        with:
          fetch-depth: 0          # full history and tags: git describe needs them
      - name: Install toolchain
        run: sudo apt-get update && sudo apt-get install -y ninja-build
      - name: Configure, build, test
        run: ci/local-ci.sh ${{ matrix.compiler }}-${{ matrix.config }}

Two lines in that file encode lessons people learn the hard way: fail-fast: false (if Clang-TSan fails, you want to know whether GCC-TSan does too — a race shows up in both, a compiler bug in one), and fetch-depth: 0 (CI services default to a shallow clone without tags, so git describe fails or silently falls back, and every CI artifact gets the wrong version).

Also: cache the compiler's work (ccache/sccache keyed on the compiler and flags) for speed, but never cache the build directory between jobs — a stale build tree is how "passes in CI" diverges from "builds from a clean checkout".

2. Warnings: on everywhere, errors only in CI

Turn on a strong warning set for your own targets (-Wall -Wextra -Wpedantic -Wconversion -Wshadow, or /W4 on MSVC), and make CI fail on any warning, so new warnings never land. But don't put -Werror in CMakeLists.txt:

# Don't: every user and packager inherits it, and the next GCC release adds a
# warning that breaks the build of your *released* source.
target_compile_options(splat_core PRIVATE -Wall -Wextra -Werror)

# Do: warnings for your targets, errors decided by whoever runs the build.
target_compile_options(splat_core PRIVATE
    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wall -Wextra -Wpedantic -Wconversion -Wshadow>
    $<$<CXX_COMPILER_ID:MSVC>:/W4>)
# CI: cmake --preset gcc-debug -DCMAKE_COMPILE_WARNING_AS_ERROR=ON

CMAKE_COMPILE_WARNING_AS_ERROR (CMake ≥ 3.24) initialises the COMPILE_WARNING_AS_ERROR property of every target, which adds -Werror or /WX for the compiler in use; cmake --compile-no-warning-as-error overrides it from the command line. Put it in the CI presets ("cacheVariables": {"CMAKE_COMPILE_WARNING_AS_ERROR": "ON"}) or pass it in the CI script, as above.

3. Reproducible builds

A build is reproducible when the same source, built with the same toolchain, produces bit-for-bit the same artifact — on another machine, in another directory, next year. It's how you prove a binary came from a given commit (supply-chain security: the Reproducible Builds project), how caches stay valid, and how you can rebuild last year's release to debug it. The compiler is deterministic; the nondeterminism leaks in from the environment:

Timestamps. __DATE__ and __TIME__ expand to the moment of compilation, so a --version line that prints them changes every build. Don't embed the build time; if you need a date, use the commit's date via the SOURCE_DATE_EPOCH convention: CI exports SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct), GCC and Clang use it for __DATE__/__TIME__, and many tools (tar, CPack's archivers, Python's packaging) honour it too. -Werror=date-time stops new uses. Your own code can read it the same way:

#include <chrono>
#include <cstdlib>
#include <format>
#include <print>
#include <string>

// The "build date" printed by --version: the commit time from SOURCE_DATE_EPOCH
// (set by CI), never the wall clock, so two builds of a commit print the same thing.
std::string build_date() {
    const char* env = std::getenv("SOURCE_DATE_EPOCH");
    long long secs = env ? std::stoll(env) : 1'760'000'000;  // fixed fallback, not now()
    auto tp = std::chrono::sys_seconds{std::chrono::seconds{secs}};
    return std::format("{:%Y-%m-%d}", tp);
}

int main() {
    std::println("splat-render 1.4.3-dev.3+gdeadbee (built {})", build_date());
}

Paths. Absolute source paths get into binaries through __FILE__ (assert messages), std::source_location and debug info, so the same commit built in /home/you/src and /builds/runner-7/src differs. Run this and look at where the path in the output comes from:

#include <cassert>
#include <print>
#include <source_location>

void log_here(std::source_location loc = std::source_location::current()) {
    std::println("{}:{}  <- this string is stored in the binary", loc.file_name(), loc.line());
}

int main() {
    log_here();
    std::println("__FILE__ = {}", __FILE__);
    std::println("compiler = {}", __VERSION__);  // also part of the build's inputs: pin it
}

-ffile-prefix-map=OLD=NEW (GCC ≥ 8, Clang ≥ 10) rewrites the prefix in both __FILE__-style macros and debug info:

add_compile_options("-ffile-prefix-map=${CMAKE_SOURCE_DIR}=.")

Ordering. If the list of sources (or of object files given to the linker or ar) comes from a directory listing, its order depends on the filesystem, and so does the symbol order in the output. List sources explicitly in target_sources; CMake's file(GLOB) does sort its results, but it also silently picks up stray files and doesn't notice new ones without a re-configure. The same applies to anything you generate: iterate a sorted container, not an unordered_map, when writing a file.

Toolchain. "GCC" isn't an input; "GCC 16.2.1 from container image …@sha256:4f1c…" is. Pin the compiler, CMake and dependency versions (a container image referenced by digest, a toolchain file in the preset, exact FetchContent tags or commit hashes rather than branches).

Checking it. Build twice, in two different directories, and compare sha256sum of the outputs. When they differ, diffoscope a b shows where (a timestamp in a .comment section, a path in DWARF…).

Clean-checkout build instructions

The other half of reproducibility is human: a new teammate (or an interviewer looking at your repository) must get from git clone to a passing test run with commands written in the README — and those commands must be tested, by CI, from a fresh clone:

git clone https://example.com/you/scene-viewer.git && cd scene-viewer
cmake --preset release        # needs: CMake ≥ 3.25, GCC ≥ 13 or Clang ≥ 17, Ninja
cmake --build --preset release
ctest --preset release

List the required tools and versions, and nothing else: no "first install these headers into /usr/local", no step that only works because your machine happens to have something lying around. If it isn't in the README and isn't fetched by CMake, a clean-checkout build in CI will tell you.

4. Versions: SemVer and git describe

Every artifact needs a version that answers two questions: what is it? (which commit, which changes) and is it newer than what I have? (ordering). Semantic Versioning 2.0.0 is the common contract for libraries and tools:

MAJOR.MINOR.PATCH[-PRERELEASE][+BUILD]      e.g.  2.0.0-rc.1+gabc1234

• MAJOR — incompatible API changes (removed or changed public functions, changed behaviour callers rely on). Reset MINOR and PATCH to 0.
• MINOR — new functionality, backwards compatible. Reset PATCH to 0.
• PATCH — backwards-compatible bug fixes.
• 0.y.z is initial development: anything may change at any time.
• A pre-release (-alpha, -rc.1, -dev.3) sorts before the release of the same core: it's a version on its way to that release.
• Build metadata (+gabc1234, +dirty) identifies a build but is ignored for ordering: 1.0.0+a and 1.0.0+b are the same version for precedence.

Precedence compares the core numbers numerically, then the pre-release identifiers left to right — numeric ones as numbers, alphanumeric ones in ASCII order, numeric below alphanumeric — and a longer list wins when one is a prefix of the other:

The classic bug is comparing versions as strings. Because valid numeric identifiers have no leading zeros, "compare lengths, then characters" gives numeric order with no integer conversion at all — it works for identifiers of any length:

#include <print>
#include <string>

bool numeric_less(const std::string& a, const std::string& b) {
    if (a.size() != b.size()) return a.size() < b.size();  // no leading zeros: longer = bigger
    return a < b;
}

int main() {
    std::string two = "2", eleven = "11", huge = "123456789012345678901234567890";
    std::println("as strings:  2 < 11? {}", two < eleven);                       // false!
    std::println("as numbers:  2 < 11? {}", numeric_less(two, eleven));          // true
    std::println("huge > 11?         {}", numeric_less(eleven, huge));           // true, no overflow
}

Versions between releases: git describe

A tag marks a release; most builds are between tags. git describe --tags --long --dirty names any commit relative to the nearest tag:

$ git describe --tags --long --dirty
v1.4.2-3-gdeadbee-dirty
  │    │  │        └─ uncommitted changes in the working tree
  │    │  └────────── g + abbreviated commit hash
  │    └───────────── 3 commits after the tag
  └────────────────── nearest tag

--long matters: without it, a build exactly on the tag prints only v1.4.2, and a script that parses fields breaks (or, worse, guesses). The build task converts this into a SemVer version that sorts correctly: 3 commits after 1.4.2 is on its way to 1.4.3, so it becomes 1.4.3-dev.3+gdeadbee.dirty — greater than 1.4.2, less than 1.4.3, with the hash and dirtiness kept as build metadata. A clean checkout of the tag (distance 0) is the release itself: 1.4.2.

Getting the version into the binary

CMake's project(VERSION …) accepts only numeric components (1.4.2), so keep the release number there and compute the full string at configure time, with a fallback for source tarballs that have no .git:

project(splat VERSION 1.4.2 LANGUAGES CXX)

set(SPLAT_VERSION_FULL "${PROJECT_VERSION}")       # fallback: no git (source tarball)
find_package(Git QUIET)
if(GIT_FOUND AND EXISTS "${CMAKE_SOURCE_DIR}/.git")
  execute_process(COMMAND "${GIT_EXECUTABLE}" describe --tags --long --dirty
                  WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"
                  OUTPUT_VARIABLE SPLAT_GIT_DESCRIBE OUTPUT_STRIP_TRAILING_WHITESPACE
                  RESULT_VARIABLE git_rc ERROR_QUIET)
  # convert SPLAT_GIT_DESCRIBE with the rule from the build task (a small CMake
  # or Python helper) into SPLAT_VERSION_FULL when git_rc is 0
endif()

configure_file(version.hpp.in "${CMAKE_CURRENT_BINARY_DIR}/generated/splat/version.hpp" @ONLY)
target_include_directories(splat_core PUBLIC
    "$<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/generated>")

// version.hpp.in — configure_file replaces the @VAR@ placeholders
#pragma once
#include <cstdint>
#include <string_view>
namespace splat {
inline constexpr std::uint64_t version_major = @PROJECT_VERSION_MAJOR@;
inline constexpr std::uint64_t version_minor = @PROJECT_VERSION_MINOR@;
inline constexpr std::uint64_t version_patch = @PROJECT_VERSION_PATCH@;
inline constexpr std::string_view version_string = "@SPLAT_VERSION_FULL@";
}

Generate numbers for code that compares versions and a string for humans — comparing the string is the lexical-order bug again, at compile time:

#include <cstdint>
#include <print>
#include <string_view>

// What configure_file produced (for a build 3 commits after v1.10.0):
namespace splat {
struct version_t {
    std::uint64_t major, minor, patch;
    auto operator<=>(const version_t&) const = default;  // member-wise, numeric
};
inline constexpr version_t version{1, 10, 1};
inline constexpr std::string_view version_string = "1.10.1-dev.3+gdeadbee";
}  // namespace splat

static_assert(splat::version >= splat::version_t{1, 9, 0}, "needs splat::core 1.9+");

int main() {
    std::println("splat-render {}", splat::version_string);
    std::println("numeric 1.10.1 >= 1.9.0: {}", splat::version >= splat::version_t{1, 9, 0});
    std::println("string \"1.10.1\" >= \"1.9.0\": {}", std::string_view("1.10.1") >= "1.9.0");
}

One trap: execute_process runs at configure time. After a new commit, an incremental build doesn't re-run it, so the embedded hash is stale. CI always configures from scratch, so release artifacts are right; for local builds either accept it or move the step into a build-time custom command that rewrites the header only when the output changes.

Shared libraries have a second version: the ABI version, set with VERSION/SOVERSION target properties (libsplat_core.so.1.4.2, soname libsplat_core.so.1). Bump SOVERSION with MAJOR — whenever the ABI breaks.

5. Install rules, RPATH and a release tarball

A build tree is not an artifact: it's full of object files and absolute paths. install() rules describe what a user gets, and CPack packs exactly that:

include(GNUInstallDirs)                      # bin, lib or lib64, include, share/doc …

set_target_properties(splat_core PROPERTIES
    VERSION ${PROJECT_VERSION} SOVERSION ${PROJECT_VERSION_MAJOR})

# Installed binaries find their libraries relative to themselves.
set(CMAKE_INSTALL_RPATH "$ORIGIN/../${CMAKE_INSTALL_LIBDIR}")

install(TARGETS splat-render splat_core
        RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
        LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
        ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR})
install(FILES README.md LICENSE DESTINATION ${CMAKE_INSTALL_DOCDIR})

set(CPACK_GENERATOR TGZ)
set(CPACK_PACKAGE_VERSION "${SPLAT_VERSION_FULL}")
include(CPack)                               # last: it reads the CPACK_* variables

How the dynamic loader (ld.so) finds a library, in order: the binary's DT_RPATH (the old-style entry, only when there is no DT_RUNPATH), then LD_LIBRARY_PATH, then DT_RUNPATH (what modern linkers write for -Wl,-rpath), then the system cache (/etc/ld.so.cache) and the default directories. $ORIGIN expands to the directory of the binary being loaded, which is what makes the tree relocatable. In the build tree CMake sets an absolute RPATH to the build directories so you can run tests in place; at install it rewrites it to CMAKE_INSTALL_RPATH.

Stage, pack and — the step people skip — run it like a user would:

cmake --install build/release --prefix "$PWD/stage"   # what will be in the package
readelf -d stage/bin/splat-render | grep RUNPATH        # [$ORIGIN/../lib]
ldd stage/bin/splat-render | grep splat                 # → …/stage/bin/../lib/libsplat_core.so.1
(cd build/release && cpack -G TGZ)                      # splat-1.4.2-Linux.tar.gz
# smoke test in a clean container: no build tree, no compiler, no LD_LIBRARY_PATH
docker run --rm -v "$PWD/build/release:/pkg:ro" ubuntu:24.04 sh -c \
  'cd /tmp && tar xf /pkg/splat-*.tar.gz && ./splat-*/bin/splat-render --version'

A tarball that only runs on the machine that built it fails exactly here, with error while loading shared libraries: libsplat_core.so.1: cannot open shared object file. Also check what the binary needs from the system (objdump -T bin/splat-render | grep GLIBC_ lists the glibc symbol versions): a binary built on a newer distro may not start on an older one, which is why release builds often run in the oldest environment you support.

6. The release checklist and the changelog

Releases go wrong in boring ways — a tag on the wrong commit, an artifact built from a dirty tree, a changelog that says 1.5.0 in a tarball named 1.4.9 — so write the steps down and follow them every time (ideally, script them):

1. Main is green on the whole matrix; no open release-blocking issues.
2. Decide the version from the changes (removed/changed API → MAJOR; new features → MINOR; only fixes → PATCH).
3. Update the changelog and anything else that states the version (project(VERSION …)), in one commit.
4. Create an annotated tag on that commit: git tag -a v1.5.0 -m "…".
5. Build the artifact from the tag in a clean checkout with the release preset; run the tests; check --version says exactly 1.5.0.
6. Install, pack, unpack in a clean environment, smoke-test.
7. Push the tag; publish the artifact with its SHA-256 and the changelog section as release notes.
8. Never move or delete a published tag and never re-upload a different artifact under the same version — users and caches have already seen it. Fixes go into 1.5.1.

A changelog is for humans: what changed for users, newest first, not a dump of commit messages. The Keep a Changelog convention:

# Changelog

## [Unreleased]
### Fixed
- `splat-render` no longer crashes on PLY files with zero splats.

## [1.5.0] - 2026-10-02
### Added
- `load_ply_streaming()` loads large scenes without reading the whole file.
### Deprecated
- `load_ply_legacy()`; it will be removed in 2.0.0.

## [1.4.2] - 2026-09-14
### Fixed
- Wrong alpha blending order for overlapping splats with equal depth.

Deprecate before you remove: a MINOR release marks the function ([[deprecated("use load_ply_streaming")]]) and the changelog says when it goes away, and only the next MAJOR release removes it.

7. When CI fails

Interviewers like this one because it shows whether you debug or retry:

• Fails in one cell only. The cell is the clue. Only ASan/TSan: a real memory or concurrency bug that the other builds don't detect — reproduce it with that preset locally (ci/local-ci.sh clang-tsan), see cpp-debugging-failures. Only Release: undefined behaviour the optimiser exploited, or an assert with a side effect that vanished under NDEBUG. Only one compiler: a portability issue (a missing #include that another standard library includes transitively, an extension, a real compiler bug — rarely).
• Fails in CI, passes locally. Something differs: compiler version, a clean versus incremental build directory, a shallow clone without tags, missing files that are untracked on your machine, the number of CPUs, the locale or time zone, a test that depends on the current directory. Run the failing job's exact commands in a clean container of the CI image.
• Fails sometimes. A flaky test is a bug — usually a race or a timing assumption. Reproduce it with ctest --repeat until-fail:100 -R name under TSan, fix it, and don't add automatic retries that hide the next one.

In the project

Ungraded practice in projects/scene-viewer/ (stage 3); the evidence is your own build, test and artifact output.

• Read and run ci/local-ci.sh (the matrix: GCC and Clang with warnings as errors, ASan, TSan, release, then a package that is unpacked and run) and ci/clean-checkout.sh. Compare ci/github-actions.example.yml with the script: why is the hosted config a thin layer over the same presets?
• cmake/Version.cmake embeds git describe (tags scene-viewer-vX.Y.Z). Tag a commit, rebuild, check splat-render --version, then change the module to follow your build task's SemVer rule (-dev.N, .dirty).
• Build release twice in two directories with SPLAT_VERSION_OVERRIDE set and compare sha256sum of the binaries; make them identical (-ffile-prefix-map, SOURCE_DATE_EPOCH, no __DATE__).
• Try -DBUILD_SHARED_LIBS=ON: install to a prefix, see the binary fail to find libsplat_core.so, and fix it with an $ORIGIN/../lib install RPATH (readelf -d shows the RUNPATH).
• Write your release checklist into the project README.