Linux I/O, IPC and process lifecycle

For a step-by-step systems sequence in C++, choose Unix & Linux systems. It starts with descriptors and open file descriptions, then builds through file I/O, processes, mapped memory, signals, epoll, TCP and descriptor passing. This overview connects those mechanisms to application/service lifecycle.

A Linux application or service is mostly plumbing around its core: it reads and writes file descriptors, talks to other processes over pipes and Unix sockets, survives peers that are slow, gone or hostile, shuts down cleanly on signals, starts and reaps child processes, and runs under systemd. None of this is exotic, and interviews for Qt/Linux, embedded Linux and rendering-backend roles test it directly: "what does read return?", "why did my process die with no error message?" (SIGPIPE), "how do you shut down cleanly on SIGTERM?", "what is a zombie?"

This lesson builds on raii-scope (a descriptor is a resource like any other) and cpp-project-build (the code lives in a library with tests). The build task implements the bottom layer of a local IPC link: an owning descriptor type, full-transfer loops with deadlines, and a frame parser.

1. A file descriptor is a resource: unique_fd

A descriptor is a small int indexing the process's descriptor table; each entry points at a kernel open file description (file offset, flags), which points at the file, pipe or socket itself. fork copies the table, so parent and child share open file descriptions; exec keeps every descriptor that is not marked close-on-exec.

Three rules follow, and an owning wrapper enforces all of them:

• Close exactly once. Descriptor numbers are reused immediately (the lowest free number is handed out next), so a second close doesn't fail harmlessly — it closes somebody else's file. Make the type move-only.
• Close-on-exec by default. Create descriptors with O_CLOEXEC, SOCK_CLOEXEC, pipe2(…, O_CLOEXEC), accept4(…, SOCK_CLOEXEC). A leaked pipe write end in a child means the reader never sees end of file.
• Never retry close on EINTR. On Linux the descriptor is released before close can be interrupted; retrying races with another thread that just received the same number.

#include <fcntl.h>
#include <unistd.h>
#include <print>
#include <utility>

class unique_fd {
public:
    unique_fd() = default;
    explicit unique_fd(int fd) noexcept : fd_(fd) {}
    unique_fd(unique_fd&& o) noexcept : fd_(std::exchange(o.fd_, -1)) {}
    unique_fd& operator=(unique_fd&& o) noexcept {
        if (this != &o) reset(o.release());
        return *this;
    }
    ~unique_fd() { reset(); }
    int get() const noexcept { return fd_; }
    int release() noexcept { return std::exchange(fd_, -1); }
    void reset(int fd = -1) noexcept {
        int old = std::exchange(fd_, fd);
        if (old >= 0 && old != fd) ::close(old);   // once; no EINTR retry
    }
private:
    int fd_ = -1;
};

bool is_open(int fd) { return ::fcntl(fd, F_GETFD) != -1; }

int main() {
    int raw[2];
    if (::pipe2(raw, O_CLOEXEC) != 0) return 1;
    unique_fd rd(raw[0]), wr(raw[1]);
    std::println("cloexec: {}", (::fcntl(rd.get(), F_GETFD) & FD_CLOEXEC) != 0);
    {
        unique_fd moved = std::move(wr);
        std::println("after move: source {}, target open {}", wr.get(), is_open(moved.get()));
    }
    std::println("after scope: write end open {}", is_open(raw[1]));
}

std::unique_ptr doesn't fit directly: its "empty" value is nullptr, a descriptor's is -1 (and 0 is a valid descriptor — stdin).

2. Short reads, short writes, and the three return values

read(fd, buf, n) returns:

A stream socket or pipe has no message boundaries: two writes can arrive as one read, one write as three reads. The bytes come out in order, that is all.

#include <sys/socket.h>
#include <unistd.h>
#include <print>
#include <string_view>

int main() {
    int sv[2];
    if (::socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, sv) != 0) return 1;
    for (std::string_view part : {"hello ", "world"}) ::write(sv[0], part.data(), part.size());
    ::close(sv[0]);   // the peer hangs up

    char buf[4];
    for (;;) {
        ssize_t n = ::read(sv[1], buf, sizeof buf);
        if (n > 0) { std::println("got {} bytes: '{}'", n, std::string_view(buf, n)); continue; }
        if (n == 0) { std::println("EOF"); break; }
        std::println("error"); break;
    }
    ::close(sv[1]);
}

Writes are short too: a nonblocking send accepts only what fits in the socket's send buffer and then fails with EAGAIN until the peer reads. Blocking writes can be cut short by a signal. So every "send this buffer" is a loop that advances by what each call actually moved:

#include <fcntl.h>
#include <sys/socket.h>
#include <unistd.h>
#include <cerrno>
#include <print>
#include <vector>

int main() {
    int sv[2];
    if (::socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, sv) != 0) return 1;
    int sndbuf = 64 * 1024;
    ::setsockopt(sv[0], SOL_SOCKET, SO_SNDBUF, &sndbuf, sizeof sndbuf);
    ::fcntl(sv[0], F_SETFL, ::fcntl(sv[0], F_GETFL) | O_NONBLOCK);

    std::vector<char> big(1 << 20, 'x');
    ssize_t n = ::send(sv[0], big.data(), big.size(), MSG_NOSIGNAL);
    std::println("asked 1 MiB, kernel took less: {}", n > 0 && n < static_cast<ssize_t>(big.size()));
    ssize_t again = ::send(sv[0], big.data() + n, big.size() - n, MSG_NOSIGNAL);
    std::println("next send: {} with EAGAIN: {}", again, errno == EAGAIN);
    ::close(sv[0]);
    ::close(sv[1]);
}

Signals and EINTR. A signal arriving while a thread is blocked in read, write or poll makes the call fail with EINTR (for read/write only if the handler was installed without SA_RESTART; poll always). It is not an error: retry — with the remaining timeout.

3. Nonblocking I/O, poll, and deadlines

A call that may block forever is a bug in a service: a peer that stops reading must not freeze your thread. Wait for readiness with poll (or epoll for many descriptors) and give the whole operation one deadline:

const auto deadline = std::chrono::steady_clock::now() + timeout;   // once, before the loop
while (done < size) {
    auto left = std::chrono::ceil<std::chrono::milliseconds>(deadline - std::chrono::steady_clock::now());
    pollfd p{fd, POLLIN, 0};
    int r = ::poll(&p, 1, std::max<int>(0, left.count()));
    if (r < 0 && errno == EINTR) continue;
    if (r == 0) return timeout;
    // read once, advance, or handle 0 / errors
}

• Use steady_clock, never system_clock: wall-clock time jumps (NTP, the user changing the date).
• Round the remaining time up; rounding down turns "0.4 ms left" into a busy loop of poll(…, 0).
• Recomputing deadline inside the loop is a classic bug: a peer that sends one byte just before each wait expires keeps the call alive forever.

Set O_NONBLOCK on descriptors an event loop owns. For a single call on a descriptor you don't own, send(…, MSG_DONTWAIT) makes just that call nonblocking without changing flags that other code shares.

#include <poll.h>
#include <sys/socket.h>
#include <unistd.h>
#include <print>

int main() {
    int sv[2];
    if (::socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, sv) != 0) return 1;
    pollfd p{sv[1], POLLIN, 0};
    std::println("empty socket, poll(0): {}", ::poll(&p, 1, 0));
    ::write(sv[0], "x", 1);
    int ready = ::poll(&p, 1, 0);
    std::println("after a write: {} with POLLIN {}", ready, (p.revents & POLLIN) != 0);
    ::close(sv[0]);
    char c;
    ::read(sv[1], &c, 1);
    p.revents = 0;
    ::poll(&p, 1, 0);
    std::println("peer closed: POLLHUP {}", (p.revents & POLLHUP) != 0);
    ::close(sv[1]);
}

4. Local IPC: pipes, socketpair, Unix domain sockets

Unix sockets come in SOCK_STREAM (bytes, like TCP), SOCK_DGRAM and SOCK_SEQPACKET (message boundaries kept). The address is a filesystem path (permissions apply; remove the file before bind, or bind fails with EADDRINUSE) or, on Linux, a name in the abstract namespace (first byte NUL, no file, gone when the socket closes; ss -x shows it as @name). SO_PEERCRED tells the server the pid/uid of the client — real authentication for local services. SCM_RIGHTS passes descriptors.

#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <cstddef>
#include <cstring>
#include <print>
#include <string>

int main() {
    std::string name = std::string(1, '\0') + "demo-render-" + std::to_string(::getpid());   // abstract
    sockaddr_un addr{};
    addr.sun_family = AF_UNIX;
    std::memcpy(addr.sun_path, name.data(), name.size());
    auto len = static_cast<socklen_t>(offsetof(sockaddr_un, sun_path) + name.size());

    int srv = ::socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
    if (::bind(srv, reinterpret_cast<sockaddr*>(&addr), len) != 0 || ::listen(srv, 4) != 0) return 1;
    int cli = ::socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
    if (::connect(cli, reinterpret_cast<sockaddr*>(&addr), len) != 0) return 1;
    int conn = ::accept4(srv, nullptr, nullptr, SOCK_CLOEXEC);

    ucred who{};
    socklen_t wl = sizeof who;
    ::getsockopt(conn, SOL_SOCKET, SO_PEERCRED, &who, &wl);
    std::println("client pid is ours: {}", who.pid == ::getpid());

    ::send(cli, "render", 6, MSG_NOSIGNAL);
    char buf[16];
    ssize_t n = ::recv(conn, buf, sizeof buf, 0);
    std::println("server got '{}'", std::string(buf, n));
    for (int fd : {conn, cli, srv}) ::close(fd);
}

SIGPIPE. Writing to a pipe or socket whose reader is gone sends the writer SIGPIPE, whose default action is to terminate the process — silently, no exception, no log line, exit status "killed by signal 13". A viewer that disconnects at the wrong moment must not kill the render server. Use send(…, MSG_NOSIGNAL) (sockets) and, because libraries write too, signal(SIGPIPE, SIG_IGN) once at startup; then the write fails with EPIPE, which you handle like any other error.

#include <sys/socket.h>
#include <unistd.h>
#include <cerrno>
#include <cstring>
#include <print>

int main() {
    int sv[2];
    if (::socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, sv) != 0) return 1;
    ::close(sv[1]);                                    // the viewer went away
    ssize_t n = ::send(sv[0], "frame", 5, MSG_NOSIGNAL);
    std::println("send returned {}: {}", n, std::strerror(errno));   // still alive
    ::close(sv[0]);
}

5. Message framing

Because a stream has no boundaries, a protocol must put them back. The simplest robust scheme is a length prefix: a fixed-size, fixed-endianness length, then that many bytes. The reader is an incremental parser: it is fed whatever chunks read returns, emits every complete frame, and keeps a partial frame for the next chunk.

Decisions every framing protocol has to make, and interviewers ask about:

• Endianness, explicitly. Decode byte by byte (b0 | b1 << 8 | …): that is correct on every host. memcpy into a uint32_t is only right on little-endian hosts; ntohl is for big-endian ("network order") protocols.
• A maximum size, checked when the header arrives. The length comes from the other side of a trust boundary. Without the check, one corrupt header makes you wait for, and buffer, 4 GiB.
• EOF inside a frame is an error (truncated), distinct from a clean EOF between frames.
• Versioning. Real protocols add a type or version field next to the length so the format can evolve.

#include <cstddef>
#include <cstdint>
#include <print>
#include <vector>

std::uint32_t decode_le(const std::byte* b) {
    std::uint32_t v = 0;
    for (int i = 0; i < 4; ++i) v |= std::to_integer<std::uint32_t>(b[i]) << (8 * i);
    return v;
}

int main() {
    std::vector<std::byte> hdr{std::byte{0x02}, std::byte{0x01}, std::byte{0x00}, std::byte{0x00}};
    std::println("little-endian: {}", decode_le(hdr.data()));
    std::uint32_t be = 0;
    for (std::byte b : hdr) be = (be << 8) | std::to_integer<std::uint32_t>(b);
    std::println("misread as big-endian: {}", be);
}

6. Bounded queues and backpressure

When a producer is faster than a consumer, something has to give, and "an unbounded std::deque that grows until the OOM killer arrives" is not a design. Every queue between components gets a bound and a policy:

• Block the producer (a full socket buffer does this for you: EAGAIN / POLLOUT is backpressure). Right for file transfers and uploads.
• Drop or coalesce: keep only the newest camera pose or render request (latest wins — qt-background-work builds one). Right for state that supersedes itself.
• Disconnect the slow client when its per-connection send queue exceeds N bytes. Right for a server with many viewers: one stuck client must not grow the server's memory.

// Per-connection outgoing queue in an event loop.
bool enqueue(Connection& c, std::vector<std::byte> frame) {
    if (c.queued_bytes + frame.size() > kMaxQueuedBytes) return false;   // caller disconnects c
    c.queued_bytes += frame.size();
    c.out.push_back(std::move(frame));
    want_pollout(c);   // flush when the socket is writable again
    return true;
}

7. Reconnecting: exponential backoff with jitter

A client that loses its server retries — but not in a tight loop (it burns CPU and floods logs), and not on a fixed schedule (when a server restarts, every client reconnects in the same instant: a thundering herd). Use capped exponential backoff with full jitter: wait a random time in [0, min(cap, base · 2^attempt)], and reset attempt after a connection that stayed up.

#include <algorithm>
#include <cstdint>
#include <print>
#include <random>

int main() {
    const std::int64_t base_ms = 100, cap_ms = 5000;
    std::mt19937_64 rng(42);   // seed from std::random_device in real code
    for (int attempt = 0; attempt < 8; ++attempt) {
        std::int64_t ceiling = std::min(cap_ms, base_ms << attempt);
        std::int64_t wait = static_cast<std::int64_t>(rng() % static_cast<std::uint64_t>(ceiling + 1));
        std::println("attempt {}: ceiling {:4} ms, wait {:4} ms", attempt, ceiling, wait);
    }
}

Also close the failed socket every time (a leaked descriptor per attempt exhausts the 1024 soft limit overnight — one of the build task's bugs), and log the first failure and then only occasionally.

8. Signals and clean shutdown

systemctl stop, Ctrl-C and container runtimes send SIGTERM/SIGINT and, after a timeout, SIGKILL. A clean shutdown stops accepting work, finishes or cancels what is in flight, flushes output, closes connections and exits 0. None of that can happen inside a signal handler: a handler interrupts your code at an arbitrary instruction, so it may only call async-signal-safe functions (write, _exit, … — not printf, malloc, std::mutex, logging, or anything that might hold a lock the interrupted code holds). The two standard designs turn the signal into an event your main loop reads:

• Self-pipe: the handler does write(pipe_wr, "x", 1) (saving and restoring errno) and nothing else; the event loop polls pipe_rd.
• signalfd (Linux): block the signals with pthread_sigmask before creating any threads (threads inherit the mask), then read them as data from a descriptor. No handler at all.

#include <poll.h>
#include <signal.h>
#include <sys/signalfd.h>
#include <unistd.h>
#include <print>

int main() {
    sigset_t set;
    sigemptyset(&set);
    sigaddset(&set, SIGTERM);
    sigaddset(&set, SIGINT);
    ::pthread_sigmask(SIG_BLOCK, &set, nullptr);    // first thing, before any thread starts
    int sfd = ::signalfd(-1, &set, SFD_CLOEXEC);

    ::kill(::getpid(), SIGTERM);                    // what `systemctl stop` does
    pollfd p{sfd, POLLIN, 0};
    if (::poll(&p, 1, 1000) == 1) {
        signalfd_siginfo si{};
        ::read(sfd, &si, sizeof si);
        std::println("event loop: got signal {} — stop accepting, drain, exit 0", si.ssi_signo);
    }
    ::close(sfd);
}

Worker threads are stopped the way threads-jthread-stop-token taught: request_stop() and join, with every blocking wait cancellable (a stop token, or a descriptor in the same poll set).

9. Child processes: spawn, wait, reap

fork duplicates the process; the child then execs a program. Between the two, a multithreaded parent's child may only call async-signal-safe functions (other threads' locks were copied in whatever state they were in). posix_spawn does fork+exec in one call (glibc uses clone(CLONE_VM | CLONE_VFORK), so it is also fast for big processes) and is the default choice; file actions set up the child's stdin/stdout.

A child that has exited but not been waited for stays a zombie: its pid and exit status are kept until the parent calls waitpid. Zombies hold no memory, but they hold pids, and a long-running parent that never reaps fills the process table. Decode the status with WIFEXITED/WEXITSTATUS and WIFSIGNALED/WTERMSIG.

#include <signal.h>
#include <spawn.h>
#include <sys/wait.h>
#include <unistd.h>
#include <fstream>
#include <print>
#include <string>

extern char** environ;

pid_t spawn(const char* path, char* const argv[]) {
    pid_t pid = -1;
    return ::posix_spawn(&pid, path, nullptr, nullptr, argv, environ) == 0 ? pid : -1;
}

void report(pid_t pid) {
    int status = 0;
    ::waitpid(pid, &status, 0);
    if (WIFEXITED(status)) std::println("exited with {}", WEXITSTATUS(status));
    else if (WIFSIGNALED(status)) std::println("killed by signal {}", WTERMSIG(status));
}

int main() {
    char* f[] = {const_cast<char*>("false"), nullptr};
    pid_t a = spawn("/usr/bin/false", f);

    siginfo_t info{};
    ::waitid(P_PID, a, &info, WEXITED | WNOWAIT);   // wait for the exit, but don't reap
    std::ifstream stat("/proc/" + std::to_string(a) + "/stat");
    std::string pid, comm, state;
    stat >> pid >> comm >> state;
    std::println("before waitpid the child is in state {} (zombie)", state);
    report(a);

    char* s[] = {const_cast<char*>("sleep"), const_cast<char*>("30"), nullptr};
    pid_t b = spawn("/usr/bin/sleep", s);
    ::kill(b, SIGTERM);
    report(b);
}

In an event loop, reap without blocking: on SIGCHLD (via the signalfd) call waitpid(-1, &st, WNOHANG) in a loop until it returns 0 — signals coalesce, so one SIGCHLD may stand for several exits. Linux's pidfd_open gives a pollable descriptor per child, which avoids the SIGCHLD bookkeeping entirely.

10. Running under systemd

On a Linux box (and in many embedded Linux images) a long-running program is a systemd service. You write a unit file; systemd starts the process, captures stdout/stderr into the journal, restarts it on failure and stops it with SIGTERM.

# /etc/systemd/system/splat-render.service   (or ~/.config/systemd/user/ for a user unit)
[Unit]
Description=Splat render server
After=network.target

[Service]
Type=notify
ExecStart=/opt/splat/bin/splat-render --serve @splat-render
Restart=on-failure
RestartSec=2
TimeoutStopSec=10
# Hardening for a service that writes nowhere but its socket and the journal:
NoNewPrivileges=yes
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

• Type=simple (the default): "started" as soon as the process is forked, even if it then fails to open its socket. Type=exec: once exec succeeded. Type=notify: only when the program sends READY=1 — so units ordered After= it really find it ready.
• sd_notify is a datagram to the AF_UNIX socket named in $NOTIFY_SOCKET (READY=1, STATUS=…, STOPPING=1, WATCHDOG=1 with WatchdogSec=). Link libsystemd, or send the datagram yourself — it is ten lines with the socket code above.
• Restart=on-failure restarts after a non-zero exit code, a crash (SIGSEGV, SIGABRT, …), a start timeout or a missed watchdog — never after a systemctl stop. Still exit 0 on SIGTERM: a non-zero exit there leaves the unit marked failed. StartLimitBurst= stops a crash loop.
• Logs: write plain lines to stdout/stderr; journalctl -u splat-render -f follows them, journalctl -u splat-render -b -p err shows this boot's errors. Don't add your own timestamps — the journal records them.
• After TimeoutStopSec systemd sends SIGKILL: your shutdown path must be bounded (deadlines again).

systemctl daemon-reload && systemctl start splat-render
systemctl status splat-render           # state, main pid, last log lines
journalctl -u splat-render -f           # follow the logs
systemctl kill -s SIGTERM splat-render  # exercise the shutdown path

Installing and running services needs a real systemd and root (or a user session), so it is not part of routine tests: test the parts underneath (signal handling, readiness after the socket is listening, exit codes) as ordinary processes, and try the unit by hand on your machine.

In the project

Ungraded practice in projects/scene-viewer/; the evidence is your own build, test and terminal output.

• Put your build task's unique_fd, write_all/read_exact and FrameReader into the splat::core library in core/, and port its tests to doctest in tests/ (socketpair + a writer thread in tiny chunks); run them under the asan preset with ctest --output-on-failure.
• Give splat-render in cli/ a clean shutdown: block SIGTERM/SIGINT at the top of main, watch a signalfd, stop a long render via its stop token and exit 0. Check with kill -TERM mid-render and echo $?.
• Run ls -l /proc/$(pidof splat-render)/fd while it runs: every descriptor should be one you can name, and CLOEXEC set (see /proc/<pid>/fdinfo/<fd> flags).
• Write a user unit for it by hand (systemctl --user), start and stop it, and read the logs with journalctl --user -u. Note the commands you used in docs/ARCHITECTURE.md.

Summary

• Own descriptors with a move-only unique_fd; create them close-on-exec; close once and never retry on EINTR.
• read returns fewer bytes than asked, 0 at end of file, -1 with EINTR/ EAGAIN to retry. Loop and advance by what moved.
• Give operations a deadline computed once; poll with the time left.
• MSG_NOSIGNAL / ignoring SIGPIPE turns a vanished peer into EPIPE.
• Stream sockets have no message boundaries: length-prefix frames, explicit endianness, a maximum checked at the header, truncation detected at EOF.
• Bound every queue; reconnect with capped exponential backoff and jitter.
• Signal handlers only set a flag or write a byte; better, signalfd.
• Spawn with posix_spawn, reap with waitpid, or children become zombies.
• Under systemd: Type=notify + READY=1, Restart=on-failure, exit 0 on SIGTERM, logs to stdout and journalctl -u.