Fault detection and recovery under an RTOS

Firmware that runs for months will meet faults its author never saw on the bench: a sensor that stops answering, a bus that locks up, a deadlock that needs a once-a-week interleaving, a stack that overflows on the one input that takes the deepest path. Firmware interviews ask the same question in many forms — what happens when a task gets stuck, and how does the system notice and recover? — and the first weeks of a firmware job often include reading the code that answers it: the watchdog, the fault handler, the reset-reason log.

This lesson builds the answer in layers, each catching what the layer before it can't:

Every snippet runs under FreeRTOS on the emulated Cortex-M4 (mps2-an386, tick = 1 ms). Timing here is emulated: qemu runs with -icount, so every tick count below is deterministic and good for reasoning about schedules — but it does not establish physical deadlines, cache or flash effects, or real interrupt latency, and running here is not hardware-in-the-loop testing. The last section says what changes on a real board.

1. Timeouts on every wait

portMAX_DELAY means "block forever" (this configuration has INCLUDE_vTaskSuspend on). A task that waits forever for a reply from a hung driver hangs with it — and from the outside, a task blocked forever looks exactly like a task with nothing to do. So in fault-tolerant code every wait gets a bounded timeout, and every return value is checked: xQueueReceive, xSemaphoreTake and ulTaskNotifyTake all have a normal "it didn't happen" outcome.

What to do on a timeout is a design decision, made per wait: retry, use a safe default (the last good value, a "sensor fault" flag), report the failure upwards, or give up and let a supervisor deal with it.

#include "FreeRTOS.h"
#include "queue.h"
#include "task.h"
#include <cstdio>
#include <cstdlib>

QueueHandle_t requests, replies;

// A driver task that answers requests — and hangs on the third one.
void driver(void*) {
    int req;
    for (int served = 0;; ++served) {
        xQueueReceive(requests, &req, portMAX_DELAY);
        if (served == 2) for (;;) vTaskDelay(10000);   // stuck: a bus that never answers
        vTaskDelay(5);                                  // the work
        int reply = req * 10;
        xQueueSend(replies, &reply, 0);
    }
}

void client(void*) {
    TickType_t wake = xTaskGetTickCount();
    for (int req = 1; req <= 4; ++req) {
        xQueueSend(requests, &req, pdMS_TO_TICKS(10));
        int reply;
        if (xQueueReceive(replies, &reply, pdMS_TO_TICKS(20)) == pdTRUE)
            std::printf("request %d: reply %d at tick %lu\n", req, reply, (unsigned long)xTaskGetTickCount());
        else
            std::printf("request %d: no reply by tick %lu, using the safe default\n", req,
                        (unsigned long)xTaskGetTickCount());
        xTaskDelayUntil(&wake, pdMS_TO_TICKS(50));
    }
    std::exit(0);
}

int main() {
    requests = xQueueCreate(2, sizeof(int));
    replies = xQueueCreate(2, sizeof(int));
    xTaskCreate(driver, "driver", 512, nullptr, 3, nullptr);
    xTaskCreate(client, "client", 512, nullptr, 2, nullptr);
    vTaskStartScheduler();
}

The client keeps its 50-tick rhythm whatever the driver does: requests 3 and 4 fail exactly 20 ticks after they were sent (ticks 120 and 170), and the client carries on with a safe default. The driver is still stuck — the client can't fix it; it can only avoid being dragged down with it, and report. Noticing that the driver is stuck is the next layer's job.

Two related habits:

• Pick timeouts from the requirement, not by feel. If the reply should take 5 ticks, 20 is generous; if the control loop needs a value every 10 ticks, a 1000-tick timeout is a hang in disguise.
• Ticks wrap. A 32-bit tick counter at 1 kHz wraps after about 49.7 days. Compare durations (now - start > timeout) with unsigned arithmetic, never absolute ticks (now > deadline):

#include <cstdint>
#include <cstdio>
#include <initializer_list>

int main() {
    std::uint32_t last_beat = 0xFFFFFFF0u;     // 16 ticks before the counter wraps
    std::uint32_t deadline = last_beat + 25;   // wraps to 0x00000009
    for (std::uint32_t now : {0xFFFFFFF8u, 0x00000010u}) {
        std::printf("now = 0x%08lx: elapsed %lu, late by duration: %s, late by comparing ticks: %s\n",
                    (unsigned long)now, (unsigned long)(now - last_beat), now - last_beat > 25 ? "yes" : "no",
                    now > deadline ? "yes" : "no");
    }
}

The subtraction gives 8 and then 32 elapsed ticks, right on both sides of the wrap. Comparing absolute ticks calls the worker late only 8 ticks after its beat, because the deadline itself wrapped.

2. Watchdogs and heartbeats

A hardware watchdog is a counter, clocked independently of the code, that resets the chip when it runs out. Software "kicks" (refreshes) it to start the count again. On many microcontrollers it runs from its own oscillator and can't be stopped by software once started, so it still works when the CPU is stuck in a loop with interrupts masked. Some are windowed: kicking too early is also a fault, which catches code that kicks in a tight loop.

The hardware watchdog only knows that something kicked it. What a kick proves depends entirely on where the kick is:

The last row is the pattern to know. Each monitored task sends a heartbeat — "I completed a loop" — and has a deadline: the longest it may legitimately go between beats (its loop period plus its worst blocking time, with margin). A software watchdog — the supervisor task — checks every heartbeat each period, and only if all are fresh does it kick the hardware watchdog. If one is late, the supervisor knows which task failed and can act on just that one, long before the hardware watchdog's coarse timeout.

Choosing the numbers: the deadline must be longer than the slowest legitimate gap between beats, or the supervisor restarts healthy tasks; the hardware watchdog's timeout must be longer than the supervisor's period plus its own worst-case delay, or it resets a healthy system. An off-by-one in either comparison is a classic source of "random" restarts in the field.

The supervisor must also outrank what it supervises. A worker spinning at a priority above the supervisor starves it, and then only the hardware watchdog is left. That's fine — it's the case the hardware layer exists for — but put the supervisor high enough that ordinary task faults don't take it down too.

3. Restarting a task

The program below is a whole supervisor in miniature: one worker, a 25-tick deadline, a pass every 5 ticks, and a restart that deletes the stuck instance and creates a fresh one:

#include "FreeRTOS.h"
#include "task.h"
#include <atomic>
#include <cstdint>
#include <cstdio>
#include <cstdlib>

std::atomic<TickType_t> last_beat{0};
TaskHandle_t worker = nullptr;
int instance = 0;

void worker_fn(void* arg) {
    const int me = static_cast<int>(reinterpret_cast<std::intptr_t>(arg));
    for (int i = 0;; ++i) {
        if (me == 1 && i == 3)
            for (;;) vTaskDelay(1000);   // instance 1 hangs after three beats
        last_beat = xTaskGetTickCount();
        vTaskDelay(10);
    }
}

void start_worker() {
    ++instance;
    last_beat = xTaskGetTickCount();   // a fresh instance gets a full deadline
    xTaskCreate(worker_fn, "worker", 512, reinterpret_cast<void*>(std::intptr_t{instance}), 2, &worker);
}

void supervisor(void*) {
    const TickType_t deadline = 25, period = 5;
    unsigned passes = 0, kicks = 0;
    start_worker();
    TickType_t wake = xTaskGetTickCount();
    while (xTaskGetTickCount() < 200) {
        xTaskDelayUntil(&wake, period);
        ++passes;
        const TickType_t now = xTaskGetTickCount();
        if (now - last_beat > deadline) {
            std::printf("tick %lu: worker %d silent since %lu -> restart\n", (unsigned long)now, instance,
                        (unsigned long)last_beat.load());
            vTaskDelete(worker);   // the stuck instance and its stack are gone
            start_worker();        // fresh state, fresh deadline
        } else {
            ++kicks;               // the only place a hardware watchdog would be refreshed
        }
    }
    std::printf("tick %lu: %u kicks in %u passes, instance %d running\n", (unsigned long)xTaskGetTickCount(), kicks,
                passes, instance);
    std::exit(0);
}

int main() {
    xTaskCreate(supervisor, "supervisor", 512, nullptr, 3, nullptr);
    vTaskStartScheduler();
}

It prints tick 50: worker 1 silent since 20 -> restart and then 39 kicks in 40 passes. Three details carry the design:

• Delete, then recreate. vTaskDelete on another task frees its TCB and stack immediately; the fresh instance starts from its entry function with fresh local state. Suspending the stuck task instead keeps its memory and everything it holds — each "restart" would leak a task.
• A fresh deadline. start_worker stores the restart tick as the last heartbeat. Without it, the next pass sees the old, stale beat and restarts the brand-new instance again.
• No kick on a bad pass. Restarting is recovery, not health.

What deleting doesn't clean up

FreeRTOS frees exactly two things when a task is deleted: its TCB and its stack. It does not release a mutex the task holds (the mutex stays taken by a task that no longer exists, and everyone who takes it next blocks), free memory the task allocated, finish a half-done bus transaction, or restore a peripheral it was reconfiguring. So "restart by delete" is only safe for tasks designed for it: their resources are owned by something that outlives them (the supervisor, a driver object it can reset), and they never hold a lock across a long wait.

The gentler alternative is a cooperative restart: ask the task to exit at a safe point, where it releases what it holds. If the task is asleep in a long delay or a blocked wait, xTaskAbortDelay moves it from Blocked to Ready at once — its vTaskDelay ends early, its xQueueReceive returns pdFALSE as if it had timed out — so it sees the request now rather than after its full sleep:

#include "FreeRTOS.h"
#include "task.h"
#include <atomic>
#include <cstdio>
#include <cstdlib>

std::atomic<bool> stop_requested{false};
TaskHandle_t worker, boss;

void worker_fn(void*) {
    unsigned units = 0;
    while (!stop_requested) {
        ++units;            // one unit of work...
        vTaskDelay(500);    // ...then sleep until the next one
    }
    std::printf("worker: stop seen at tick %lu after %u units; releasing what it holds\n",
                (unsigned long)xTaskGetTickCount(), units);
    xTaskNotifyGive(boss);
    vTaskDelete(nullptr);
}

void boss_fn(void*) {
    vTaskDelay(30);
    stop_requested = true;
    xTaskAbortDelay(worker);                     // end its 500-tick sleep now
    if (ulTaskNotifyTake(pdTRUE, pdMS_TO_TICKS(100)) != 0)
        std::printf("boss: worker exited by tick %lu\n", (unsigned long)xTaskGetTickCount());
    else
        std::printf("boss: worker didn't exit in time: escalate\n");
    std::exit(0);
}

int main() {
    xTaskCreate(worker_fn, "worker", 512, nullptr, 2, &worker);
    xTaskCreate(boss_fn, "boss", 512, nullptr, 3, &boss);
    vTaskStartScheduler();
}

Both lines report tick 30; without xTaskAbortDelay the worker would notice the flag at tick 500. The boss's own wait is bounded: a task that doesn't exit in time is the next layer's problem (delete it anyway, or escalate). Note that xTaskAbortDelay returns pdFAIL if the task isn't Blocked — for example if it's spinning, which is precisely the task that will never look at the flag.

Escalation

Restarting helps against transient faults (a glitch, a rare race). Against a persistent one — a broken sensor, a corrupted configuration — the worker fails again right after every restart, and restarting forever only hides the fault. So the supervisor counts restarts within a window and escalates when restarting isn't working:

A software-requested reset on a Cortex-M is a write to the core's AIRCR register (SCB->AIRCR = 0x05FA0004, which is what CMSIS's NVIC_SystemReset() does), or simply stopping the kicks and letting the hardware watchdog do it. Before either, drive outputs to their fail-safe state and write the fault record (section 5).

4. Stack overflows and configASSERT

A task's stack is a fixed array chosen at xTaskCreate. Too small, and the deepest call path writes past it into whatever comes next — another task's stack, a TCB, a queue — and the failure appears somewhere else, later.

Measure, then budget. FreeRTOS fills each new stack with 0xa5 bytes; uxTaskGetStackHighWaterMark reports how many words have never been touched — the minimum free stack so far:

#include "FreeRTOS.h"
#include "task.h"
#include <cstdio>
#include <cstdlib>

int depth(int n) {
    volatile char frame[64];   // 64 bytes of locals per call
    frame[0] = static_cast<char>(n);
    return n == 0 ? frame[0] : depth(n - 1) + frame[0];
}

void probe(void*) {
    std::printf("stack: 1024 words; least ever free so far: %lu words\n",
                (unsigned long)uxTaskGetStackHighWaterMark(nullptr));
    depth(4);
    std::printf("after depth(4):  %lu words\n", (unsigned long)uxTaskGetStackHighWaterMark(nullptr));
    depth(20);
    std::printf("after depth(20): %lu words\n", (unsigned long)uxTaskGetStackHighWaterMark(nullptr));
    depth(4);
    std::printf("after depth(4) again: %lu words (a high-water mark never goes back up)\n",
                (unsigned long)uxTaskGetStackHighWaterMark(nullptr));
    std::exit(0);
}

int main() {
    xTaskCreate(probe, "probe", 1024, nullptr, 2, nullptr);
    vTaskStartScheduler();
}

The first reading is taken before printf has ever run; the first printf alone used about 150 words (newlib's formatting is deep), which is why the drop to "after depth(4)" is far larger than four 64-byte frames. The mark never goes back up — it's a high-water mark. Budgets come from running the worst-case paths (every error branch, the longest message) and adding a margin; a stack that passes the tests with 3 words to spare fails in the field.

configCHECK_FOR_STACK_OVERFLOW (2 in this course's configuration) adds a check at every context switch:

• Method 1: is the saved stack pointer outside the stack? Cheap, but misses a deep excursion that returned before the switch.
• Method 2: method 1, plus "do the last bytes of the stack still hold the fill pattern?" (16 bytes in this kernel's Cortex-M build). Catches most overflows — but not a large frame that skips past those bytes without writing them.
• Either way the check runs after the overflow, from the context switch. The hook (vApplicationStackOverflowHook) gets the task's handle and name but can't safely recover — memory is already corrupted. Here the hook prints FreeRTOS stack overflow in task "name" and ends the run; in a product it records the name and resets. Hardware help — an MPU guard region below each stack, or the stack-limit registers of ARMv8-M cores — turns the overflow into a fault at the offending instruction instead.

configASSERT is the kernel's own sanity check: a NULL handle, a FromISR call from an interrupt above configMAX_SYSCALL_INTERRUPT_PRIORITY, a corrupted list. This course defines it to print the file and line and end the program. A failed assert means the program's assumptions are already broken, so the right reaction is the same as for any fatal fault: record where it happened, go to the safe state, reset. Compiling asserts out of release builds doesn't remove the bug, only the evidence.

// A production-style assert handler (sketch): record, make safe, reset.
extern "C" void app_assert_failed(const char* file, int line) {
    taskDISABLE_INTERRUPTS();
    fault_record_write(FaultKind::assert_failed, file, line, __builtin_return_address(0));
    outputs_to_safe_state();
    NVIC_SystemReset();   // never returns
}
#define configASSERT(x) do { if (!(x)) app_assert_failed(__FILE__, __LINE__); } while (0)

5. A fault record that survives the reset

After a field reset the question is why. The answer has to survive the reset itself, so it can't live in ordinary variables: the startup code zeroes .bss and re-initialises .data on every boot. The usual answer is a small record in a RAM section the startup code leaves alone — conventionally .noinit, created by the linker script (this course's mps2-an386.ld has none, so this stays a sketch):

struct FaultRecord {
    std::uint32_t magic;   // 0xFA017EC0 when valid
    std::uint32_t kind;    // watchdog, assert, stack overflow, HardFault, supervisor escalation
    std::uint32_t tick;    // xTaskGetTickCount() at the fault
    std::uint32_t pc;      // faulting or calling address
    char task[16];         // pcTaskGetName() of the culprit
    std::uint32_t crc;     // over everything above
};
[[gnu::section(".noinit")]] FaultRecord g_fault;   // not zeroed by the startup code

void report_last_fault_at_boot() {
    if (g_fault.magic == 0xFA017EC0 && crc32(&g_fault, offsetof(FaultRecord, crc)) == g_fault.crc)
        log_to_flash_and_uplink(g_fault);   // slow and wear-limited: do it here, not in the fault handler
    g_fault.magic = 0;                      // consumed: the next reset must not report it again
}

Many MCUs also latch the reset cause in a register (power-on, reset pin, watchdog, software) — read it at boot next to the record. A RAM record survives a warm reset but not power loss; copy it to flash at the next boot if it must outlive that.

6. Fail-safe states

Before a reset — and whenever a supervisor gives up — outputs go to their fail-safe state: the state that is safe if the software can't be trusted. It's chosen per output when the system is designed (a heater off, a motor driver disabled, a valve closed — or held open, if that's what's safe for that valve), and the hardware should default to it too, so that a hung CPU or a reset leaves the actuator safe without any code running. Software's part is small and must not depend on the parts that failed: write the outputs, record the fault, reset. No unbounded waits, and no asking the failed task to finish first.

7. The build task

You'll write the supervisor from section 3 as a reusable component: workers register with a deadline and a factory that creates a fresh instance; check() is one pass (detect late workers in id order, log {worker, tick, reason}, restart or escalate, kick only when all are alive); a task runs it every period with xTaskDelayUntil and stops promptly with xTaskAbortDelay; after max_restarts restarts within window ticks the supervisor escalates once, calls the reset hook and latches. The tests inject workers that block forever or spin at a low priority, and check the exact detection ticks, restart counts and escalation.

On real hardware

• A real tick source. SysTick (or a low-power timer) is clocked by a crystal or an RC oscillator with real tolerance and drift, and interrupt latency varies with what is masked and what the bus is doing. Deadlines need margins measured on the board, not the exact ticks of an emulator.
• A hardware watchdog. Configure it early in boot, give it a timeout longer than the supervisor's worst-case pass, and decide what should happen under the debugger (many parts can freeze the watchdog while the core is halted — otherwise a breakpoint resets the board). Test that it actually bites, for example with a debug command that stops the kicks.
• Reset cause and fault records. Read the MCU's reset-cause register at boot, log it with the .noinit record, and exercise the whole path — a deliberate stack overflow, an assert, a watchdog bite — on the real board.
• Seeing what happened. Tracing tools such as SEGGER SystemView or Percepio Tracealyzer record task switches, blocking and interrupts on the target, which shows a stuck task or a starved priority directly. For timing, toggle a GPIO at the events you care about (heartbeat, supervisor pass, kick) and measure the gaps with a logic analyser or an oscilloscope.
• Stacks. Re-measure high-water marks on the target build, with its real compiler flags and worst-case inputs; where the core has an MPU, a guard region below each task stack catches an overflow at the faulting instruction.