Instrument the actual handler and reproduce an incident

The capstone connects the pieces to a real handler. Every event and counter transition is produced around actual execution. The downloadable notes service can run on an owned ephemeral loopback listener, emit structured JSON locally and expose its bounded metric snapshot.

The incident replay injects deterministic timing and a known failing repository. Those are controlled scenario inputs. The emitted status/body/log/counter relations are real; the synthetic durations are not a benchmark.

Preserve the HTTP result. A completed handler with no body write has an implicit200. A body write can fail after a status was committed. A panic before commitment has no final response status; a panic after a503 retains that committed status.

Temporarily recover only to record a bounded panic outcome, then re-panic. Do not serialize the panic value or raw request data. The ordinary JSON wrapper does not promise optional streaming/hijack interface transparency.

package main
import ("fmt";"net/http";"net/http/httptest")
func main(){
handler:=http.HandlerFunc(func(w http.ResponseWriter,r *http.Request){w.Header().Set("X-Request-ID","lab-1");w.WriteHeader(201);_,_=w.Write([]byte("ok"))});rec:=httptest.NewRecorder();handler.ServeHTTP(rec,httptest.NewRequest("POST","/notes",nil));fmt.Println(rec.Code,rec.Header().Get("X-Request-ID"),rec.Body.String())
}

Counters record the observed attempt even if the clock is invalid. Invalid timing remains null in the event and adds no latency sample. A sink failure is reported through a trusted diagnostic callback; a counter alone cannot establish complete event delivery.

The local module includes the notes reference, injected instrumentation, structured writer, service command, deterministic replay, fixtures, starter and tests. Its diagnostic metric endpoint is excluded from application request counters so viewing the snapshot does not inflate the measurement. All telemetry remains local; these are not CodeRigor analytics or learner records.

Run the healthy and induced-failure modes, then connect a specific server-owned ID to its event, route/status counter row and timing distribution. The generic finding reports a symptom and exemplar, not a magically proven root cause. Controlled setup and comparison supply separate evidence.

package main
import ("fmt";"time";"encoding/json")
func main(){
start:=time.Unix(0,0);end:=start.Add(1200*time.Millisecond);event:=map[string]any{"request_id":"lab-4","route":"/notes","status":500,"duration_ms":end.Sub(start).Milliseconds()};data,_:=json.Marshal(event);fmt.Println(string(data))
}

Complete the middleware and run the download’s public, privacy, metric, timing, concurrency and local-service tests. Use the printed loopback URL, reproduce the failure mode, inspect its emitted evidence, then stop the owned server with Ctrl-C.

This finishes a complete small instrumentation lab. Boot.dev’s Logging and Observability in Go course adds the wider logger/storage, Prometheus/Grafana, alerting, profiling and distributed-tracing stack.

Download the complete instrumented service lab: standard-library Go service, actual telemetry replay, original fixtures, starter, tests and run/cleanup instructions.