The running service will emit an event and counter transition for each observed request. Before drawing conclusions, check that the supplied snapshot reconciles.
An aggregate can look correct while its route attribution is wrong. This lesson checks each method/route/status-class row, not just the sum. It then identifies one deterministic failing route and request ID to investigate.
Reject dropped/extra/duplicate rows and duplicate event IDs. A snapshot with the right request total but a wrong route is inconsistent evidence, not an almost-correct report.
For 5xx events, choose the route with the highest failure count and use a lexical tie rule. Select a stable request exemplar and the largest available failing duration for that route. Do not invent timing when every matching event lacks it.
package main
import ("fmt")
func main(){
expected:=map[string]int{"GET /healthz 2xx":1,"POST /notes 5xx":2};observed:=map[string]int{"GET /healthz 2xx":1,"POST /notes 5xx":1};fmt.Println(expected["POST /notes 5xx"]==observed["POST /notes 5xx"])
}A request ID connects the log detail to its group-level counter and latency distribution. That narrows investigation. It does not automatically show why a backend failed or prove a causal chain.
The downloadable replay deliberately induces a failing repository and supplies known synthetic timings through the real middleware. Because the fault setup is controlled, you can compare before/after runs. The generic finding still says RootCauseKnown false; external setup evidence is different from inferring a cause from correlation.
This small complete snapshot assumes no intentional log sampling. Real systems can sample/drop events and scrape at different times; their reconciliation windows and completeness claims need a different explicit contract.
package main
import ("fmt")
func main(){
failures:=map[string]int{"/notes":2,"/healthz":0};route:="/notes";fmt.Println(route,failures[route],false)
}Complete the exact reconciliation and test equal totals with wrong dimensions, missing events, duplicate IDs, no failures and unknown timing. Keep the finding’s causal claim honest.
Next we wire these pieces into actual middleware and reproduce the incident in a running local service.
Follow one request across the signals.
These eight exchanges ran through the real notes handler and middleware. Choose a request to connect its safe event, bounded counter group and latency population. Durations were injected deliberately; they are not host benchmarks.
lab-000004500POST /notes1,200 ms · injected timingControlled repository failure1 · The emitted event
Only six approved fields. No raw query, authorization, body or panic value is emitted.
{
"request_id": "lab-000004",
"method": "POST",
"route": "/notes",
"status": 500,
"duration_ms": 1200,
"outcome": "completed"
}2 · Its counter group
POST /notes · 5xx
This row aggregates the selected route/method/status class. The request ID is an event field, never a series label.
3 · Its timing population
All statuses for POST /notes: 4 valid samples, mean 528.75 ms.
| Duration interval | Observations |
|---|---|
| 0–10 ms | 1 |
| > 10 to 50 ms | 1 |
| > 50 to 100 ms | 0 |
| > 100 to 500 ms | 0 |
| > 500 to 1000 ms | 1 |
| > 1000 to 5000 ms | 1 |
| > 5,000 ms | 0 |
P95 bucket upper bound: ≤ 5,000 msThis is a bound at the chosen bucket resolution, not an exact percentile.