medlab / case

medlab

what got hurt, and what now fails loudly

↑ sublayer

2026-10-11 · fixed: yes

Every exported error occurrence warrants a separate incident task

RuntimeInvariantsObservability EventsRuntime Invariants

now fails loudly regression tests reject historical exports and consumer tool errors, group repeated failures, and enforce a persisted rolling delivery budget

What broke

A telemetry relay converted retained error logs and spans into tasks for an incident worker. Its delivery identity distinguished occurrences. It had no freshness rule, failure grouping, or workload budget. A webhook creation ceiling limited ingress objects but did not limit the tasks they created.

How it showed

The first scan created 25 historical tasks. A later measurement found 38 delivered events: 34 error spans, two error logs, and two validation probes. Fourteen spans described failed telemetry claim calls by the consumer of that same queue. The retained delivery count subsequently reached 59. An earlier queue measurement found 29 pending tasks, one claimed task, and one completed task.

What was watching

The event stream recorded errors and tool-call attributes without distinguishing failed agent actions from runtime failures. Health checks and a validation probe verified transport and worker access. Retry tests verified delivery identity, but permitted another task for the same text at a later timestamp. None bounded work.

A human questioned the count within two hours of activation. The ensuing measurement identified historical exports and consumer tool-call spans. No runtime invariant bounded task creation before deployment.

Where it sits

This is a runtime failure because valid exports overloaded a running queue. It is an invariants failure because actionable work had no enforced bound. Observability events were present and held evidence, but event count was treated as task count. Runtime invariants were absent: export age and task rate had no enforced limits.

What we did

The installed relay rejects exports received more than five minutes ago or with unusable receive timestamps. It excludes structured tool-call records while preserving runtime errors. Similar failures are grouped by service, signal, severity, and normalized summary in fixed 15-minute windows. A persisted rolling budget limits new-policy deliveries to eight per hour. Legacy IDs remain for deduplication and do not consume the new budget. Later occurrences do not update the first task in a group. The consumer cleared the old queue without direct store edits. The replacement was tested and installed atomically. The upstream change is in the telemetry pull request.

What now fails loudly

Nine offline scan-and-state tests cover freshness, grouping, budget persistence, legacy migration, retry identity, dry-run isolation, malformed timestamps, and tool-span and tool-log filtering. The first four fail against the old relay. Receiver and validator integration tests also pass. The installer requires the tested source and runs the offline suite before replacing the executable. Scheduled scans report stale exports, groups, deliveries, and budget deferrals.