What broke

A telemetry relay turns the errors a worker exports into work items for the station pathologist. On 11 October the relay gained two limits: identical errors are grouped into fixed fifteen-minute windows, and deliveries are capped at eight an hour. A failure that outlasts its window moves into the next window and becomes a new work item. The hourly cap bounds how fast the items arrive. It does not bound how many a single failure creates.

How it showed

From 12 October 14:22 to 14 October 03:17 the station's sole model provider endpoint was unreachable to every worker, with both shapes arriving hourly and no gap at any hour boundary inside that span. Each retry exported an error span. The relay grouped those spans by summary shape into fifteen-minute windows and delivered one work item per window per shape. Two shapes repeated: a domain lookup that found no such host, and a lookup that timed out. The pathologist's queue held at least 200 pending work items, the list cap, every one of them from this single outage.

What was watching

Observability events recorded the failures and forwarded each one. The grouping and budget limits were in place and worked as written: they capped each window and each hour. Neither correlated identical events across windows into one incident. No runtime invariant bounded the total work items one root cause may produce. No sensor told a continuing failure apart from a new one.

Where it sits

This is a runtime failure because valid exports from one outage overloaded a running work queue. It is an invariants failure because the number of work items a single root cause may create was unbounded; only its rate was capped. Observability events were present and held the evidence, but they were grouped by clock window, so one thirty-hour outage read as dozens of separate failures. Incident correlation was absent: nothing folded the repeating shapes into a single incident. Runtime invariants were absent for the total and exposed only the hourly rate.

What we did

The pathologist coalesced the pending items by failure shape into one specimen instead of treating each export as a distinct bug. The missing control was specified: coalesce identical failure shapes across the whole outage rather than per window, and emit a recovery signal on the first recurrence after the shape goes quiet. That change, plus the question of whether the single provider endpoint the whole fleet depends on should have a fallback, was handed to the operator to install and to decide.

The coalescing half is installed. What launchd runs is byte-identical to the copy this case was written against, installed at 05:36 on 14 October, and it carries a bounded version of the request: OUTAGE_WINDOW = 21600 beside GROUP_WINDOW = 900 and MAX_PER_HOUR = 8, so a shape coalesces for six hours at a time rather than for the length of an outage. fixed: partial stands for the fallback question, which is still open, not for the coalescing.

What now fails loudly

Identical failure shapes sharing a service, signal, severity, and summary coalesce into one task per outage horizon, with a running occurrence count, and a shape that reappears after a quiet horizon carries a recurrence marker so the end of an outage is loud rather than silent.

Measured against the relay's own scan with a synthetic sink, 74 exports of the two shapes spread hourly across 37 hours:

installed relay, OUTAGE_WINDOW=21600 14 tasks 7 per shape previous relay, OUTAGE_WINDOW=0 74 tasks 37 per shape

The budget was lifted for that measurement so the grouping was what moved; with the shipped eight-an-hour cap the delivery is additionally limited, which is the reason the real outage's 268 exports arrived as a long trickle rather than one burst.

There is no test of this relay in the tree, so the evidence here is the code's horizon and that reproduction, not a suite. Saying so is the point of this section: the coalescing is installed and watched by hand, and nothing yet fails when it is removed.