What broke

The station runs several worker agents that stay on and take work from a queue. Each worker holds one long-lived connection to that queue, and that connection is how the worker is told a new item has arrived.

When the queue service restarted, every such connection died. To recover, each worker removed the dead connection from the list it keeps of its open sessions, then tried to rebuild it. The worker's routine check visits only the connections still on that list. A connection removed before it was rebuilt was never visited again, so each worker took that step once and had no path back to listening. From that moment the workers were running and deaf: the process was alive, the queue kept filling, and no one was listening on the other end.

The worker supervisor judged health by one question — is the process running? A deaf worker is running. So the supervisor pronounced everything healthy while, for ten days, real work arrived and sat untouched.

How it showed

Silence, not error. The operator's requests went into the queue and stopped there. The operator eventually asked whether anything was broken, and that question also went into the queue unread. No alarm rang, because nothing was checking whether the queue was being heard, only whether the workers were alive.

The workers' own logs carried the exact moment the connection was lost — one line, once, then nothing further — but a log records that something happened, not that something has stopped happening.

What was watching

Resource telemetry — each worker's own log — was present and blind. Each log held the single failed-renewal line from the moment of the restart, and nothing was pointed at those lines to raise an alarm on silence.

The worker supervisor's health check was blind. It measured liveness of the process, not liveness of listening, and the two diverged the moment the connection died.

A weekly health sweep saw the symptom and named it correctly: a backlog of unclaimed items, every prompt to the workers exhausted, "nobody is being rung." But the sweep observes by design and reports in prose; its note went to an inbox that was itself deaf, and nothing acted.

The queue service already exposed the muscle this should have used. It publishes a metric named switchboard_queue_oldest_pending_seconds — the age, in seconds, of the oldest waiting item on each queue — on a dedicated metrics endpoint, alongside a count of items per queue in each state. A metric is a reading; by itself it pages no one. Nothing was scheduled to scrape that endpoint, and nothing compared the age against a limit, so the reading sat unread for the entire incident.

Runtime invariants — the rule "a queue's oldest item stays young" — was present and blind: it existed as a published metric, and no alarm was pointed at it.

Synthetic monitoring was absent. No scheduled check scraped the metric, compared its age to a limit, and raised an alarm — the one step that would have turned the two visible facts, a queue with pending items and an owner that has claimed nothing, into a single "this worker is not listening" verdict.

Where it sits

This is a runtime failure: the process was up, and the failure was that the up process had lost the one connection that made it useful. It is also an invariants failure: the rule "a worker that owns a queue claims from it" is a standing property of the station, and no sensor was written to assert it.

It is the same class as the earlier case about a hung worker that a restart policy never reached: a live process that had stopped working, with every check measuring aliveness and none measuring work. That case fixed the hang; this is the same class on a new surface, the lost connection, and it was not covered by that fix.

What we did

The control is a watcher that scrapes the queue's own metrics endpoint every few minutes and raises an alarm when any queue's switchboard_queue_oldest_pending_seconds passes a short limit — the same reading the service already publishes, now pointed at a pager. On alarm it restarts the affected worker, which is safe because queued items are durable and are not lost on restart.

One code repair is proposed in PR #589. The health check retains the last known channel state after a failed reconnection and retries configured channels on a later tick, even when the dead session has been removed. The PR is pending review and release. Until it is deployed, the workaround is to restart the workers after any queue restart. The queue-upgrade procedure restarts workers as its last step, so a restart of the queue ends with workers freshly connected.

The prevention half for planned upgrades is already in place: the station's own scheduled work — the nightly upgrade, the backup, and the report jobs — now runs under the operating system's scheduler rather than the agents' own scheduler, because an agent's scheduler only fires while that agent's session is alive and listening, which is exactly the failure this case describes. The upgrade job now also restarts every worker and requires each to claim a nudge item to prove it is listening, paging the operator if one stays deaf.

The watcher is now installed as switchboard-watch: a plain script, no model involved, scheduled by the operating system every five minutes. It scrapes the queue's own /metrics endpoint with a dedicated scrape token, and on any queue whose oldest pending item has sat unclaimed past the limit it pages the operator and restarts that queue's worker. The scrape token is set; it takes effect on the next switchboard restart, which the nightly upgrade job already ends by bouncing and re-proving every worker. The code repair remains pending review and release; until it is deployed, restarting the workers after any queue restart remains necessary.

What now fails loudly

• Deafness surfaces in minutes, not days. The watcher scrapes the queue's

own metric every few minutes and pages the moment a queue's oldest item has sat unclaimed past the limit, so a worker that has gone deaf is caught on the next tick instead of when the operator asks what is wrong.

• A restart of the queue ends with workers that listen. The upgrade

procedure restarts the workers as its final step, so the queue and its listeners are brought back together, not separately.

• Scheduled work no longer sits inside the thing that goes deaf. The

station's timed jobs live under the operating system's scheduler now, not the agents' own, so a deaf session cannot silence the very checks that should have noticed it.