What broke

The station runs worker agents that stay on and pull one job at a time from a queue. One worker's queue stalled a second time: work sat waiting while every check reported the worker healthy.

The stall survived the change that was meant to end it. The worker had been given a new software build that repaired the way it reconnects to its queue, and then it was restarted. The restart was assumed to hand the piled-up work back to the worker. It did not. The restart recorded an outcome of "replaced," and the step that re-rings a freshly restarted worker is wired to fire only on two other outcomes, so the worker came back up without being told the old work was still there.

The check that watches for this called the worker healthy. It declared a worker "wedged" only when the worker's own message log had gone quiet. This worker was not quiet — it was busy answering the small recovery prods it did receive — so its log kept advancing while its queue stayed full. Five items sat waiting, the oldest eight hours old, and the check's verdict was "ok."

The recovery prod that did reach the worker passed without proving recovery. It asked the worker to show it could hear its queue. The worker took one item, finished it, and left the older ones untouched. The check counted that as recovered.

How it showed

A queue that looked fine. Five waiting items, the oldest more than eight hours old, a worker reporting running and busy, and a health check whose verdict was "ok."

A person eventually noticed the backlog and re-rang the worker by hand with a stronger instruction. The worker answered within seconds, signalled it was holding its job, and began the oldest queued work. That manual re-ring — not any automated step — is what started the drain.

What was watching

Synthetic monitoring — the scheduled liveness check — was present and blind. It had a "wedged" verdict, but the verdict required two things at once: waiting items older than a limit, and a message log that had stopped. This worker's log kept moving because it kept answering prods, so the check never reached "wedged" and reported "ok" while the queue sat full.

Runtime invariants — the rule "a worker draws down its queue" — was present and blind. The recovery prod asked only that the worker show it could hear its queue. A worker can hear its queue, take one item, and finish it while leaving older ones stuck, so the check could not tell "heard us" from "caught up."

Resource telemetry — the harness's running state and the outcome it recorded for the restart — was present and blind. The restart wrote "replaced," and nothing treated that outcome as needing a re-ring; only two other outcomes triggered one.

A person noticed the backlog and re-rang the worker by hand. That was the first thing that fired, eight hours in.

Where it sits

This is a runtime failure: the worker process was up, and the failure was that it was not drawing down its queue. It is also an invariants failure: "a worker that owns a queue drains it" is a standing property of the station, and the checks asserted a weaker one, "a worker can hear its queue."

It is the third specimen of one class. The first was a worker gone deaf. The second was a worker blocked inside one command. This one is a new surface on the same class: a worker that was up, work that was still queued, and checks and recovery steps that all answered normal because none could tell "the worker is busy" from "the worker's queue is draining."

What we did

The recovery instruction now states what recovery means. The prod a restarted worker receives no longer asks merely to answer; it tells the worker to list its queue, take the oldest waiting ordinary item and do its work, and not to report the prod done while older waiting items remain. It requires a heartbeat at the moment of taking a job — a signal that says "I still hold this job" — and between slow steps, with a stop order if the hold is lost.

The worker's own instructions now forbid the class of command that wedged it the first time — a recursive search over the whole filesystem or home directory — and require an immediate heartbeat after taking a job.

The detection is tightened at the source. The liveness check now treats a stalled queue as its own alarm: a queue whose oldest waiting item has aged past the limit can be declared stuck even while the worker's message log is still moving.

The step that re-rings a restarted worker was not changed: it is still wired to the two outcomes named in "What broke," and "replaced" still does not trigger it. What changed is that a missed re-ring no longer passes for health: the stalled-queue alarm fires on the queue's own evidence, so a worker that comes back without being told about the old work is caught by the next check instead of by a person noticing the backlog. Making the trigger fire on every restart outcome is a named follow-up, not something this case can point at.

The stronger prod was then applied to the live stalled queue, and it worked on this incident. Within two minutes of the manual re-ring the worker signalled that it held two tasks, then finished the queued work; the queue reported no waiting items, no expired holds, and a clean verdict. The same worker had skipped the heartbeat on its first recovery pass, so the heartbeat requirement is proven here as a disposition of this run, not yet as a rule that holds everywhere.

What now fails loudly

• A stalled queue alarms even while the worker chatters. The liveness check no

longer waits for the message log to go quiet; a queue whose oldest item is past the limit is stuck, and restarting the worker no longer passes for recovered unless the queue drains.

• A recovery prod fails if it is answered without draining. The prod is not

complete while older waiting items remain, so "the worker heard us" can no longer stand in for "the worker cleared the backlog."

• The heartbeat is instruction, not yet a gate. The prod demands an immediate

heartbeat after taking a job, and the worker has still been observed to finish recovery work without one. Nothing mechanical refuses a completion that skipped it. That enforcement is the remaining piece.