A busy worker is assumed to be making progress
RuntimeInvariantsResource TelemetryRuntime InvariantsSynthetic Monitoring
https://medlab.sublayer.space/c/busy-worker-assumed-progressing/
now fails loudly a liveness check declares a worker wedged when its message store stops advancing while it is running and busy, then restarts it and pages the operator
What broke
A station worker — a long-running agent that takes one job at a time from a queue — went looking for a piece of text by running a recursive search over its entire home directory, and piped the results through a filter that stops after the first few matches. The search string was rare. The pipe stayed empty, so the filter never closed it, and the search kept reading until it had walked the whole tree, which is large.
The worker had told its harness to wait for that command to finish. Waiting is the worker's only instruction, so the worker sat blocked inside the one command for just over 23 hours.
The checks that watched the worker judged health from three readings: was the process alive, was it waiting for input, and was it using CPU. A blocked worker is alive. It is not waiting for input, because it is in the middle of a call. And it is using CPU, because the search runs for its whole life. All three readings read as healthy, so the worker went unwatched for the full 23 hours.
How it showed
Silence, not error. The worker's message log stopped advancing at 23:19 on 2026-10-08. The next entry, at 22:34 on 2026-10-09, was the blocked call being cancelled by a person. The gap between the two is 23 hours.
The last entry before the gap carried the exact place the worker stood: a call that asked to wait for a shell command, with the command's own description reading "search the home directory for" followed by the search term. The entry names the command; only the silence after it shows that something stopped.
What was watching
Resource telemetry — process state, CPU, and the activity on the worker's database — was present and blind. All of it kept moving during the block, because the search was genuinely running. Resource telemetry measures "doing something," not "finishing something."
Runtime invariants — the rule "a worker that owns a queue claims from it," already written into the station's liveness check after the earlier deaf-worker failure — was present and blind. That check fires only when a worker is idle and waiting, which describes a worker that never started its call, not a worker blocked in the middle of one. The blocked worker reported "not waiting," so the rule never matched.
Synthetic monitoring was absent. No scheduled check read the worker's own message store and compared the age of its newest message against a limit. That one reading — "the newest message is 23 hours old" — was the fact that contradicted the assumed health, and nothing was scheduled to take it.
A person noticed the worker had gone silent and cancelled the blocked call, which is what finally freed it after 23 hours.
Where it sits
This is a runtime failure: the process was up, and the failure was that it had stopped inside one unbounded call. It is also an invariants failure: "a worker advances its message log" is a standing property of the station, and no sensor pointed at that property until this case.
It is the sibling of the earlier worker case, where a live worker had gone deaf. Both are the same class: a live process that stopped working while every check measured whether it was alive. The deaf fix asked "is it idle and waiting?" This failure needed the question "has it moved at all?"
What we did
The detection is in place. The station's liveness check now reads each worker's own message store and declares the worker wedged when the newest message is older than 60 minutes while the worker is running, has pending work, and is not waiting for input. On that verdict it restarts the worker and pages the operator. A worker blocked inside one tool call is caught on the next check, not when a person asks what went wrong.
What is still missing is the capture the operator asked for. When a worker is declared wedged, the check should record what the worker was doing before the restart erases it. Today the check writes the child command lines, truncated, into its own verdict line, which is lost when the run ends. The capture should instead be a durable snapshot, written before the restart, holding four things: the in-flight tool call from the worker's own message store (its name, its arguments, and its shell id), the full command lines of the worker's child processes, each child's elapsed and CPU time, and whether each child is still running. The snapshot should land as one timestamped file per wedged event in the station's telemetry state directory, the same directory the station's log sink already writes its rotating files into, named by worker and timestamp so it survives the run.
What now fails loudly
- A wedged worker surfaces in minutes. The liveness check declares a worker
wedged when its message store has not advanced for 60 minutes while it is running and busy, then restarts it and pages the operator. This would have caught the 23-hour block at its first hour.
What is not yet built is the capture described in "What we did": today the check writes truncated child command lines into a verdict line that is lost when the run ends, so a wedged worker is still restarted without a durable record of what it was doing. That snapshot, plus a per-tool-call timeout so one unbounded command cannot hold a worker for a day, are named follow-ups.