What broke

The station runs long-lived worker agents under a supervisor, each declared in a config file with a restart policy. restart = "always" restarts a worker when it exits. A process that stays alive but stops working never exits, so it is never restarted.

A worker that hangs — blocked behind a foregrounded server, or a wedged terminal — is alive in every way the supervisor can see and not working in every way that matters. The supervisor has a timeout setting that bounds how long one run may live before it is killed, but that setting was scoped to scheduled, one-shot jobs only; a persistent worker had no timeout at all. Restart guarded against death; timeout did not exist for this worker class; a hang is neither, so neither guard acted on it.

How it showed

A worker wedged while starting a preview server and did not return. Its log ended mid-way through the startup sequence — output up to the point it blocked, then nothing further. The process was still running, so restart = "always" did not fire, and there was no ceiling after which the supervisor would kill a run that had stopped making progress.

What was watching

Resource telemetry — the worker's own log — was on the path and blind. The log recorded the exact line where the process stopped producing output, but a log records what happened, not that output has stopped happening; nothing was checking the log for silence.

Runtime invariants were absent from this path specifically. The supervisor already enforced, for scheduled jobs, the exact check the persistent worker lacked: a bound on how long a run may live before being killed. That invariant existed in the code and was armed only for the scheduled case.

No instrumented sensor fired. A human noticed the worker had gone quiet and killed it by hand; the worker had been live but not working, in production, for an unknown duration before that.

Where it sits

This is a runtime failure: the process was running, and the failure was that the running thing had stopped making progress — a liveness problem no runtime signal was configured to detect. It is also an invariants failure: the rule "a persistent worker either makes progress or is restarted" existed for one class of job and had never been written for another.

The resource telemetry sensor held the evidence and had no alert pointed at it. The runtime invariants sensor is the gap itself: the invariant was written once, for scheduled runs, and not extended to the worker class that wedged.

What we did

The run timeout, previously accepted only on scheduled harnesses, now applies to persistent workers too. Past the ceiling, the process group receives a graceful stop, then a hard kill after the grace period, and the run is recorded as timed out. The restart policy treats that kill the way it treats a natural exit: restart = "always" brings the worker back. The timeout re-arms on every respawn, so a worker that wedges repeatedly is recovered each time rather than going unguarded after its first restart. A new test proves a never-exiting worker with a short timeout is killed, recorded timed out, and respawned.

The fix is filed as a pull request, Supervise wedged persistent workers with a per-harness timeout, and not yet merged, so this case is marked partial.

What now fails loudly

• A wedged worker cannot sit stuck indefinitely. A persistent worker now

has a run-time ceiling; past it the supervisor kills the process and, under restart = "always", respawns it.

• The guard re-arms every lifetime. The timeout restarts with each

respawn, so a worker that wedges repeatedly is killed and brought back repeatedly rather than being unguarded after the first recovery.

• The behavior is pinned by a test. TestPersistentWorkerTimeoutRestarts

runs a never-exiting worker against a short deadline and asserts it is killed, marked timed out, and respawned.