medlab / case

medlab

what got hurt, and what now fails loudly

↑ sublayer

2026-09-12 · fixed: yes

Renaming a repo and updating its topology entry is one operation

ChangeInvariantsPre-Promotion Invariant GatesStatically Checked Invariants

now fails loudly The deploy gate refuses when the topology file and the filesystem disagree about which repos exist; a worker with a schedule attached is not a worker — it is skipped by collect and never rendered as down.

What broke

A station runs a status page built by a script called collect that reads the fleet's state from several sources: the forge API for merged pull requests, the queue for completed work, the supervisor for which worker processes are up, and local repos for architecture decisions. The script has a gate at the end: it refuses to render the page if a repo on disk is not listed in the station's topology file as either live, retired, or foreign. The topology file is a hand-maintained manifest of which repos belong to the station.

One repo had been renamed, from the old name to the publication archive. The rename happened in the forge — the repo itself — but station/ops.md still listed it under the old name. When collect ran at deploy time, it found a repo on disk at ~/agents/repos/the publication archive with architecture decisions in it, looked up the publication archive in the topology file, and found nothing there. The build refused with the publication archive has ADRs but ops.md lists it as neither live, retired nor foreign. The deploy failed and master could not ship.

A second, unrelated defect landed in the same pull request. The supervisor runs scheduled one-shot jobs, called sweeps, that fire on a cron schedule, do work, and stop; between runs they sit at state: stopped by design. collect's worker list read every supervised process through an adapter that treated anything shaped like an agent as a worker — running, stopped, failed, or down. A scheduled sweep sitting at state: stopped between runs was reported the same way a crashed crew member would be. The status page showed a worker down that had never been a worker at all.

How it showed

At 20:34 the deploy step failed with the topology error in the CI output: the publication archive has ADRs but ops.md lists it as neither live, retired nor foreign. The message named both the missing repo and the file that needed to change.

The worker-down display was visible on the status page itself: the crew section showed one process stopped that was actually a scheduled sweep between runs. A reader of the page would see what looked like a crashed worker, with nothing distinguishing it from a real one.

What was watching

The pre-promotion invariant gates sensor — the deploy script's own gate — caught the topology mismatch. It fired in seconds, at deploy time, refusing the build and naming the missing repo and the file to fix.

Nothing was watching the topology file's agreement with the filesystem before that last moment; the check that caught this ran at deploy, not at commit or pull-request time. No gate checked the worker list for a scheduled process. The second defect shipped unseen and showed only on the rendered page.

Where it sits

This is a change failure: renaming a repo is two operations — the rename itself, and updating the topology entry that names it — and the second one was missed. It is also an invariants failure: the invariant that caught the topology mismatch existed only at deploy time, not before the pull request merged, and the invariant "a scheduled process is not a worker" had never been written at all.

The pre-promotion invariant gates sensor is the one that fired, at deploy time. A statically checked invariant — a lint check that diffs the topology file against the filesystem, which needs no running station — would have caught the same mismatch at pull-request time instead.

What we did

Two pull requests, one minute apart. The first added a schedule skip to collect: any supervised process carrying a schedule key is no longer treated as a worker, so a sweep between runs is not rendered as one down. The test harness's worker list was updated to include a sweep entry, and the test asserts it is filtered out. This first pull request merged without the topology fix.

The second added the publication archive to the topology file's live list in place of the old name — one word on one line. The deploy turned green immediately after.

Master was red for the minute between the two merges. In that minute the gate refused a build that would have shipped a status page showing a worker down that was not a worker, and omitting a repo that existed on disk.

What now fails loudly

  • Scheduled processes are not workers. collect skips any process

carrying a schedule key. A sweep between runs is invisible on the page, not rendered as a worker down.

  • The deploy gate refuses a topology mismatch. If a repo exists on disk

and is not in the topology file, the build fails, naming both the repo and the file that is wrong.