What broke

A queue service loaded its metrics credential at startup. The watcher read the credential file for each scrape. Rotating the file did not change the credential held by the running service.

How it showed

The credential file changed on 2026-10-07 at 21:19 CEST. The first recorded authenticated /metrics HTTP 401 was on 2026-10-08 at 22:32 CEST, 25 hours and 13 minutes later. The operator received a BLIND alert on 2026-10-10 at 16:38 CEST. Queue age was no longer observable through that scrape.

What was watching

switchboard-watch scraped the queue metrics every five minutes and logged recurring 401 responses. It did not restart the service or retry with the same on-disk credential. check-fleet.sh continued to run, but it checked process health and did not measure queue staleness.

The service published oldest-pending metrics, but the scrape authorization failure blocked the sensor from reading them. Synthetic monitoring detected the failed endpoint. Runtime invariants had no assertion that the watcher could read the queue-age signal after a credential rotation.

Where it sits

This is a runtime and invariants failure: the process stayed live while the monitoring credential used by the watcher and service diverged. Synthetic monitoring fired on the HTTP failure. The queue-age invariant existed as a metric but was blind while scrape authentication failed.

What we did

On HTTP 401, the watcher now restarts the queue service once per six-hour cooldown, then retries the authenticated scrape. It pages if the retry still fails. It records a restart cooldown only after the restart succeeds. A successful scrape clears the blind-alert cooldown so a new failure can page immediately.

What now fails loudly

The five-minute watcher reloads the configured token by restarting the service after 401, verifies the same credential with a retry, and preserves the blind alert if the retry fails.