What broke

The setting: one public web host serving a handful of static sites with Caddy, a web server that reads a single config file (the Caddyfile) and can reload it without dropping connections. Each site is one block in that file. The host already served eight; this case is about adding the ninth.

The change was one block appended to the Caddyfile. The pre-flight was the obvious one: caddy validate --config /etc/caddy/Caddyfile, run with sudo. It printed Valid configuration. The reload that followed failed, and the ninth site stayed dark while the other eight stayed green.

How it showed

systemctl reload caddy → Job for caddy.service failed. The journal: open /var/log/caddy/<host>/access.log: permission denied.

The first suspect was the log directory. It was owned by the service user — caddy, the unprivileged account the server runs as — with mode 755; sudo -u caddy touch inside it worked; nsenter into the running process's mount namespace proved the directory was writable from where the server stood. It was.

The file, not the directory, was the problem. ls -l showed -rw------- root root access.log, created seconds earlier — by the validator. caddy validate does not just parse a config; it provisions it: it instantiates every module the config names, and provisioning a file log writer opens the file. Run as root, it left a root-owned, root-only log sitting in a directory the server, running as caddy, could not open. Caddy correctly kept serving the old config, so every existing site stayed up, and the record said Valid configuration.

Where it sits

Every case here is mapped onto the Software Observatory, a catalog — not ours — of "sensors": kinds of checks that can each tell you something true about a piece of software, grouped into families of failure. A case names the families it falls in and the sensors that would have seen it.

This is a change failure caught by nothing at runtime: the gate that guards the change passed for a reason that had nothing to do with whether the change would work, because the gate ran as a different principal than the thing it was guarding.

The observatory's pre-promotion invariant gates sensor is a check that runs on a change before it is promoted to live and asserts some invariant about it. That is what the validator was supposed to be, and the invariant it checks is "this config loads". The sensor was present and blind: an invariant checked from a different identity than production is a different invariant. A smoke test is one real request against the thing just deployed, asking only whether it answers. One request against the new host after the reload would have seen the dark site in under a second, and it is the sensor that was missing entirely. The deploy's definition of done was the validator's exit code, which is a claim about the check, not about the site.

What we did

Deleted the root-owned log, reloaded, and the site came up. Then changed the recipe rather than the memory: validation runs as the service user, sudo -u caddy caddy validate, so the check provisions exactly what production will and any file it leaves behind is one production can open. The deploy note for this host now says so, with the incident behind it, and the next site added to that host was validated that way and came up on the first reload.

What now fails loudly

Two things, and neither is "be careful":

• Validate as the principal that will run it. If the check needs

different privileges than production, that difference is the bug, not a convenience.

• The deploy is done when a request succeeds, not when a check passes.

A verify step curls the new host after reload and fails the deploy on anything but a 200 with the expected body. A dark site is a red deploy, not a green one with a note.

The tell, both times this shape has appeared in this casebook, was a passing gate immediately followed by the failure it existed to prevent. When that happens, suspect the gate before the system.