What broke

One public web host serves a handful of static sites with Caddy, a web server that reads a single config file (the Caddyfile) and can reload it without dropping connections. Each site is one block in that file. The host already served eight; this case is about adding the ninth.

The change was one block appended to the Caddyfile. The pre-flight check was caddy validate --config /etc/caddy/Caddyfile, run with sudo. It printed Valid configuration. The reload that followed failed, and the ninth site stayed dark while the other eight stayed up.

How it showed

systemctl reload caddy returned Job for caddy.service failed. The journal read: open /var/log/caddy/<host>/access.log: permission denied.

The log directory was not the problem: it was owned by caddy, the unprivileged account the server runs as, mode 755; sudo -u caddy touch inside it worked, and nsenter into the running server's mount namespace confirmed the directory was writable from where the server stood.

The file was the problem. ls -l showed -rw------- root root access.log, created seconds earlier — by the validator. caddy validate does not only parse a config; it provisions it: it instantiates every module the config names, and a file log writer is instantiated by opening its file. Run as root, the validator created a root-owned, root-only log file in a directory the server, running as caddy, could not open. Caddy kept serving the old config, so the existing eight sites stayed up, and the validator's record said Valid configuration.

What was watching

The pre-promotion invariant gates sensor — the deploy script's caddy validate step — was on the path and blind. It checked the config as root; production runs the server as caddy. An invariant checked under a different identity than the one that will run in production is a different invariant, and this one passed for a reason unrelated to whether the ninth site would come up.

The smoke tests sensor was absent from the deploy path entirely. One HTTP request against the new host after the reload would have shown the dark site in under a second.

A human caught the failure when systemctl reload caddy returned an error, at the deploy step, before it reached a state where any site was down for a visitor. Latency was seconds: the deploy failed as soon as the reload ran.

Where it sits

This is a change failure that nothing caught at runtime: the gate that guarded the change passed for a reason unrelated to whether the change would work, because the gate ran as a different principal than the thing it was guarding. The deploy's definition of done was the validator's exit code — a claim about the check, not about the site.

This is the first of two cases in this casebook with the same shape: a pre-flight check that is not read-only. See Looking up a DNS name before creating it is harmless, where an existence check wrote a cache entry the way this validator wrote a log file — both created state the real run then tripped over, and both reported success from inside the check that had just caused the problem.

What we did

Deleted the root-owned log file and reloaded; the site came up. Then changed the recipe: validation now runs as the service user, sudo -u caddy caddy validate, so the check provisions exactly what production will, and any file it creates is one production can open. The deploy note for this host records the change, and the next site added to it was validated this way and came up on the first reload.

What now fails loudly

• Validate as the principal that will run it. If a check needs

different privileges than production, that difference is the defect, not a convenience.

• The deploy is done when a request succeeds, not when a check passes.

A verify step requests the new host after reload and fails the deploy on anything but a 200 with the expected body.