What broke

The setting: a handful of static sites on one public web host, each on its own subdomain. Adding a site means creating a DNS record for the new name at the DNS provider, then giving the web server a block for it. This case is the DNS half.

Before creating the record, the obvious check: does the name already exist? A lookup answered NXDOMAIN — the DNS reply that means "no such name" — so the name was free. The A record was created. Public resolvers answered with the address within a minute. Every server-side check passed: curl from the host, the certificate, a public DNS-over-HTTPS query.

Two and a half hours later the person the site was for still could not load it.

How it showed

DNS name does not exist, in their browser, for a name that every public resolver had been answering for two hours. A raw UDP query to their home router — the resolver every device in that house asks first — returned rcode=NXDOMAIN for the new name, and NOERROR for its sibling on the same zone. The same query to 1.1.1.1: the address.

The existence check had gone through that router. Resolvers cache "no such name" answers as well as real ones, for as long as the zone says they may (the negative TTL, here 300 seconds). The router cached the negative answer and held it far past those 300 seconds — consumer resolvers do not honour the SOA minimum the way the RFC assumes. Every verification that followed ran from somewhere else, so every one of them reported success to a user staring at an error.

Where it sits

Every case here is mapped onto the Software Observatory, a catalog — not ours — of "sensors": kinds of checks that can each tell you something true about a piece of software, grouped into families of failure. A case names the families it falls in and the sensors that would have seen it.

A change — a new record — verified from the wrong vantage point, and a runtime failure that existed only on the path the real user took. The observatory's synthetic monitoring sensor is a scripted probe that exercises a system the way a user would, from outside it, rather than from the machine that runs it. It is the one that names this: a probe that runs from where the user stands, not from where the operator is comfortable. The checks that ran were synthetic in form and useless in vantage — each one was a real request from a place with no memory of the mistake.

What we did

Flushed the router's cache, which the person had to do themselves. Then changed the order of operations for every new name since: check for absence against the authoritative servers only — the provider's own nameservers, dig @ns1.<provider>, which answer from the zone rather than from a cache — create the record, verify against a public resolver, and touch the user's resolver last, and only to read what it already thinks, never to find out whether something exists.

What now fails loudly

• Never probe for absence through a path that remembers. A lookup that

misses creates a cache entry wherever it passed through, and that entry outlives the fix. Existence checks go to an authority.

• Verify from the user's vantage, not the operator's. A green check from

the server proves the server can reach itself. The verification that counts is the one that takes the same path the user will.

The general shape, shared with the case before this one: pre-flight checks are not read-only. A validator that provisions creates files as whoever ran it; a lookup that misses creates a cache entry wherever it passed through. Both write state the real run then trips over, and both report green, because from inside the check nothing went wrong.