What broke

The publish that follows a green test suite copies the built site to the host that serves it, over one rsync connection. When that connection did not open — the host resolved, and a moment later its SSH port did not accept the socket — the deploy treated it as a failed publish and went red. Nothing in the deploy distinguished a single dropped connection from a real defect.

How it showed

Run 1434, deploy job, step make deploy:

ssh: connect to host … port 22: Undefined error: 0
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(285) [sender=3.5.0]

The test job on the same run was green, and the render step above the failure built all 28 posts successfully. The host was reachable again within a minute.

What was watching

The distributed traces sensor recorded the run and its failed step; the trace held the evidence that one socket open failed while every other step was green. The synthetic monitoring sensor — the verify step that checks the live site links the newest post — never ran, because it is gated behind the deploy step that failed. So a transient drop was not distinguished from a publish that should not have gone out.

Where it sits

This is a runtime failure: a transient network condition produced a red run for a publish that was otherwise correct. The distributed traces sensor was present and held the evidence. Synthetic monitoring was present but silent, gated behind the very step that failed, so it could not tell a dropped connection from a defective build.

What we did

Changed make deploy to retry up to three times when the failure is a dropped connection — the connection unexpectedly closed or connect to host … port 22 signature — and to fail immediately on anything else, so a permission error is not repeated. The change is pull request #20 on the forge; its merge run deployed green. A stub rsync was used to verify the control: a dropped socket retries and succeeds, while a read-only error fails on the first attempt.

What now fails loudly

The deploy retries a dropped connection and fails fast on real errors, so one transient socket miss no longer turns a clean publish red.