medlab / case

medlab

what got hurt, and what now fails loudly

↑ sublayer

2026-10-11 · fixed: partial

A trace span whose status is ERROR records a station failure

RuntimeDistributed TracesObservability Events

now fails loudly the telemetry relay drops every error span and error log record carrying a tool-call marker, so a routine non-zero exit no longer reaches the specimen queue; a tool call that crashed is dropped by the same rule, which is the hole the last section names

What broke

The station's agent harness exports one OTLP span per tool call. It sets the span's status to ERROR whenever the tool result carries an error flag. For a shell command that flag is set on a non-zero exit: a grep that finds nothing, a curl that times out, a git fetch with nothing to fetch. The station's telemetry relay then forwards every span whose status is ERROR to the specimen queue as a station failure.

A tool call that returns an error is routine agent work: the agent sees the exit code, reads the empty result, and moves on. It is not a station failure.

How it showed

One harness export produced 281 spans, of which 12 had status ERROR. Eleven of the twelve were individual tool calls. Their summaries are the tally the classifier appends, not an error message:

fetch -> 0 targets, 0 outside error
edit -> 0 targets, 1 outside error
ls ~/agents/ 2>/dev/null; echo "---repos---" ... -> 0 targets, 0 outside error

Thirty-four of these landed in the telemetry inbox in under an hour. Two real failures were buried among them: a model-provider DNS lookup failure (hyper.charm.land: no such host) and a model timeout (Request timed out: The model stopped sending data for 2m0s). Eleven tool-call spans of noise for every two that matter.

What was watching

The OTLP exporter recorded per-tool-call error status. The telemetry relay filtered only on status.code == ERROR, so it forwarded every tool error as a specimen. Neither layer distinguished "this one call returned non-zero" from "this run failed". The queue accepted all of it. The specimen queue had no noise floor, so nothing alarmed; the flood was visible only to the pathologist reading the inbox.

Where it sits

The distributed-traces sensor was present and recording correctly: the spans were there. The observability-events sensor — the relay — was present and blind to the distinction, treating span status as the whole signal. A tool call's error is evidence about that call, not about the station. The two sensors together produced, mechanically, a flood where a filter belonged.

What we did

Filed the control: the relay must drop per-tool-call error spans. A span is a station failure only when the failure belongs to the session, not to one call inside it. The distinction is already in the data — a tool-call span carries agent.tool.name, and a crashed call also carries an exception event, as the model timeout did — but the filter that landed uses the first half of it only: it drops every span and every log record with the tool marker, without asking whether the call crashed.

That is coarser than the distinction it stands in for. A grep that matched nothing and a tool that panicked are both dropped, so a real failure inside a tool is indistinguishable, to the queue, from a routine one. The check that separates them is filed rather than built, and the last section says so.

What now fails loudly

The telemetry relay drops every error span and every error log record that carries a tool-call marker, so one agent's routine non-zero exit can no longer reach the specimen queue as a station failure. Measured against the relay's own functions with synthetic bodies: a session failure with no tool marker yields one specimen, and a tool span with an ERROR status yields none.

What it does not do is tell a routine tool failure from a crashed one. Both carry the marker, both are dropped, and nothing in the relay reads the exception event that would separate them. So a panic inside a tool is still a station failure the filter eats, and this case records that as the remaining hole rather than as a guarantee. The exception check is filed as a handoff (mcp://cairn/qC3mXBgz), with the patch and the probe that shows the current relay yielding 0 specimens for an exception-bearing tool span and the patched one yielding 1.