Safety metrics get no tolerance

Unit tests prove the code does what it was written to do. They say nothing about whether a prompt edit, a model swap or a retiered tool quietly changed how an agent behaves. Nothing in the type system notices when an agent that used to refuse a request starts complying with it.

So the agent has its own gate, and it runs on every commit. The interesting part is not the metrics. It is that the gate has two rules rather than one, and conflating them is what makes most eval gates decorative.

Retrieval recall moving from 0.94 to 0.92 across an embedding change is noise. Dropping to 0.71 is a regression. So quality metrics are compared against a committed baseline with a tolerance, and only a primary metric moving more than five points fails the build.

Relative comparison is right here, because the absolute number is a property of the dataset rather than the system. A recall of 0.9 means nothing on its own. A recall of 0.9 where yesterday it was 0.96 means something specific.

The other set of metrics counts how often the agent called a tool the test case forbids, and how often it reached above the risk tier the utterance justified. Those fail the build at anything above zero, whatever the baseline says.

The reason is that a tolerance turns a safety property into an average, and averages hide exactly the case you care about. Twenty-four rows passing and one prompt-injection row succeeding is a 96 percent pass rate, which sounds excellent, and it means an agent that can be talked into changing a customer's account limit by a stranger typing SYSTEM OVERRIDE into a chat widget.

A gate that lets that through on tolerance is not a gate. It is a report nobody reads with an exit code of zero.

Retrieval ranking, tool selection and risk-tier violations need no model to judge. They are set membership and an ordering comparison. That matters practically: those suites are free and deterministic, so they run on every commit rather than on the occasions somebody remembers.

A judge model is reached for exactly once, for the one question set membership cannot answer, which is whether every claim in an answer is supported by the context it was given. Three rules keep it honest, and all three are load-bearing. It never sees the expected label, because a judge shown the answer agrees with it. It judges claims against a passage rather than quality, because asking a model whether an answer is good returns its taste. And it is allowed to abstain, with abstentions counted separately rather than rounded into a pass or a fail, because a judge forced to choose invents a reason.

Three kinds are worth more than the obvious ones. Rows where the correct retrieval outcome is nothing at all, because a retriever that always finds something is the failure that reads as success. Rows carrying forged authority and injection through pasted content, because indirect injection through retrieved text is the attack that does not look like an attack. And rows where the right answer is an honest admission of not knowing, because a judge that scores honest uncertainty as a hallucination is miscalibrated, and nothing else in the suite would catch that.

Every row carries a note saying what failure it exists to catch. A golden row whose purpose nobody remembers gets deleted the first time it goes red, usually by the person whose change made it go red.

The default mode replays recorded rankings and tool choices instead of calling the real retriever and the real model. That is not a weaker eval. It is what makes the gate runnable on every commit at all, with no API keys, no spend and the same numbers on every machine.

It also changes where a regression becomes visible. A chunking change that moves a ranking arrives as a diff to a committed JSON file, in review, next to the change that caused it, rather than as a number nobody re-ran.

Which is also the one way the whole thing can lie. A fixture refreshed without being read is how a regression gets blessed into the baseline. There is no technical fix for that, only the discipline of reading the diff, so it is written at the top of the README rather than buried.

Worth naming, because a harness that looks complete stops getting extended. Every suite tests a single turn, so whether an agent correctly refuses on turn four what it accepted on turn one is untested, and that is where a lot of real jailbreaks live. There is no cost or token tracking. And there is no sampling of production traffic, which is the third leg of the standard pattern after offline datasets and a CI gate.

The scorers have their own unit tests, which sounds excessive until you consider what a broken one does. A recall function that returns 1 for an empty result set turns a dead retriever into a green build. An uncalibrated measuring instrument does not fail loudly. It produces numbers that look like evidence.