Green is not the same as correct
This build is green. Every test passes, coverage is above the threshold the team agreed, and the pipeline is telling the truth. It is also missing three of the fourteen things the release was supposed to guarantee.
Coverage counts lines the tests happened to execute. It has no opinion about which requirements those lines were written to satisfy. Mapping the suite onto the release’s acceptance criteria instead produces a different picture from the same run.
Line coverage measures how much of the code ran. It cannot tell you which promises were kept.
The three uncovered criteria are not an oversight by a careless team. They are the three hardest things in the release to write a test for, and coverage as a metric actively hides that — because the easy code is also the code that inflates the number.
| Module | Line coverage | Criteria exercised | Why they diverge |
|---|---|---|---|
| RefundCalculator | 98.2% | 3 of 3 | Pure arithmetic. Trivial to test, and every test adds lines |
| RefundValidator | 94.6% | 4 of 6 | Balance rule needs a part-refunded order in a specific state |
| ApprovalPolicy | 76.1% | 1 of 3 | Approval needs a second actor. No fixture exists for one |
| OrderStateGuard | 88.9% | 3 of 4 | Cancelled-order path reachable only through the batch consumer |
The module with the highest line coverage has full criteria coverage,
and the module with the lowest has the worst — but the aggregate hides both,
because RefundCalculator contributes far more lines than
ApprovalPolicy. The number goes up as the risk concentrates. Nothing in the
pipeline is lying; the metric is simply answering a question nobody asked.
Criteria coverage is one of four gates added to the existing pipeline. None of them replace the test suite — they run beside it and fail for reasons the suite cannot express.
3 of 14 acceptance criteria have no test that exercises them.
The gate names the criteria, not a percentage. A number invites negotiation; “refunds over £500 require manager approval, and nothing tests it” does not.
Published response shape unchanged against 3 registered consumers.
Field removals and type narrowing break consumers without breaking any test in this repository, because the consumers live in other repositories.
POST /refunds/{id}/approve carries no authorisation
annotation.
The endpoint was added for AC-7 and left open. Every test calls it with an authenticated fixture, so the suite is green and the endpoint is unprotected. This is the finding a passing build is least able to surface.
p95 refund issue 210ms against a 500ms budget.
Measured against the budget in the ticket rather than against the previous run, so a slow drift over many releases cannot pass by being only slightly worse each time.
| Finding | Visible to the existing pipeline | Where it would have surfaced |
|---|---|---|
| Unprotected approve endpoint | No — all tests authenticate | Penetration test, or an incident |
| Approval threshold untested | No — coverage was 91% | First refund over £500 in production |
| Balance rule untested | No | A customer refunded more than they paid |
| Cancelled-order path untested | No | Batch consumer, out of hours |
The build stays green on the checks it already had. It now also reports that three promises are unverified and one endpoint is open — before the release, rather than after the first refund over £500.