The scenario that asserts the opposite of what it was named to check
A model can write a test suite from an API contract in under a minute. It will be syntactically valid, it will map to real step definitions, it will execute, and it will return a definite verdict on every scenario. None of that establishes that the tests are worth running. The artifacts below came out of our own generator, and one of them is wrong in a way that nothing downstream of it is equipped to notice.
The input is an OpenAPI document. The generator reads the contract, proposes the tests the endpoint admits, grades each one for whether it can honestly be automated from a specification alone, and formalises the selected set as Gherkin before any code is written.
| Category | Scenarios | Feasibility | Offered for selection |
|---|---|---|---|
| Functional | 3 | Fully automatable | Yes |
| Boundary | 2 | Fully automatable | Yes |
| Error handling | 2 | Fully automatable | Yes |
| Response validation | 3 | Fully automatable | Yes |
| Edge cases | 3 | Fully automatable | Yes |
| Security | 3 | Partial | No — gated off |
The last row is the generator behaving well. SQL injection, cross-site scripting and brute force cannot be established from a contract; asserting that a single crafted request did not return a 500 is theatre, and the tool refuses to offer them rather than producing a green tick that means nothing. That instinct — decline the test you cannot honestly write — is exactly the one that fails two rows above it.
These are the first and fourth scenarios of the generated feature file, reproduced as
written. Read the Given line of each, then the Then.
@Functional Scenario: Successful Login Given the user provides credentials {"email": "eve.holt@reqres.in", "password": "cityslicka"} When the user sends a POST request to "/api/login" Then the response status code should be 200 And the response should contain a valid "token"
@Boundary Scenario: Maximum Character Limit Given the user provides credentials {"email": "eve.holt@reqres.in", "password": "cityslicka"} When the user sends a POST request to "/api/login" Then the response status code should be 400 And the response should contain an error message
The payloads are identical, character for character. One scenario asserts that this
request succeeds. The other asserts that it fails. They cannot both be right, and the
second one is not testing what its name says: nothing in it sits at a maximum character
limit. It is the happy-path payload with the expected status changed to 400.
A generated test that runs tells you the generator ran. It does not tell you the test was worth writing.
This defect survives every gate a normal pipeline puts in front of it, because every one of those gates is asking a different question.
steps module. Nothing is undefined.Whichever verdict the run returns, the outcome is bad, and the second is worse than the first.
A red boundary test points at the service. The engineer reads
Maximum Character Limit, sees a 200 where 400 was expected, and goes
looking for missing length validation in an endpoint that never had any. The most
likely resolution is to change the assertion to 200 — which permanently enshrines
a boundary test that exercises no boundary.
Against an environment that rejects that request for an unrelated reason — rate limiting, a stale fixture user, a different tenant — the scenario goes green and stays green. The suite now reports that maximum-length input is covered. It has never once been sent. Nobody revisits a passing test.
The agent reviews generated tests the way it would review generated code: against the thing they were supposed to be, not against whether they run. Four findings on this feature file, one of which is a commendation.
Maximum Character Limit sends a 18-character email and a
10-character password.
Neither value approaches any limit declared in the specification, and no
maxLength is referenced anywhere in the scenario. The name asserts a
boundary the body never visits. A boundary scenario must contain at least one value at
or across the boundary it names, and this rule is mechanically checkable.
Scenario 01 expects 200. Scenario 04 expects
400. The request bodies are byte-identical.
For a deterministic endpoint these cannot both hold. Whichever way the run goes, one of the two scenarios is reporting a falsehood, and the suite has no way to say which. This is the cheapest check in the set — group scenarios by request payload, and flag any group asserting more than one status class.
The specification declares no length constraint on either field.
A test asserting that over-length input returns 400 is testing a rule
that does not exist in the document it was generated from. Either the contract is
incomplete and should say so, or the test is inventing a requirement. Both are worth a
person’s attention; neither is worth a silent green tick.
Three scenarios graded partial and made unselectable.
Injection, cross-site scripting and brute force were proposed, assessed as not honestly automatable from a specification, and withdrawn from selection rather than shipped as weak assertions. This is the correct behaviour, and it is the standard the boundary scenarios failed to meet.
None of these require a model. They are structural properties of a generated suite, and they run in the same pipeline stage that already parses the feature file.
| Check | Question it asks | Caught here |
|---|---|---|
| Payload divergence | Does a scenario claiming to deviate from the happy path use data that differs from it? | Scenario 04 |
| Assertion collision | Do two scenarios with the same request assert different status classes? | Scenarios 01 and 04 |
| Boundary presence | Does a scenario tagged @Boundary contain a value at or beyond a declared limit? |
Scenario 04 |
| Contract grounding | Is the expected response one the specification actually declares for this operation? | Scenario 04 |
| Feasibility honesty | Is any scenario asserting a property that cannot be established by the request it sends? | None — already withheld |
| Left as generated | Consequence | When it would surface |
|---|---|---|
| Boundary scenario with happy-path data | Maximum-length input is reported as covered and has never been sent | The first production request long enough to break something |
| Contradictory sibling assertions | One of the two scenarios is always lying; the suite cannot say which | Never — the failing one gets its assertion “fixed” |
| Assertion with no basis in the contract | A requirement that exists only inside a test, enforced against a service that never agreed to it | At the next contract change, as an unexplained regression |
Thirteen scenarios were generated in well under a minute, and twelve of them are useful. Finding the thirteenth took four structural checks over the feature file — no model call, no live environment, no human reading every line. The point is not that generation is unreliable. It is that a suite nobody verified is a suite nobody should be relying on, and the speed of writing tests is exactly what makes reviewing them impossible by hand.
The pipeline these artifacts came from — specification in, graded scenarios, Gherkin, containerised execution, an exportable project the team keeps — is our API Test Generator and Execution. The assurance pass described here is the stage that decides whether anything it produces is allowed to count as coverage.