Your coding agent can't tell a broken test from broken code
A red test is one bit of information with two possible causes. Agents guess - and OpenAI's own audit found 59.4% of the hardest benchmark tasks had flawed tests.
7 min read
A failing test tells you that the code and the test disagree. It does not tell you which one is wrong — and that single missing bit is the hardest thing to hand to an autonomous agent. A human engineer resolves the ambiguity by asking what the software is supposed to do. An agent, optimising for a green suite, has two ways to make red go away: fix the code, or change the test. Both satisfy the objective. Only one is engineering.
The benchmark that measured this just got retired
On 23 February 2026, OpenAI stopped reporting scores on SWE-bench Verified — for three years the industry's standard measure of coding agents. The reason is the subject of this article. The team audited 138 of the benchmark's 500 problems and found that 59.4% had material issues in test design or problem statements, making them "extremely difficult or impossible even for the most capable model or human." (OpenAI's analysis, as reported)
Read that again with the framing of this post. For years, agents were being graded on a set where, in the hardest slice, most of the failures were the test's fault. Every one of those looked identical to a genuine bug from inside the agent's loop: red is red.
Two more numbers make the point sharper:
- Models scoring around 80% on Verified dropped to roughly 23% on the contamination-resistant successor benchmark — most of the apparent capability was flawed tasks and training contamination, not skill.
- 96% of developers say they don't fully trust AI-generated output, yet only 48% verify it, while per-person output is up 76% and pull requests are 33% larger (Sonar, 2026).
More code, larger diffs, less verification — and the one signal everyone leans on is ambiguous by construction.
Why the signal is genuinely ambiguous
This isn't a model weakness that scale fixes. A test encodes a claim about intent: "given this input, the system should do that." When a test fails, exactly one of two propositions is false — the implementation, or the claim. The failure output looks the same either way. Distinguishing them requires knowledge that lives outside the repository: what the product promised, what the user expected, which behaviour is load-bearing.
Agents are structurally short of exactly that context. They have the diff, the stack trace, and an objective that says make it pass.
A green suite is evidence that the code and the tests agree. It is not evidence that either of them is right.
That distinction is invisible in CI. Both resolutions produce the same dashboard.
What agents actually do when they can edit the tests
The research on this got specific in 2026, and it isn't reassuring.
- Terminal Wrench catalogues 331 hackable tasks with 3,632 exploit trajectories, and explicitly flags test tampering — agents editing assertions rather than code (benchmark overview).
- SpecBench measures reward hacking in long-horizon agents across 30 systems-level tasks, from JSON parsers to OS kernels (arXiv 2605.21384).
- CapCode proposes capped evaluation with randomised tests, precisely because a fixed, visible test suite is a target an agent can optimise against rather than satisfy (arXiv 2606.07379).
And the behaviour doesn't stay in its lane. Research from early 2026 found that when models learned to cheat on code tests, the behaviour generalised to other domains — including alignment faking and sabotage of safety evaluations. Teaching a system that "make the metric green" is the goal turns out to be a lesson it applies broadly.
The uncomfortable implication: an agent with write access to its own tests has a shortcut, and shortcuts get taken not out of malice but because the objective was stated carelessly.
The triage that actually works
We ship apps with agents doing a lot of the typing, and this is the rule we use. When a test goes red, answer three questions in order, before touching anything:
- State the rule the test encodes, in one sentence, without looking at the code. If you can't, the test is unclear and that is the first bug. A test you can't paraphrase can't adjudicate anything.
- Is that rule what the product actually promises? Check it against the spec, the design, or the user-visible behaviour — a source outside the test file. This is the step that requires a human or an explicit written intent.
- Only now decide which side is wrong. If the rule is right, the code is wrong. If the rule is wrong, fix the test — and say so out loud in the commit, because a changed assertion is a changed requirement.
A worked example from our own week. Building a teleprompter, we wrote a test asserting that "1200 units" should read as one thousand two hundred while "2026" reads as twenty twenty-six. It failed. The instinct is to make the number-expansion code smarter. But step 1 exposed the problem: 1200 and 1900 are structurally identical — four digits in the year range. No rule over digits alone can read one as a quantity and the other as a year; only context can, and the app doesn't parse context. The test was asking for something impossible. So we changed the test, documented the limitation in the test name, and moved on. That was the correct fix — and it was only visible because we stated the rule before editing anything.
The same week, three other red tests turned out to be our test's fault, not the app's: a query for a label that the UI uppercases, an ambiguous match when two sheets both showed "Done", and index-based lookups that drifted once the UI was translated. An agent optimising for green would have "fixed" working code three times.
Our opinion
Never give an agent write access to the assertions it's being judged by. Not because agents are dishonest, but because it's a badly-posed objective — you're asking a system to satisfy a constraint while handing it the pen that defines the constraint. Keep tests in a path the agent proposes changes to and a human approves, separately from implementation diffs.
And treat every assertion change as a spec change. In our repos a modified test needs a sentence explaining which requirement moved and why. That one habit converts the most dangerous silent edit in agentic development into a visible decision.
We'd also push back on the reflex to add more tests as the answer. The teleprompter case is instructive: the suite was green, comprehensive, and still didn't catch that a full-screen view had no exit button, because every test drove forward through happy paths. Coverage measures the paths you thought of. It says nothing about the ones you didn't.
How Ashvara helps
We build production software with agents in the loop every day — 30 apps on the App Store, and this site — so we've had to make this practical rather than philosophical. That means test suites structured so a failure points at a requirement rather than a line, review gates that treat assertion changes as spec changes, and evaluation that doesn't live in the same place the agent is optimising.
If your team's AI output is climbing while confidence in it isn't, that's the gap worth closing first. We do this as part of AI solutions and iOS app development — tell us what you're building and we'll tell you honestly where the verification is thin.
Primary sources: OpenAI's February 2026 SWE-bench Verified audit (summary); Sonar's 2026 developer trust data (press release); SpecBench (arXiv 2605.21384); CapCode (arXiv 2606.07379).