False Positives in Automated Testing

False positives in automated testing are results that report the wrong thing in either direction: a false alarm marks the app broken when only the test or the agent driving it misfired, and a false pass marks the app healthy when a real defect slipped through. Fixing them takes two separate mechanisms, one per direction. TestAutomate attacks the pair with a skeptical verifier against false passes and one disclosed retry on a stronger model against false alarms.

Most AI test automation tooling only worries about one direction, and usually treats it with the bluntest instrument on the shelf, retrying until the red goes away. I think the blunt instrument does more damage than the disease. So what follows is both failure directions, the mechanism we designed for each, and the decision rules I’d apply to any suite, ours or not.

What’s the difference between a false positive and a false negative in software testing?

A false positive is a run that reports a defect that doesn’t exist, and a false negative is a run that stays green past a defect that does. The vocabulary trips people constantly, because the statistical convention testing borrowed defines positive as the condition being detected. The condition a test screens for is a defect, so a positive is a red result, and a false positive is red with nothing real behind it. Plenty of engineers use the words the other way around, since a positive result sounds like it ought to mean a pass. Both camps are internally consistent, which is exactly why the false positive vs false negative in software testing argument never converges.

My decision rule is to drop the statistics vocabulary at the door and name errors by the harm they do. A false alarm wastes an investigation and erodes attention. A false pass ships a bug. Neither phrase gets misunderstood in a standup.

Naming them separately matters because their costs behave differently. A false pass is worse per incident, since the defect it hides travels to production wearing a green stamp. False alarms are worse in aggregate, because they compound. Each red that turns out to be nothing lowers the energy anyone brings to the next red, and a suite whose reds are mostly noise ends up unread, which converts every future false pass from unlucky to inevitable. That’s why I treat the two directions as one problem with two fronts. A fix that buys fewer false alarms by quietly accepting more false passes, which is what blind retries do, isn’t a fix. It’s a transfer.

Why do agent-driven tests fail when the app is fine?

Because two different systems can fail during a run, the application or the agent operating it, and both failures surface identically as a red verdict. In agentic testing, an intent-based test is executed by an agent that reads the live page and acts the way a person would. That removes the classic false-alarm factory of scripted suites, the stale stored selector, but it introduces a new source: judgment. The agent can pick the wrong record from a list, misread an ambiguous menu, or act on a page that merely resembles the right one. The verifier judging the run then reports, correctly, that the expected outcome wasn’t met. Correct judging, wrong culprit.

Here’s a real one from our QA environment, recorded against a test tenancy full of demo data:

The test needed the agent to work inside a review cycle named 360-QA-Regression-2026. It did everything inside 360-Review-Fiscal-Year-2026 instead. The application behaved fine the whole time, and the verifier’s reasoning names the wrong-cycle mistake outright. The red is real in the narrow sense, because the expectations tied to the required cycle genuinely weren’t met, and a false alarm in the sense that matters, because no defect exists. That’s the exact species of failure that needs disambiguating before a human gets paged.

Scripted automation has its own well-measured version of the problem. Google reported that almost 16% of its tests showed some level of flakiness, and the flaky test literature catalogs how timing, environment, and test interdependence produce reds with no code change behind them. Agentic suites don’t escape that gravity. They swap some failure sources for others. So the honest question isn’t whether first attempts can be wrong. They can. It’s what the runner does next.

How do we cut false positives in automated testing?

By making a failure earn its verdict twice, under strict conditions, in the open. When the verifier judges a run failed, TestAutomate re-runs the test’s own steps exactly once on a stronger model than the first attempt used, then judges the new attempt from scratch. Setup isn’t repeated. The precondition the setup phase already established is carried into the retry, so the second attempt starts from the same guaranteed state as the first and the comparison stays clean.

The guards around that retry matter as much as the retry itself, because escalation is only meaningful while it’s scarce. It fires solely for a failure the verifier genuinely judged. A blocked run, where setup couldn’t establish the precondition, never escalates, because re-running a missing precondition on a smarter model fixes nothing. A skipped run never escalates, since nothing was judged. A run aborted mid-flight never escalates. And when the verifier itself was unreachable and the run fell back to a synthetic failure, that never escalates either, because a verdict nobody actually judged hasn’t earned a second attempt.

The two outcomes read very differently, by design. If the stronger attempt passes, the first red was an agent-execution issue rather than an app defect, and the verdict says so. It’s annotated as escalated and passed-on-retry, never laundered into ordinary green, so a test that habitually needs the stronger model stays visible in your history instead of hidden by it. If the stronger attempt fails too, you now hold two independent attempts, judged separately, agreeing the expectations weren’t met. That’s a high-confidence red, and it’s the only kind we ask a human to treat as a probable defect.

One edge case shows how conservative the bookkeeping is. If the verifier goes down during the retry, the runner doesn’t fabricate a confident failed-both-attempts story out of a judgment that never happened. It keeps the first, genuinely judged failure as the authoritative verdict and records that the retry couldn’t be judged.

Why one retry and not three? Because every extra blind attempt is another roll of the dice, and dice rolls launder flakiness. A retry on the same model mostly tests your luck. A single retry on a stronger model changes the capability behind the attempt, which turns the re-run into an experiment: if capability was the problem, the retry passes, and if the app is the problem, no amount of capability will make broken behavior pass. One disclosed, escalated attempt extracts nearly all the information a re-run can offer, and it’s bounded like everything else in the run, so a failing test can’t spiral into an open-ended retry loop.

How do other approaches handle a first-attempt failure?

With one of three defaults: go red immediately, retry blind until green, or queue the red for a human to triage. Each default answers the agent-error-or-app-defect question differently, and the differences show up most clearly in what happens to each kind of red.

Honest caveats, because every row is defensible in its own house. Failing fast is the right default when reds are cheap to triage and the suite is small. Blind auto-retry exists because transient infrastructure hiccups are real, and a bounded retry genuinely absorbs them. Its damage isn’t the retry, it’s the silence, a pass that hides its own history, and teams that log retried outcomes and review the log have rebuilt much of the missing disclosure by hand. A good human triage queue is the most accurate classifier on the table, since people are excellent at telling agent error from defect. You’re paying latency and per-red cost for a distinction structure could have made upfront. And our policy has a cost too. The failing subset of a suite runs twice, with the second attempt on a stronger model that costs more per run. I think that’s the right place to concentrate spend, on exactly the runs where a second data point changes the conclusion, but it is spend.

The disclosure is also what makes flaky test detection and quarantine workable without a separate tool. A test that passes on retry once probably had an ordinary agent stumble. A test whose history shows escalation after escalation is telling you something is unstable, in the test, the feature, or the environment, and because every escalation is annotated on the run’s record, that history is evidence rather than anecdote. Quarantine should then be a decision, not a side effect. Pull the test out deliberately, with a ticket and an owner, the way Martin Fowler’s quarantine advice has long recommended, rather than letting silent retries quarantine it for you inside a green checkmark. There’s a longer treatment of what AI changes about flaky tests on this blog if flakiness is your suite’s dominant noise source.

What stops the false pass in the other direction?

A verifier that’s structurally separate from the agent and openly skeptical of it. The agent that drives the browser never grades its own run. A separate judge model reads the recorded trajectory against the test’s expected outcome and returns an itemized verdict, every expectation met and every expectation missing. Its instructions encode the asymmetry from the top of this piece: claiming a pass when a core behavior broke is far worse than a false alarm, an agent’s unevidenced claim of success counts as not met, and when in doubt on a core expectation, the verifier marks it missing and fails the run.

That skepticism is deliberately biased toward false alarms, which is precisely why the escalation retry exists. A judge that errs toward red needs a disclosed mechanism on the other side to keep its strict failures honest, and a retry that errs toward rescue needs a strict judge to keep its greens meaningful. Neither mechanism works alone, and I wouldn’t ship either one by itself.

Two walls keep the remaining noise out of the failure channel entirely. Expectations prefixed [incidental] can never fail a run, so a missing success toast doesn’t produce a red that outranks the point of the test, and the cosmetic assertions that generate so many trivial false alarms in scripted suites get a place to live where they can’t hurt anyone. And runs that never exercised the app, blocked on a precondition or skipped by configuration, end in their own outcomes rather than as failures. Four outcomes, passed, failed, skipped, blocked, each meaning one thing, so the failure column only ever contains runs where the application itself did something wrong.

The far end of the pipeline holds the same line. A judged failure can become a bug candidate, deduplicated against existing issues by a strict check, but nothing is ever filed anywhere without a human clicking to confirm it. Blocked and skipped runs can’t produce candidates at all. By the time a defect report reaches your tracker, it has survived a skeptical verdict, a stronger-model re-run, a dedup check, and a person.

Your half of the bargain is the expected outcome itself. A vague assertion is a false-pass generator no verifier can rescue, because a test that can’t lose isn’t testing anything, and I keep a separate checklist for writing expected results that’s mostly about closing that hole.

What all of this buys is test suite trust, which is the only asset a suite really has. Green has to mean examined and found whole. Red has to mean twice-earned and worth your minutes. Every mechanism above exists to keep one of those sentences true, and the day either stops being true, the suite stops being a safety net and starts being a dashboard. I built the verdict pipeline so we’d never have to choose which direction of wrong to tolerate. The answer is neither, enforced separately.