Flaky Tests AI Can Fix and the Ones It Can't

I spent close to five years testing a VOIP telephony product, and our suite was red most mornings. Calls connected half a second late, presence indicators lagged behind a reconnect, notifications landed after an assertion had already given up. Plenty of those reds weren’t bugs at all. They were the network being the network, and it was my job to sort one from the other before standup.

So here’s my honest read on the flaky tests AI test automation can absorb and the ones it can’t. No layer of intelligence makes a nondeterministic app deterministic, and anyone promising zero flakes is selling something. What an agentic layer can do is execute tests adaptively instead of through brittle waits, judge every red on recorded evidence, retry the genuinely suspicious failures on a stronger model, and refuse to count environment problems as failures.

What causes flaky tests and how do you prevent them?

Four cause families cover nearly everything, and each one wants a different prevention. The widely cited empirical study of flaky tests in open-source projects by Luo and colleagues found the three most common causes to be async waits, concurrency, and test order dependency, and Martin Fowler’s essay on eradicating non-determinism in tests covers the same ground from the practitioner side. Here’s how I sort them when a suite starts getting noisy.

Timing and asynchrony come first because they dominate. The test asserts before the app finishes doing the thing, so the verdict depends on a race the test doesn’t know it entered. Prevention means waiting on conditions instead of clocks. A test should proceed when the confirmation appears, not after two seconds have politely elapsed.

Shared state comes second. Two tests touch the same record, or one leans on data another created, and the suite’s pass rate starts depending on execution order. The prevention is isolation, with every test owning its data and never assuming a prior step left something behind.

Environment problems are third. Missing fixtures, a misconfigured tenant, a service that isn’t up yet. These reds say nothing about your app, and preventing the damage is partly hygiene and partly reporting that refuses to count them as failures, which I’ll come back to.

External dependencies are the fourth family, and the one I know best. My VOIP years were a live demonstration. The app under test talked to carriers, push services, and other clients we didn’t control. Some mornings the reds told me about a regression. Most mornings they told me about the weather between our lab and the nearest data center, and no amount of selector hygiene was going to change that. You can stub externals in lower layers, but an end-to-end test of a real-time product eventually touches the real world, and the real world doesn’t do determinism.

Size amplifies all four. Google’s analysis of its own test corpus found flakiness climbing steadily with test binary size, which matches intuition, since a bigger test crosses more async boundaries, holds more state, and touches more of the environment per run. The end-to-end tests that matter most are structurally the flakiest kind you can write. And a real-time app concentrates every family at once, because its correct behavior is itself timing-dependent. A call that connects in 300ms today may take 900ms tomorrow over a congested link, and both are correct. The test that hard-codes the wait is the thing that’s wrong.

Why do tests pass locally but fail in CI?

Because CI is a different machine keeping different time, and most suites quietly encode their author’s machine. Your laptop has a warm cache, an idle CPU, and one test running at a time. A CI runner is a shared, throttled container where services boot cold and parallel workers fight over the same fixtures. A wait that comfortably covers a local render loses the race on a loaded runner, and a data assumption that holds when tests run alone breaks when they interleave. Nothing about the app changed between the two runs. The assumptions around it did.

My sorting rule for these has three branches. If it reproduces locally too, treat it as a candidate defect or a test-logic bug. If it appears only in CI, suspect timing and environment before you suspect the app, and go read what the environment did. If it comes and goes in both places, think concurrency or shared data.

An agentic runner shifts this picture in one specific way. A TestAutomate test carries no machine-tuned waits at all, because there’s nowhere in the format to put one. The agent observes the page after each action and proceeds when the app has actually responded, at whatever pace the app produces, so the artifact that breaks under CI timing in a scripted suite simply doesn’t exist here. Runs are still bounded, so nothing hangs forever, but a run that finishes inside its budget is judged purely on the evidence of what happened, never on how long it took. Slow is not a defect unless slowness itself violates an expectation you wrote down.

The flaky tests AI can absorb, and the ones it can’t

The absorbable kind is flakiness the test inflicted on itself, and in my experience that’s most of it. A scripted check goes red when a selector string stops matching or a fixed wait loses a race, and neither event says anything about the app. TestAutomate removes both failure modes from the test artifact. A test is three user-authored fields, an id, a prompt, and an expected outcome, and the runner refuses to load a test missing any of them. The generation rules require the prompt and expected outcome to be plain natural language, flatly banning code, JSON, YAML, and selectors from appearing in them. There’s no locator to go stale and no sleep to mistune, because there’s nowhere in the format to write one. The agent finds elements on the live page by reading it the way a person would.

The expected outcome does its own de-flaking work. It’s written as itemized end-state assertions, including negatives like “no other records were modified”, so the verdict has concrete things to check instead of a vibe. And any bullet you prefix with [incidental] is explicitly a nice-to-have. A missing success toast marked incidental can never fail the run. That gives you a dial for rigidity, where the load-bearing assertions stay strict while the cosmetic ones stop generating noise.

What no AI absorbs is nondeterminism that really lives in the app or its dependencies. A race condition that drops a message one run in fifty is invisible to any single execution, and an adaptive runner sails through the other forty-nine as honestly as a scripted one. The claim worth holding any tool to isn’t zero flakes. It’s that every remaining red gets classified correctly, which is where the verdict layer earns its keep.

How does an evidence-first verdict separate noise from signal?

By splitting the actor from the judge. In TestAutomate the model that drives the browser never grades its own work. A separate verifier model reads the recorded trajectory of the run, every action the agent took and what came back, and judges each expectation strictly on that evidence. It’s skeptical by design. If the trajectory doesn’t contain clear evidence for an expectation, that expectation is not met, however confidently the agent declared success. And the verdict itemizes, listing each expectation met and each one missing, so a red arrives with its reason attached instead of a bare X. I’ve written before about how this verifier grades runs, and for flakiness, the itemizing is the part that pays. In my VOIP days that combination would have deleted a whole category of morning triage, the reds you investigate for an hour just to learn what the failure even was.

Here’s a real one from our recorded QA dataset. The verifier failed a test because the agent had operated on the wrong review cycle, a similarly named one instead of the cycle the test required, and the verdict’s reasoning says exactly that, naming the cycle mismatch as the reason the expectations weren’t met.

The vocabulary matters as much as the strictness, and this is where the false positive vs false negative distinction in software testing does real work. A false positive is a red with no defect behind it, the kind that burns trust. A false negative is a green that let a defect through, the kind that burns customers. Tolerance applied naively trades the first for the second. The verifier’s answer is assumption flags. When the app did something different that still satisfies the intent, a renamed button or an extra confirmation step, the test passes with a flag recording the observed behavior, the expected one, and the assumption made, for a human to review. But a flag can never excuse an unmet core expectation. Cosmetic drift gets absorbed without behavior ever getting a pass it didn’t earn.

What happens when a red might not be the app’s fault?

It gets classified, with separate machinery for the two innocent explanations. This is the mechanism I appreciate most as the person who used to do this triage by hand, because it encodes exactly the question I asked about every VOIP red. Was this the app, or was this everything around the app?

The first innocent explanation is the agent itself. When the verifier judges a test failed, TestAutomate re-runs the act phase once on a stronger model and lets the second verdict stand. A pass on retry is recorded as exactly that, evidence the first failure was agent error rather than an app defect. A second failure is recorded as high confidence that the issue is real. The escalation is deliberately narrow. Only genuinely judged failures qualify, a blocked or skipped run never burns the retry, and if the verifier itself was unavailable, the runner won’t manufacture a confident double-failure out of a fallback verdict. False alarms and why they get re-run walks the full path.

The second innocent explanation is the environment. Every run ends in one of four outcomes, passed, failed, skipped, or blocked. A blocked run means setup couldn’t establish the test’s precondition, so the app was never exercised. It’s recorded as neither a pass nor a failure, never counts toward failure totals, and never becomes a bug candidate, because there’s no app behavior to report. Skipped gets the same neither-nor treatment when a test can’t meaningfully run in the mode you chose. Bug candidates themselves only ever come from judged failures, and even then a human clicks to file each one after a strict dedup check. Environment noise has no path into your bug tracker.

Should you rerun, quarantine, or triage a flaky red?

Triage always happens somewhere, so the real question is who does it and what it costs. The standard plays for a nondeterministic red are rerun-until-green, quarantine, and human triage, and most flaky test detection and quarantine tooling exists to automate the first two.

Some honest caveats on that table. Rerun-until-green is defensible as a stopgap while a real fix is in flight, and quarantine done with an exit date is what Fowler’s essay linked above actually recommends, a holding area with a short fuse rather than a dumping ground. Purpose-built flaky test detection and quarantine tools also do something TestAutomate doesn’t attempt, watching pass and fail patterns across long run histories to surface statistical flakiness. The bet here is on classifying each red at the moment it happens instead. The escalation retry costs a real extra run on the failing subset, which is a price. And the last row’s failure mode is genuine. A pass on retry means the stronger attempt succeeded, not that the app is innocent, so the same test earning that label repeatedly is worth a human look.

The recorded runs in our own QA test environment, an HCM app with 25 suite runs on record, show how much of that triage the verdict split does before you arrive. Run 33 failed 2 of 2 tests in 12m 10s, a real investigation. Run 32 finished with one failed and one skipped in 2m 11s. Runs 34 through 36 each ended blocked in about a minute, 36s, 49s and 1m 3s, which says the environment needed fixing and says nothing bad about the app. On the dashboards I grew up with, all six of those runs would have been the same color.

What I’d have given for that split on my telephony mornings. Flakiness didn’t go away when I changed domains, and it won’t go away from yours. The realistic goal is a suite where every red has already been cross-examined, the environment problems wear a different color, and the agentic approach to testing spends its adaptability on your app instead of on your patience.