How Do AI Agents Test Software? A Deep Dive

Agentic testing means an LLM-driven agent operates your application the way a person would, reading the page, deciding what to click, typing into forms, while a separate model judges whether the run actually satisfied a stated outcome. Nothing about the test steps is scripted in advance. The agent decides the path, and a verifier decides the verdict.

I build TestAutomate, so I hear the skeptical version of the question a lot: what makes this “agentic” rather than a fancy wrapper around the same old scripts? Fair challenge. The term gets used loosely enough that the only useful answer is a concrete one, so this guide walks through what actually happens between you typing a sentence and a verdict landing on your dashboard, with real product behavior at every step and real screenshots of it.

How does an agent actually perceive a web page?

Not by holding a script: the agent reads the live page each time it acts and finds the control it needs the way a person would, by what the element is and says rather than by a stored locator string, so a test never contains a CSS class, an XPath, or a screen coordinate. It decides every next step from what the page shows right now, not from a picture of how the page looked when someone wrote the test. That’s the entire reason there’s nothing for you to author when you create a test and nothing for you to repair when a redesign renames half your markup.

It also changes what a test is. In TestAutomate a test is three things you write in natural language: an id, a prompt (the intent, what to do), and an expected_outcome (what must be true afterward). You can add an optional setup step to establish a precondition and a cleanup step to undo whatever the test created, both written the same way. There’s no step recorder to babysit and no exported script that starts rotting the day you record it. When the UI changes, the sentence “create a project called Demo and verify it appears in the list” doesn’t change with it.

Is agentic testing deterministic or probabilistic?

Both, in different places: the test’s declaration is deterministic, the same expected outcome graded against the same contract every single run, while the agent’s execution is probabilistic and free to vary its route through the page between runs, and that split is the real fault line between agentic and scripted automation. Which order the agent reads fields in, how it goes hunting for a control, whether it takes an extra look because a menu was still animating in: none of that is fixed. Two runs of the same test can take visibly different routes through the UI and both correctly pass, because the pass condition is the stated outcome, not a fixed sequence of clicks.

Runs aren’t purely binary either. A run comes back blocked when a setup step couldn’t establish what the test needed, say the app errored or a required control was missing. Blocked is scored as neither a pass nor a failure, because the test never actually exercised the feature, and a blocked run never becomes a bug candidate. That tri-state (pass, fail, blocked, plus a separate skipped outcome for a genuinely inapplicable test) exists specifically so an environment problem doesn’t get miscounted as your app being broken. There’s a full write-up of that distinction in why automated tests get blocked instead of failed.

Here’s what that looks like on a real run rather than in theory. This is our dashboard showing a recorded suite run where the test “Add Participants to Existing Cycle” came back blocked, with the verifier’s reasoning spelled out on the row: the setup phase couldn’t establish the precondition, the app was never exercised, and the run is explicitly called out as a test-environment problem, not an app defect.

How does the verifier grade an agentic test run?

Every run gets graded by a separate verifier model, not by the agent that did the driving: the verifier reads the full record of the run against your expected_outcome and returns an itemized verdict naming which expectations were met and which are missing, so what you get back is a checklist with reasoning, not a single red or green light. It’s built to be skeptical by design. It looks for evidence in what actually happened on the page rather than taking the agent’s own account of success at face value, and when a core expectation is genuinely in doubt it errs toward failing the run, because a false pass that lets a broken feature ship is worse than a false alarm that costs you a second look.

The verifier can also attach an assumption flag to a passing run when something looked off without breaking a core expectation, but a flag is never a way to pass a broken test. If a core expectation is unmet, the run fails, full stop. The mechanics of the judging setup have their own write-up in how the LLM-as-a-judge verifier grades runs.

A probabilistic actor doing the driving is only trustworthy if something separate and skeptical checks its work. Here’s a real failed verdict from a recorded run, where the agent operated on the wrong item, a review cycle named “360-Review-Fiscal-Year-2026” instead of the required “360-QA-Regression-2026”, and the verifier caught it and failed the run with that exact reason on the row:

That screenshot is worth pausing on, because it shows the failure mode people worry most about with agentic testing, the agent doing something plausible but wrong, being caught by the layer that exists to catch it. And a genuine judged failure isn’t final on the first attempt: the test gets exactly one automatic re-run on a stronger model before the failure is called real. A pass on that retry is recorded as agent error rather than an app defect, and failing both attempts is recorded as high confidence the issue is real. False alarms get filtered without a human triaging every red; the full logic is in why we re-run failures before calling them real.

How is agentic testing different from traditional test automation?

A scripted test is a bet on a locator staying put, a specific class, id, or DOM position, and when a redesign or even a class-rename voids that bet, the test fails on a string mismatch that has nothing to do with whether the feature works. An agentic test doesn’t hold that locator at all. It reads the live page each run and finds what it needs by description, so a rename or a reflow that would break a selector-based script is invisible to it. What it can still fail on is different, not absent. A hidden element, a slow-loading menu, or a description that’s genuinely ambiguous will still leave the agent with nothing to click. Removing the selector removes one whole failure mode; it doesn’t remove all of them.

Laid against the ways teams actually automate today, the differences stack up like this:

The other real difference is in what the test declares. A scripted test’s assertion is usually “element X has value Y,” a check against the DOM. An agentic test’s expected_outcome is closer to how a person would describe success: “the task is assigned to the right person, no other tasks were created.” Writing that well is genuinely a skill. Each line of an expected outcome is a core assertion that must hold for the test to pass, unless it’s prefixed [incidental], which marks a nice-to-have, like a success toast, whose absence shouldn’t fail the run. A bad expected outcome produces a confidently wrong verdict in either direction.

What does it cost to run an agentic test?

Whatever your Anthropic usage costs, and nothing on top: none of the calls in a test run are metered or marked up by TestAutomate itself, every one bills straight to whichever Anthropic credentials you’ve configured, and the real cost sits on your own provider dashboard rather than hidden inside a per-seat price. That’s a deliberate contrast with how this market usually charges, where the model spend is bundled into a seat or a quote and you can’t see what any given test actually cost you.

Honesty requires saying that per-test cost isn’t a single number. A long, driving-heavy test costs more than a short one almost by definition, which is exactly the kind of thing a bring-your-own-key setup lets you see for yourself instead of taking a vendor’s blended average on faith. The BYOK post goes deeper on what bills where.

What does a full agentic test run actually look like?

Four phases run in order: an optional setup step establishes a precondition idempotently (“check whether X exists; if not, create it”), the prompt phase drives toward the goal, the verifier grades the outcome, and an optional cleanup step runs afterward to undo what the test created, without ever affecting the verdict. Every phase runs on a bounded time budget and the run itself is bounded too, one of the guardrails that decide what an agent may conclude on its own, so a confused run ends with a real, explained verdict instead of hanging your suite. If setup can’t establish its precondition, the run stops there as blocked, before the feature under test is ever touched, which is exactly what the blocked screenshot above shows.

The shape becomes concrete with a worked example: a test that creates three dependent tasks in a project tracker (“task1,” then “task2” blocked by task1, then “task3” blocked by task2) and switches the project to Timeline view. Its expected_outcome isn’t one assertion. It’s five, spelled out as separate lines: each task exists with the right dates, each dependency is actually set, the view is Timeline and not List or Board, and all three bars render across the correct date range. If the agent gets four of those five right and mislabels one due date, the verdict isn’t a blanket fail. It’s a structured list naming exactly which expectation is missing, which is a materially different debugging experience than a scripted test’s single red X.

If you want the two most concrete pieces of this in TestAutomate specifically: selectorless testing covers the no-selectors idea in more depth, and generating a suite from your own docs walks through what the test-generation side actually produces, including a case where it got an expected outcome wrong. Generated tests follow the same intent-only rule as hand-written ones, no code and no selectors allowed in them, and drafts that would depend on leftover state from another test get dropped before they ever run, because a bad test is worse than a missing one. The product walkthrough shows the same pieces assembled end to end.

For background on why fixed selectors break so often in the first place, the W3C WAI-ARIA authoring practices document the accessible names and roles that well-built pages declare on their controls, the stable, human-meaningful layer that outlives any particular class name or DOM position.