Agentic testing: what it is and how it works

Agentic testing is software testing in which an AI agent is given a plain-language goal and an expected outcome, then drives the real application the way a user would: reading each page, deciding what to do next, and adapting when the interface changes, while a separate model judges whether the recorded run actually met the expectation. The steps aren't scripted. The standard is.

The term is young and its definition race is still open, which is why so many pages define it with adjectives. We build TestAutomate, a tool in this category, so we have an obvious stake in how the term settles. This page tries to earn its definition the only way that counts: by showing the mechanism, the trade-offs, and the guardrails, and linking deeper wherever one page can't hold the detail.

How does an agentic test actually run?

In a loop of small decisions. The agent reads the current page, picks the single action that moves the stated goal forward, performs it, then reads the page that resulted, and repeats until the flow is complete or genuinely stuck. Nothing about the route is fixed in advance, which is exactly what lets the same test survive a redesigned form or a relocated menu.

The test itself is an artifact you could read aloud. In TestAutomate, a test has exactly three required fields, an id, a plain-language prompt, and an expected outcome, plus optional setup and cleanup prompts. Every field is plain language, and there is no selector field to fill in anywhere. Here is a complete one:

id: saved-filter-persists
prompt: >
  Create a saved filter named "My open tickets" that shows only open
  tickets assigned to the current user, then reload the app and open
  the filter again.
expected_outcome: |
  - A filter named "My open tickets" appears in the saved-filters list.
  - Opening it shows only tickets that are both open and assigned to
    the current user.
  - After a full page reload, the filter still exists and returns the
    same view.
  - [incidental] A confirmation toast appears when the filter is first
    saved.

When the agent needs the save button, it doesn't look up a stored locator, because none exists. It reads the page and finds elements like a person would, so there are no selectors to maintain when the front end changes. The run around it is structured rather than freeform: an optional setup phase establishes the test's preconditions, the act phase performs the test, and a cleanup phase removes what the test created, with each phase bounded so nothing hangs forever. The capability class underneath is recent, the kind of model behavior Anthropic introduced as computer use: models trained to operate real software rather than just describe it. For the loop one stop at a time, how computer use agents for QA run a test walks the full circuit, worked example included.

Is agentic test automation deterministic or probabilistic?

Both, at different layers, and knowing which layer is which is the whole trust story. The declaration you write is deterministic: the expected outcome doesn't drift between runs, and neither does the vocabulary of results it can produce. The execution that satisfies it is probabilistic: the agent may take a different route through the UI on Tuesday than it did on Monday. Our comparison of agentic testing vs traditional test automation walks that fault line in depth, so here is just the shape of the answer.

The objection writes itself: if a model decides what passed, haven't you swapped asserts for vibes? That's a fair objection. The design answer is to anchor every judgment to something fixed. In TestAutomate, a separate verifier model grades each run from the recorded trajectory rather than from the agent's own account of it, and unevidenced success counts as not met. The agent doesn't get to declare victory. It has to have left evidence.

The verdict comes back in a four-word vocabulary: passed, failed, skipped, or blocked. Blocked means the test's preconditions couldn't be established. It is neither a pass nor a fail, and it never becomes a bug candidate, because an environment problem is not an application defect. A judged failure earns exactly one retry on a stronger model, and only a judged failure, never a blocked or skipped run. If the retry passes, the first failure is written off as agent error rather than an app defect. If both attempts fail, confidence rises that the defect is real. And when the app changed in a way that still works, a renamed button, an extra confirmation step, the run passes with a flag for a human to review, but a flag never excuses an unmet core expectation.

The deterministic baseline deserves the same scrutiny. Scripted suites are deterministic in principle and flaky in practice: Google has reported that almost 16% of its tests show some level of flakiness. Determinism that fails randomly is not the safe harbor it's sold as. The useful question is not whether variance exists but where the fixed point lives, and here it lives in the declaration and the verdict, never in the route.

Where does agentic testing sit among test automation tools?

At the adaptive end of a spectrum that runs from fully scripted to fully delegated. Scripted frameworks, record and replay tools, self-healing layers, and AI testing agents differ less in ambition than in two practical questions: where the instructions live, and what a change to the UI does to them.

Read the bottom row with the same suspicion as the rest, because it isn't free. Agentic execution spends model calls on every run, is slower than a unit test will ever be, and needs judging machinery precisely because it is probabilistic. If your UI rarely changes shape and your scripted suite is quiet, the agentic approach, or autonomous QA testing as parts of the category call it, may be solving a problem you don't have. The layers also coexist. Adopting agents doesn't obligate you to delete a working suite.

The self-healing row deserves one more sentence, because it is the category most often mistaken for agentic. Self-healing fixes selectors; selectorless removes them. A repair model works to keep a scripted test alive through UI change, while an agent never had a selector to break, because no user-authored selectors exist in the test at all. Selectorless testing explained covers why selectors break tests in the first place and what a test contains instead.

The categories also answer different questions about your user. An API suite can stay green while the flow a real user walks is broken, and catching exactly that gap is the job of the end-to-end layer, which is where agentic AI in QA earns its keep. What browser agent testing verifies for the user makes that case from a practitioner's seat, layer by layer.

How does TestAutomate put this to work?

TestAutomate is an agentic testing tool for web apps: you write tests as intent-based prompts with expected outcomes, an agent runs them in a real browser, and a separate model grades every run against what you declared. The verdict itemizes which expectations were met and which were missing, and any bullet marked incidental can never fail a run, so a missing toast never takes down the test for a working workflow.

Failures don't become tickets on their own. A judged failure is triaged first, and only what looks like a real application defect becomes a pending bug candidate, carrying the verifier's reasoning and the exact unmet expectations. Filing it takes a human click, with a duplicate check before anything is created. TestAutomate never files a bug ticket by itself. That boundary between what the agent may conclude and what it must hand to a person is a design surface of its own, and agentic testing guardrails and human oversight maps it end to end.

The cost model is deliberately boring. No markup, no per-seat fee. Claude calls bill your own Anthropic account, retrieval is local and free, and only optional voice input uses a separate OpenAI key.

TestAutomate is pre-launch.

Watch a run, then decide

No page settles a trust question, but watching the mechanism move helps. See how a run works. TestAutomate is pre-launch for teams that want intent-based regression coverage on a real web app.