Computer Use Agents for QA, Explained

Every demo of computer use agents for QA tells the same story. An AI takes over a browser, glides through a signup flow, and the caption implies the testing profession is over. I built TestAutomate, whose computer use agents run intent-based QA checks in a real browser, and I still don’t buy that story. The clicking was never the hard part. What decides whether you can trust a green result is everything wrapped around the clicking.

A computer use agent is an AI model that operates software through the same interface a person uses. It looks at the current state of the screen or page, decides on one action, performs it, and looks again. Applied to QA, that loop is the engine of agentic testing: it lets you state a test’s intent and have an agent carry it out in a real browser, with a separate judge grading what actually happened.

That definition hides two design questions that matter more than any demo. First, how much autonomy should the agent get, because the research record on open-ended computer use is humbling. Second, who checks the agent’s work, because an agent grading its own run is a conflict of interest wearing a lab coat. I want to work through both, along with the comparison I find most clarifying, which is how each kind of check sees your app in the first place.

What is a computer use agent?

It’s an AI model handed the controls of ordinary software, pursuing a goal through the same buttons, forms, and pages you’d use yourself. A conventional script talks to an application through code. An agent operates the application. Ask one to submit an expense report and it opens the app, finds the form a person would find, fills it the way a person would fill it, and copes with the confirmation dialog nobody warned it about.

How do computer use agents work, then? The general recipe is a loop. The model is shown the current state of the interface, chooses a single action toward its goal, takes it, and is shown the new state. Repeat until done or stuck. Everything else, and there is a lot of else, is scaffolding around that loop, deciding what the model gets shown, which actions it may take, and when to stop it.

The research record says the unscaffolded version is genuinely hard. When the WebArena benchmark put language-model agents to work on realistic, open-ended web tasks, the best agent its authors evaluated finished 14.41% of tasks end to end, against 78.24% for humans. A year later, OSWorld measured the best model it tested at a 12.24% success rate on open-ended desktop tasks that humans completed at over 72.36%. Agents have improved a lot since both papers were published, but the gap they measured is the honest starting point for anyone proposing to let one roam free across your product.

Those numbers are also why I think QA is one of the few places where computer use agents already earn their keep. A regression test is the opposite of an open-ended task. The starting point is known. The flow is described. The finish line is a stated expected outcome. Scoping the agent that hard is what converts a research demo into something you can run every night, and it’s the first of two design decisions the rest of this piece hangs on. The second is that the agent never gets to mark its own homework.

How do computer use agents for QA run a test?

In TestAutomate, a run moves through four stations you can watch from the outside: describe, act, observe, verdict.

Describe is your half of the bargain. A test is three required fields, an id, a prompt saying what to do, and an expected outcome saying what must be true afterward, and the runner refuses to load a test that’s missing its prompt or its expected outcome. Optional setup and cleanup prompts, written the same plain way, establish preconditions before the flow and remove whatever the run created after it. Even the tests the product drafts from your product docs obey the format’s ban on embedded code and selectors, and their expected outcomes arrive as itemized end-state assertions, negatives included, like no other records being modified. A bullet prefixed [incidental] is declared a nice-to-have that can never fail the run, which keeps a missing success toast from outranking the behavior the test exists to verify.

It helps to see how little that format asks of you. A prompt for a billing flow might read, in full: sign in as the demo user, upgrade the workspace to the cheapest paid plan, and confirm the change from the billing page. The expected outcome is a short list of end-state assertions. The billing page shows the paid plan as active. An invoice for that plan appears in billing history. No other workspace settings were modified. And one bullet reading, the upgrade confirmation banner appears, prefixed [incidental] because a missing banner shouldn’t sink a run that provably upgraded the plan. Notice what’s absent. Nothing names an element, a URL pattern, or a wait. What to click is the agent’s problem on the day the test runs. What must be true afterward is permanently yours.

Act is the agent’s half. It drives a real browser through the flow you described, reading the live page and finding elements like a person would, so there are no selectors to write and none to maintain. I’ve covered what browser agent testing changes and why removing stored selectors matters elsewhere, so I’ll keep the short version. The agent finds what it needs fresh from the live page on every run, which means a renamed button or a reshuffled layout gets absorbed rather than repaired. The run is bounded, too. A confused agent can’t loop forever, and in my view a test that keeps pressing against that bound is really several smaller tests wearing a trench coat.

Observe means the agent works from what the app actually did, never from what the test hoped it would do. Screenshots hold exactly one job in this loop, judging visual state. Did the layout hold, did the header freeze, did the chart render. For everything else, the agent reads the page.

Verdict is where I spent the most design effort, because an agent that grades its own run will grade on a curve. A separate verifier model reads the recorded run against the expected outcome, under standing instructions to stay skeptical, judge only from evidence, and treat an unevidenced success claim as not met. It returns every expectation itemized as met or missing, and the run lands in exactly one of four outcomes: passed, failed, skipped, or blocked. Blocked is the outcome most tools get wrong. It means the test’s preconditions couldn’t be established, so the app was never exercised, and the record says so plainly instead of dressing an environment problem up as a failure. When a first attempt genuinely fails, the flow is re-run once on a stronger model before the failure is treated as real, because a first attempt can fail for agent reasons rather than app reasons. And when the app’s behavior has changed but still satisfies the intent, the verdict carries an assumption flag recording what was observed, what was expected, and why the change looks deliberate, so drift gets disclosed instead of silently absorbed.

How does each kind of check see your app?

Differently enough that “automated testing” is almost a useless phrase, and the seeing is the real dividing line between approaches. A check can only catch what it can perceive, so before comparing features, compare eyes.

Honest caveats belong next to that table. Scripted assertions are cheap to run at enormous scale and belong on every commit in a way agent runs don’t, because an agent run spends model inference and wall-clock time. Pixel-diff tools catch a class of visual regression, the one-pixel layout break, that an agent won’t notice unless a test asks a visual question. Human eyes remain the gold standard for meaning and the most expensive thing on this list to schedule. And the agent’s blindness to unwritten expectations is a real cost you pay in the currency of carefully written expected outcomes. The table isn’t a ranking. It’s an argument that these checks see different layers of the same product, and that the agent’s layer, the one your user actually inhabits, is the one that historically went unchecked between releases.

The split I’d actually run follows from those blind spots. Keep fast scripted assertions on every commit, since nothing else gives you feedback in seconds. Put pixel diffs only on the few surfaces where visual precision is the product, a pricing page, a chart library, an email template. Then point agent checks at the user journeys that matter commercially, checkout, onboarding, permissions, on a nightly or pre-release cadence, where their tolerance of UI churn pays for their slower runs. If a journey’s scripted tests have been rewritten twice this year for reasons that weren’t bugs, that journey is the first candidate to move up a layer.

Can AI replace manual testing?

No, and the useful version of that answer names what it does replace, which is the re-verification. Walking checkout again before every release. Checking login again after a dependency bump. The regression pass nobody has attention left for by Friday. That work is repetitive, fully described, and judged against known expectations, which is exactly the shape of task the scoped loop above is good at. Handing it to an agent isn’t replacing manual testing so much as admitting this slice of it stopped being testing years ago and became typing.

What can’t be handed over is the judgment. Exploratory testing, in the sense practitioners mean it, is simultaneous learning, test design, and test execution, a person building a model of the product and probing where that model is most likely wrong. The industry now pitches autonomous exploratory testing, agents roaming an app unsupervised and reporting whatever they find. The benchmark numbers earlier in this piece are one reason I’m skeptical of the roaming half. The other is that exploration without judgment mostly produces noise, and noise is what kills a QA program’s credibility long before any missed bug does.

So we drew the line for TestAutomate’s AI QA agent deliberately. Suite runs execute described flows with stated outcomes rather than wandering open-endedly, and the consequential clicks stay human. When a failed run produces a bug candidate, that candidate waits, and when a person clicks to file it, it’s first checked against your existing issues for likely duplicates. Nothing reaches your tracker without a person reviewing the evidence and clicking to file it. I want the agent supplying evidence and a human supplying judgment, in that order, every time.

Where does that leave testers? Doing the parts that were always the actual job. Deciding what matters enough to test, writing expected outcomes sharp enough to fail, reading verdicts and flags with product context no agent has, and exploring the new feature before anyone writes a test for it. If you want the system view of the whole loop, generation through graded verdict, it’s mapped in the agentic testing guide.

I’ll end with the design intent, because it’s the honest summary. I didn’t build computer use agents for QA out of a belief that agents test like humans. I built them because most of what we call manual testing had already stopped being human, and because with hard scoping, a skeptical second model on every verdict, and a person on every consequential click, the boring majority of QA can finally run every night without anyone having to pretend the demo was the product.