TestAutomate Blog

Automated UAT Testing: What It Really Takes

Thu, 10 Sep 2026 00:00:00 GMT

Every UAT cycle I have run ended the same way. The feature was done in every technical sense, the sprint was closing, and someone still had to walk the flows the way a user would and say the words that made the release real. In twenty years of QA, that last mile never modernized with the rest of the pipeline. Unit tests ran on every commit. Acceptance ran on whoever we could book into a room.

Automated UAT testing means running a feature’s acceptance criteria as tests in a real browser and producing evidence a business stakeholder can accept or reject without repeating the work by hand. The criteria stay in plain language, the run exercises the app the way a user would, and the output is a graded verdict with a replay behind it. The acceptance itself stays human. The execution, the checking, and the evidence-gathering are what automate.

That definition is doing a lot of quiet work, so the rest of what I have to say is about earning it.

What is automated UAT testing?

User acceptance testing is the last verification a change gets before people rely on it, performed from the user’s side of the glass and judged against what the business asked for rather than what the code does. It’s the one phase whose oracle is intent. A unit test knows its expected value. A regression script knows yesterday’s behavior. UAT asks a different question, which is whether the thing we built is the thing that was requested, and for most of my career the only instrument that could answer it was a person with a spreadsheet of steps and an afternoon they didn’t have.

That’s why UAT is the last-mile testing nobody automates well. The phases below it automate beautifully, because their expectations are already encoded. UAT’s expectations live in tickets, comments, and heads, so teams either skip the phase when the sprint runs long, or they perform it manually forever, or they write scripts, call the folder “uat”, and quietly turn acceptance testing into more regression testing.

So a working definition worth quoting back at any vendor, including mine. Automated UAT testing is user acceptance testing where the walking and the checking are automated and the accepting is not. If a tool automates the clicks but a stakeholder can’t read the outcome, you’ve automated regression. If it produces readable evidence against the criteria the business actually stated, you’ve automated UAT, and user acceptance testing automation stops being a contradiction in terms.

Why does UAT resist automation?

Because the three things a script needs to survive, stable steps, encoded expectations, and a willing maintainer, are precisely the three things UAT lacks. I have watched each of these kill an honest attempt, and none of them is a tooling defect. They’re properties of the phase itself.

Start with where acceptance criteria actually live. The ticket says the user can export the report. The real criteria, which formats, which roles, what happens to a scheduled export mid-change, live in the product owner’s head and get refined verbally across standups and comment threads. A script demands the full expectation up front, in code, before the first run. UAT’s expectations have never once been in that state when I needed them. Writing them down turned out to be most of the work, and the scripting was the cheap part stapled on afterward.

Then there’s churn. UAT tests the newest surface of the product, which is by definition the surface changing fastest. A regression script earns its keep over months of unchanged behavior. A UAT script covers a flow that shipped this sprint and may be redesigned next sprint, so the maintenance cost never amortizes. The scripts I saw teams build for acceptance flows were stale sooner than anything else in the suite, and the people who noticed first were never the people who could fix them.

Which is the third wall. Business users won’t maintain scripts, and they’re right not to. The industry’s best attempt at bridging this was behavior-driven development, and the Gherkin syntax really is readable by anyone. But readable was never the hard part. Every Given/When/Then file sits on step definitions that are code, and in my experience the business read the scenarios at kickoff and never again, while ownership of the whole layer drifted to QA all the same. My decision rule from those years is blunt. If the UAT plan requires a business owner to open a code editor, it’s a QA plan wearing a costume.

What does automation-ready UAT look like?

Three properties, and you can build toward all of them before choosing any tool. Acceptance criteria written as testable outcomes, evidence a stakeholder can read, and verdicts that separate product failures from environment failures. A UAT process with those three properties can be automated by several means. A process without them can’t be automated by anything, because there is nothing yet for automation to check.

The first property is the cheapest and the most neglected. Each acceptance criterion becomes an observable end-state, phrased so a stranger could verify it. Not “export works” but “after choosing CSV, a file downloads containing the rows currently filtered, and the saved report is unchanged”. I’ve written before about how to write expected results a verifier can grade, and the discipline is identical here because acceptance criteria are expected results, just usually unwritten ones. This is also where TestAutomate takes a hard line I agree with. A test in its format is three required fields, an id, a prompt, and an expected outcome, and the generation rules order the author to keep both halves in plain language, verbatim “Do NOT include code, JSON, YAML, or selectors”. The business-readable artifact is the artifact. There’s no compiled script underneath drifting away from it, which is the failure that ate every Gherkin suite I watched decay.

The second property is evidence. A sign-off decision is only as trustworthy as what’s in front of the person making it, and a green checkmark is not evidence, it’s a request for faith. Automation-ready UAT produces something a stakeholder can inspect at their own level, a replay of the run they can scrub through, an itemized list of which expectations were met and which weren’t, and an export they can attach to the sign-off record. Plain-language tests matter here twice, since the same phrasing that lets the business author criteria also lets them audit the result.

The third property is the one that decides whether anyone still trusts the system in month three. A UAT run fails for many reasons that aren’t product defects. Staging was mid-deploy, test data was missing, an account lacked a role. Automation that reports all of these as red teaches stakeholders to ignore red, and then the one red that matters ships. TestAutomate’s graded verdicts treat this as a first-class distinction. A run that couldn’t establish its precondition is recorded as blocked, stored as neither pass nor fail, and a blocked run never counts as a failure and never produces a bug candidate. I’ve written up why tests get blocked instead of failed at length, because in my years of triaging morning-after reds, misfiled environment noise was the single biggest destroyer of confidence in automation.

Here’s that distinction on a real recorded run from our own dashboard. Run 36 ended blocked, one test out of one, in 1m 3s, because setup couldn’t establish the precondition, and the verifier’s reasoning states outright that the outcome is neither a pass nor a failure and points at the test environment, not an app defect.

An acceptance decision built on that vocabulary can be quick without being careless, which is the whole promise of automating this phase.

How does sprint UAT run while you sleep?

It runs as a loop that starts from the board, because in a sprint the board already says what needs accepting. TestAutomate treats sprint UAT as a sweep over a board query. You point it at a JQL filter, say every issue sitting in a done-but-unverified status, and that query becomes a durable work queue, each matching ticket enqueued exactly once, with progress that survives interruptions instead of restarting the sweep from zero. The Jira integration is built around this shape because it matches how teams actually close sprints, ticket by ticket, not suite by suite.

For each ticket, the QA agent opens the app in a real browser, with real session state, and exercises the behavior the ticket describes. Then it records one of five graded verdicts, fixed, reproduces, blocked, inconclusive, or error. The bar for “reproduces” is that the agent saw the exact defect described, and it’s instructed to quote exactly what it saw and to record the verdict while the failing state is still on screen, at which point a screenshot is captured automatically as visual proof for the report. Anything the environment prevented, a login that failed, a role it wasn’t given, ends blocked rather than failed, same as everywhere else in the product.

What happens next is where most tracker integrations stop and this one keeps going. A defect becomes a bug candidate, and before anything reaches your board it’s checked against a pool of existing issues drawn from live tracker search plus a cached inventory, up to fifteen existing issues judged for duplication, so the sweep doesn’t bury a triager in copies of the same regression. Nothing files itself. Filing is a reviewed click in the dashboard, always. And when a developer marks the bug fixed, the bug loop sends the ticket back through the queue for re-verification, so “fixed” is a verdict again instead of a hope. The sweep ends in a sprint report, delivered by email, summarizing what was verified, what reproduces, and what was blocked.

The economics are governed rather than open-ended. The shipped defaults give the agent twenty-five turns per issue, estimate roughly thirty-five cents of model spend per ticket before the job starts, and cap a job at ten dollars unless you raise the ceiling, so the overnight sweep can’t wake you with a bill instead of a report. Those numbers are the code’s defaults, not benchmarks, and you can change every one of them.

That’s the honest meaning of UAT while you sleep. The queue drains overnight. The humans arrive to verdicts, replays, deduplicated candidates, and a report, and the work remaining is the one part that was never automatable, deciding.

How do you adopt UAT automation without a rewrite?

Start with the artifact, not the tool, and the sequence below works whatever you buy or build. I’ve rolled variations of it out by hand, long before agents existed, and every step pays for itself even if you stop there.

Rewrite acceptance criteria as observable outcomes on next sprint’s tickets. Minutes per ticket, no tooling, and the humans doing UAT this sprint already benefit. This step surfaces the criteria that were living in heads, which is most of them.
Pick the one flow that appears in every UAT cycle and make its evidence readable. However it’s executed, the artifact a stakeholder sees must answer what was checked and what happened, without a QA translator in the room.
Impose a three-way verdict before anything reaches a stakeholder. Product defect, environment problem, or test problem. Do it manually in triage if you must. The day reds stop meaning three different things is the day people start trusting them.
Close the loop. A finding that never becomes a tracked ticket evaporates, and a fix that never gets re-verified is a rumor. Both directions need an owner.
Only then point automation at the board. The sweep is the last step, because automation amplifies whatever process it lands on. Automating a UAT process with unwritten criteria and unreadable evidence just produces faster confusion.

The decision rule I’d leave you with is the one I apply to our own product. Judge any automated UAT testing setup by what it hands the person who signs. If they can see the criteria, watch the evidence, and tell a product failure from a staging hiccup without asking an engineer, the automation is doing UAT. Anything less is regression testing with better marketing, and the last mile stays exactly where I found it twenty years ago, on foot.

Real Browser Testing vs Cloud Browsers

Thu, 10 Sep 2026 00:00:00 GMT

While I was building TestAutomate’s extension execution path, I kept two browsers side by side. One was my own Chrome, signed into everything. The other was a fresh profile spun up for clean-slate checks. The same test against the same staging app told two different stories. My Chrome walked straight into the product. The fresh profile met a consent banner, a login page, and a bot challenge. Nothing about the app changed between those runs. Only the browser did.

Real browser testing means executing automated tests in a browser environment that matches what your users actually have: a real profile with live logged-in sessions, real cookies and extensions, and a network position inside your perimeter. The industry blurred the term by using it to mean a real rendering engine on a vendor’s cloud machine, which gets the pixels right while everything around the engine, the sessions, the identity, the network, stays synthetic.

That distinction sounds pedantic until you notice it decides which failures your tests can even encounter. So I want to lay out the full taxonomy of where browser tests actually execute, what each environment can and cannot see, and how the difference surfaces in your results.

What does real browser testing actually mean?

It means the browser executing your tests is provisioned like a user’s browser, not merely compiled from the same source code. Four properties make an environment real. Profile state, meaning the cookies, storage, and sessions the browser carries. Identity, meaning who the app believes is signed in. Network position, meaning whether the browser sits inside or outside your perimeter. And fingerprint, meaning how login defenses and bot detection read the browser when it knocks on the door.

The industry blurred the term for an understandable reason. When cross-browser platforms coined “real browser testing” years ago, the enemy was simulators, and “real” meant a genuine engine instead of an approximation of one. That was the scarce thing at the time, because rendering differences between engines were the bug class everyone chased. The term stuck while the scarce thing moved. Apps went behind SSO. Feature flags started splitting the product per session. Bot defense hardened the front door. Engine parity became cheap, environment parity became rare, and the marketing phrase kept describing the part that no longer decides outcomes.

So when a vendor page says tests run in “real browsers,” read it precisely. It’s telling you about the rendering engine, and usually about nothing else. The browser is real the way a rental car is a real car. It drives, but none of your things are in it, and your garage won’t open for it.

Where does automated browser testing actually run?

In one of four places. A headless browser inside a CI container, a browser on a vendor’s cloud grid, an emulated device inside one of those, or the Chrome on your own machine. Every automated browser testing result you’ve ever read was produced in one of these environments, and each one sees a different slice of your app.

Headless browsers in CI are the workhorse of the category. The engine argument here is settled, since Chrome’s headless documentation describes modern headless as the same browser code as regular Chrome rather than a separate implementation. What a headless CI browser can’t see is everything a profile accumulates. It’s born empty on every run, holds no session, carries no extensions, and reaches only what its container can route to. That emptiness is a genuine feature for deterministic unit-style checks, and a genuine wall for anything gated by identity.

Vendor cloud grids run real browsers on real operating systems in the vendor’s datacenter. What they can see is breadth, meaning dozens of engine and OS combinations on demand. What they can’t see is your environment. VPN-only staging needs an allowlist hole or a tunnel agent before the first test runs. Your SSO session doesn’t exist on their machines. Neither do your extensions, your account’s data, or your feature-flag cohort. And reputation systems long ago memorized datacenter IP ranges, which is why login pages greet grid browsers with extra suspicion. I’ve written separately about what that does to automated testing behind login walls.

Emulated devices are the layer most often mistaken for hardware. Device emulation approximates a viewport, a user agent string, and touch events on top of a desktop browser, and Google’s own device mode documentation calls it a first-order approximation that shouldn’t replace testing on real devices. An emulated phone also inherits the environment of whichever host browser runs it, so it stacks an approximation on top of whatever that host already couldn’t see.

Your own Chrome inverts all three. The sessions are live because you signed in this morning. The staging environment is reachable because your machine sits behind the VPN. The extensions are installed, the feature flags resolve for your real account, and bot defenses see an aged, recognized profile rather than a stranger. The trade is that it’s one browser, on one OS, on one machine, and it’s attended by default.

Environment	Engine	Sessions and SSO state	VPN-only staging	Extensions	Feature-flag cohort	How bot defenses read it
Headless in CI	Real	None, starts empty each run	Only if CI is inside the network	None	Anonymous default	Automation fingerprint, container IP
Vendor cloud grid	Real	None, fresh profile per session	Allowlist hole or tunnel required	Rarely supported	Anonymous default	Known datacenter IP ranges
Emulated device	Real engine, approximated device	Inherits its host browser	Inherits its host	Inherits its host	Anonymous default	Emulation adds detectable tells
Your own Chrome	Real	Live, already signed in	Reachable, your machine is inside	Installed and active	Your real account’s flags	Aged, recognized profile

Why does the execution environment change test results?

Because a verdict is a function of two inputs, the app and the environment, and when the environment differs from a user’s, the verdict quietly starts describing the environment instead. Four failure shapes account for most of what I’ve seen.

Login walls come first. A fresh browser must authenticate from zero on every run, and modern authentication is built to resist exactly that. Google’s OAuth policies require sign-in to happen in a secure, standard browser context and refuse contexts judged automated. Enterprise SSO layers conditional access on top, judging the device and the network before your test sees a single pixel of your app. A grid browser fails those judgments for structural reasons, not flaky ones.

Consent and cookie state comes second, and it cuts both ways. Your users cleared the consent banner months ago, so a fresh profile meets an overlay your users never see, and the test’s first click lands on it. Meanwhile a test in your own Chrome never meets the banner at all, which leaves the first-run experience untested there. Neither environment is wrong. They’re answering different questions, and the damage comes from not knowing which question you asked.

Feature flags are the quiet one. Flag targeting by user, org, or rollout cohort means an anonymous fresh session gets the default experience while your real account gets the new one. Same URL, two different products. A test that passes in one cohort and fails in the other isn’t flaky. It’s measuring the flag, accurately, and reporting the measurement as if it were a bug.

Bot detection is the loud one. A datacenter IP plus an automation fingerprint earns challenges, and a challenged run dies at the front door having proven nothing about your app.

What turns these from annoyances into damage is binary grading. A tool that only knows red and green reports every one of them as a failing test, and the team learns to distrust red. TestAutomate grades the environment problem as its own outcome instead. Every setup phase has to end by declaring one of two states, ready with a description of the state that now exists, or blocked with the reason it couldn’t be established, and a precondition that can’t be established ends the run blocked, never failed, and never files a bug. I’ve covered why tests get blocked instead of failed in its own post. Below is a real recorded run from our dashboard. Run 36 ended blocked in 1m 3s because setup couldn’t establish the test’s precondition, and the verdict says in as many words that this is a test-environment problem, not an app defect.

How does TestAutomate run tests in a real browser?

By driving the Chrome you already use, through an extension, so the test executes inside your live session, behind your VPN, on your real staging data. There’s no browser to provision because the browser already exists, and no session to construct because you constructed it yourself by showing up to work.

Most browser automation testing stacks assume they own the browser, launching a fresh instance they fully control. TestAutomate inverts that assumption. The server dispatches 18 named browser tools over a WebSocket to the extension, and the extension attaches Chrome’s debugger protocol, version 1.3, to the tab under test. After every navigation and click, the runner reads the page’s interactive elements back as text, capped at 1,500 characters with a 300 millisecond settle after clicks, and the agent acts on the element references it just read. No user-authored selectors exist anywhere in the test artifact. The real-browser execution model page walks through the full architecture.

This matters for AI browser testing in particular, because an agent decides each action from what the page currently shows, which is what separates agentic testing from replaying a recorded script. But intelligence doesn’t change the physics. An agent in a locked-out environment is exactly as locked out as a script. Where the test runs decides what the agent can reach, which is why I treat execution environment as the first question to ask about any tool in this category, ahead of authoring and ahead of reporting.

The second path covers the runs nobody watches. Scheduled and unattended runs execute on a self-hosted runner inside your network or on cloud browsers, streaming a WebRTC live view you can click into and take over when an SSO prompt or a 2FA challenge needs human hands for a moment. Sign-in for those runs comes from the Auth Catalog, named auth entries per app covering email and password, Google, Okta, magic links, and TOTP codes, encrypted per org and seeded from what attended runs already learned about each app’s login path. The two paths are one system. Your attended runs in real Chrome teach the login quirks, and the unattended runs inherit them instead of rediscovering them at 6 a.m. on a Saturday.

One boundary stays firm on both paths. TestAutomate doesn’t solve CAPTCHAs or defeat bot detection. Running as a recognized browser means challenges appear far less often, not never, and when one does appear, the mechanism for passing it is a person, by design.

When is cloud browser testing the right tool?

When breadth is the question. A browser on your machine can’t be twelve browsers across six OS versions, and it can’t be a rack of physical phones. If you ship to the public and rendering fidelity across Safari, Firefox, and Edge versions matters, a cross-browser grid is the tool built for that job. If your bug class lives in real mobile hardware, a device farm is the honest way to reach it. High-parallelism runs against a public app with tame authentication are grid territory too. None of this is grudging. Those are real bug classes, and environment-depth tooling doesn’t address them.

The decision rule I’d offer is to pick by question, not by product category. Engine-breadth questions, meaning does this render and behave correctly across browsers and devices, belong to a grid. Environment-depth questions, meaning can a signed-in user on our VPN-gated staging complete this flow today, need a browser that actually lives in that environment. Plenty of teams need both answered. The recurring mistake is buying a tool built for one question and reading its results as answers to the other, which is how a suite stays green while signed-in users are stuck, or stays red for weeks over a consent overlay no real user ever sees.

I’ve written a fuller side-by-side of the cloud AI testing platforms as a category, and if you’re evaluating the agentic end of the market, the criteria in the best agentic QA tools roundup are the ones I’d actually bring to a vendor call.

The two browsers on my desk settled the definitional question for me long before I could articulate it. The app was the same app in both runs. The verdicts disagreed because the environments did, and every test report you read is downstream of that choice. The app is what it is. What your tests say about it depends on where they were standing when they looked.

Best Agentic QA Tools in 2026: The New Category, Mapped

Tue, 08 Sep 2026 00:00:00 GMT

A redesign once wiped out a large share of the Selenium locators I maintained, in a single merge. The product still worked. Every user journey I was paid to protect functioned exactly as before. The tests couldn’t see any of that, because tests built from selectors don’t watch the product, they watch the DOM, and the DOM had moved. I spent the days that followed repairing strings while the app sat there, healthy and unverified.

That gap is what agentic QA tools exist to close. An agentic QA tool puts an AI agent, not a script, in charge of the run. You state a goal and an expected outcome. At run time the agent reads the live application, decides each action against that goal, adapts when the UI has shifted, and hands the result to a judge that grades it against the stated expectation. A scripted tool replays steps that were written in advance. An agentic tool decides its steps while it runs. That runtime difference is the whole category.

In 2026 the category finally has enough serious entrants to be worth mapping. Here are the seven I’d actually put in front of a team, graded on one frame, with my own product held to it hardest.

What separates an agentic tool from an AI-assisted one?

The test is what happens at runtime, not what happens at authoring time. Plenty of tools now accept plain English and use AI to generate tests, but if that English compiles into stored element steps, selectors, or framework code that then replays verbatim, you’ve bought a script with a nicer front door, and the vendor’s self-healing machinery exists to patch it when it breaks. A genuinely agentic tool keeps the intent as the artifact and re-derives the actions against the live page on every run, so there’s no stored binding to decay.

That’s why “self-healing” is usually the tell. Healing presumes something bound to the old page was stored. Ask any vendor two questions. What exactly is saved after I author a test? And what does the software do when my UI changes but the flow still works? The answers sort this market faster than any feature grid. I’ve written a longer version of this argument in our complete guide to agentic testing, and the category’s pillar page covers the vocabulary.

Who’s writing this list, and how the grading works

We build TestAutomate, the first tool below. This is a vendor’s list, and you should read it as one. Three things keep it useful anyway.

First, the grading frame is stated and applied to everyone, including us. Four questions: what does the tool store as the test, whose browser executes it, what can a verdict actually say and who confirms a failure, and what does the meter count. Second, every competitor entry here is grounded in a full head-to-head page with dated public sources, “not documented” written where a vendor’s docs are silent, and a standing invitation to tell us what we got wrong so we can fix it. Third, we’re pre-launch and say so plainly. Each entry below names what that tool is genuinely best at, because a list where the author wins every row is an ad, not a map.

We list ourselves first because it’s our list. The honest counterweight is that our entry carries its own “where we lose” lines, and the choosing section routes several kinds of buyer away from us by name.

The seven agentic testing tools at a glance

All pricing on this page, in this table and in every entry below, is as stated on each vendor’s public pages as of September 1, 2026, from our comparison research. Sales-led vendors publish no dollar figures, and we say so rather than guess.

Tool	What authors tests	Where tests execute	Verdict model	Pricing model
TestAutomate	Plain-English intent (prompt + expected outcome in YAML); draft suites from docs, repo, and runtime	Your own Chrome; self-hosted runner or cloud browsers for unattended runs	Pass / fail / blocked, advisory flags, a stronger model re-drives every failure before it’s reported	BYOK at provider list price, zero markup; managed keys at provider price + 15%, both numbers shown
Momentic	Plain-English YAML step lists in your repo; element cache in Momentic’s cloud	Momentic-hosted Playwright browsers, or local CLI (needs their API key + cloud cache)	Heal / warn / fail classifications plus quarantine	Free tier, then $125/mo for 10,000 credits, $0.01875 per overage credit
TestSprite	Natural-language test plans compiled to Playwright/Cypress code	Their cloud; the CLI is a client to it	Pass / failed / blocked; Auto-Heal marks recovered runs Passed with a note	Free 150 credits/mo; Starter $19/mo from month 2; Standard $69/mo for 1,600 credits
Autonoma	Planner CLI reads your codebase, drafts markdown tests you review	Their cloud builds an isolated preview per PR; free full self-host	Six verdicts (Passed, Bug, four noise classes) with published trust-rate math	100K credits free, then $100 per 150K credits; credit contents unpublished
Checksum	Agents generate Playwright from repo analysis, app scanning, optional session recording	Your CI runners or their cloud; generation always in their cloud	Pass / fail / healed, where healed means the agent edited the test	Sales-led, per maintained workflow; no public dollar figures
TestMu AI (KaneAI)	Natural language, recorded click-throughs, or imported Jira/PRD, stored as versioned steps + code view	HyperExecute cloud (60+ regions); Tunnel for private networks; free local Kane CLI	Pass / fail with step semantics; auto-heal, retries, test muting	Credit-metered, KaneAI from $17/mo on annual billing
QA Wolf	Their AI plus human QA engineers write Playwright/Appium	QA Wolf’s managed cloud; VPN reached via tunnels	Passing / failing, every failure human-triaged within 24 hours	Platform 1¢/AI credit + 15¢/runner minute; managed service custom-priced

1. TestAutomate (ours, pre-launch)

Full disclosure again: this is our product, and it hasn’t launched. TestAutomate stores a test as intent, a prompt plus an expected outcome in versionable YAML. No selector layer exists anywhere in the artifact, not in the file and not in any cloud cache, and the intent is never compiled into stored element steps or framework code. The agent drives your actual Chrome, with your real logged-in sessions, behind your VPN and SSO, on real staging data. Verdicts grade the app rather than the test: pass, fail, or blocked, where a broken environment can never register as a false fail, and no failure reaches you until a stronger model re-drives the run and agrees. Model spend is BYOK at your provider’s list price, never marked up, with a live dollar meter and a hard budget ceiling. Where we lose, plainly: no mobile or API coverage today, no SOC 2 attestation this quarter, and if your org forbids browser extensions and won’t run a self-hosted runner, our execution model needs one or the other. Judge the claims against a real graded run report, not this paragraph.

2. Momentic

Momentic is the most serious dev-native tool on this list. Tests are plain-English YAML files in git, one CLI runs them on a laptop and in CI, and JUnit, Allure and Playwright-JSON reporters meet engineers where they already work. Pricing is genuinely public, a rarity here: a free tier, then $125 a month for 10,000 credits with $0.01875 per overage credit. Hosted Android emulators and iOS simulators sit on the same public price list. The architectural catch is where the selectors went. The step list in your repo reads clean, but the element knowledge that resolves each step lives in an org-scoped cache in Momentic’s cloud, expiring 14 days after last save, and the heal/warn/fail ladder exists to keep that machinery upright. Every rung of it answers a question about the test, not about your application. Pick Momentic if your app is fully public, your team wants tests run exactly like unit tests, or you need mobile emulator coverage you can budget from a public page today. Our full head-to-head, with sources: TestAutomate vs Momentic.

3. TestSprite

TestSprite runs the deepest coding-agent loop shipping today. Through its MCP server it’s wired into Cursor, Claude Code, Windsurf, VS Code, Copilot and Trae: generate tests, run them, hand the failure bundle to the agent that wrote the code, patch, rerun. Backend and API testing is first-class rather than a bolt-on, and the cold start is real, with a PRD, URL, or OpenAPI spec in and structured test plans out. Pricing starts at a free 150-credit tier, with Starter at $19 a month from the second month and Standard at $69 for 1,600 credits. The catch sits downstream of the plan. The spec compiles into executable Playwright or Cypress code that runs on their cloud, and their Auto-Heal layer exists to absorb the drift generated code accumulates, marking recovered runs Passed with a note, automatically once enabled, with no approval step. VPN and private-network reach appear nowhere in their documentation. Pick TestSprite if the main consumer of a failure is the coding agent in your IDE, or if API coverage gates the decision. The full page: TestAutomate vs TestSprite.

4. Autonoma

Autonoma is the most transparent platform we profiled. Six named verdicts with published trust-rate math, where environment failures count as noise instead of failures and a confirmed bug raises suite health. An Environment Factory builds an isolated full-stack preview per PR and seeds fresh test data, which most of this category doesn’t document at all. Self-hosting is free with no feature limits, the test planner is MIT-licensed, and pricing is usage-shaped: 100K credits free, then $100 per 150K credits, though what one credit buys isn’t published anywhere we could find. The structural catch is the source of truth. The planner generates tests from your codebase, so expectations are inferred from the implementation they’re supposed to judge, with your review checkpoints as the corrective. Private-network reach isn’t documented either, and the route onto your own network is self-hosting the whole platform. Pick Autonoma if you live in GitHub PRs, want seeded per-PR environments, need mobile via Appium, or must self-host at zero license cost. The full page: TestAutomate vs Autonoma.

5. Checksum

Checksum generates real Playwright into your repo, verified by actually running it before the PR opens, with coverage grounded in repo analysis, live-app scanning, and an optional SDK that records real user sessions. The export story is among the category’s best: standard Playwright atop an open-source runtime package, yours to keep if you leave. On failure, their triage agent decides broken test versus real bug and tags genuine bugs in source, with roughly 70% of failures resolving autonomously, by their own figure. Two catches. The verdict grammar is pass, fail, healed, and healed means the agent edited the test until it passed, assertions included, reviewed afterward as a PR diff, so the maintenance burden becomes code review of edits to your safety net. And every documented input is the implementation itself, which means a feature that shipped wrong on day one produces tests that enshrine the wrong behavior. Pricing is sales-led, per maintained workflow, with no public dollar figures. Pick Checksum if your team is all-in on Playwright, or your org forbids extensions and lives in CI. The full page: TestAutomate vs Checksum.

6. TestMu AI (KaneAI)

Don’t read TestMu AI as a challenger brand. It’s LambdaTest, renamed in January 2026, with three million users and 18,000 enterprise customers by its own count, plus 10,000+ real devices and 60+ execution regions. KaneAI authors tests from natural language, a recorded click-through, or an imported Jira ticket or PRD, and its export is the best in the category: Selenium, Playwright, Cypress or Appium code, delivered as a pull request into your own repo. Kane CLI runs a local Chrome free. The architecture underneath is a binding lifecycle. Recorded steps and locators sit beneath the English, self-healing updates them when the app drifts and surfaces the diff for review, and the noise toolkit includes Test Muting, which stops grading a test rather than grading it. Reaching private staging means installing their Tunnel client inside your network so an outside browser can get in. Pricing is credit-metered, from $17 a month on annual billing. Pick TestMu AI for a real-device mobile matrix this quarter, or one-vendor enterprise breadth. The full page: TestAutomate vs TestMu AI.

7. QA Wolf

QA Wolf is the managed-outcome reading of the same promise. Instead of selling you an agentic tool, they sell the result: AI plus human QA engineers write and maintain Playwright and Appium tests in their cloud, and humans triage every failure into bug versus maintenance within a 24-hour window, so verified bugs arrive human-reproduced with a video walkthrough. Their service page publishes a guarantee of 80%+ automated coverage in under four months, their changelog announced SOC 2 Type II and HIPAA attestations in November 2025, and the device farm covers physical iPhones and iPads. The costs are structural rather than hidden. Scheduled runs execute on QA Wolf’s infrastructure, so reaching your VPN is a networking project (OpenVPN, IPSec, Tailscale, Twingate, static IPs), and the human confirmation layer that makes results trustworthy can’t export with the code. Vendr’s data across 58 recorded purchases puts the median at $83,100 a year, and the self-serve platform meters 1¢ per AI credit plus 15¢ per runner minute. Pick QA Wolf to buy the outcome without adopting a tool. The full page: TestAutomate vs QA Wolf.

How to choose between agentic QA tools

Choose by your hardest constraint, not by feature count. These are the decision rules I’d apply, and they don’t all point at us.

If your app lives behind a VPN or enterprise SSO, the browser’s location decides everything else. TestAutomate starts inside the wall in your own Chrome. Momentic’s local CLI and Checksum’s your-own-CI mode get a browser inside too. TestMu AI needs its Tunnel installed, QA Wolf needs VPN plumbing configured, TestSprite doesn’t document a route at all, and Autonoma’s only documented route inside is self-hosting the whole platform.

If you need mobile coverage today, we’re not your tool. TestMu AI’s real-device fleet, QA Wolf’s physical iPhones and iPads, Momentic’s hosted emulators, and Autonoma’s Appium support are all shipping. The same honesty applies to API testing, where TestSprite and Checksum are ahead of us.

If your team has no tracker discipline, a close-the-loop bug workflow has nothing to close. QA Wolf’s humans hand you narrated bug reports regardless, and TestSprite’s loop terminates usefully in your IDE.

If your engineers want free, code-native generation, Autonoma’s MIT planner and TestMu AI’s Kane CLI cost nothing to try this afternoon.

If the AI bill itself is the concern, look at what the meter counts. Credit units bundle model spend invisibly, which is most of this list. We built the economics the other way, passing model spend through at provider cost on your own key, and that difference compounds with volume.

One last calibration. Nobody, us included, has published a benchmark of verdict accuracy for these systems, so every “the AI decides” claim on this page and every vendor’s page rests on mechanisms, not measured grading quality. Read the full comparison grid with that in mind, follow the dated sources on each head-to-head, and when you find something we got wrong about a competitor, tell us. The correction invite on every vs page is real, and taking it up makes this map better for the next team.

The Best AI Testing Tools in 2026, Compared Honestly

Tue, 08 Sep 2026 00:00:00 GMT

We spent the last few weeks reading the public documentation of nine AI testing platforms, end to end, to build the head-to-head comparison pages on this site. Somewhere around the fourth vendor’s self-healing doc, a pattern locked in for me. Nearly every vendor puts plain English on top, nearly every vendor stores something else underneath, and the something else is what you end up maintaining.

So this is our list of the best AI testing tools in 2026, graded on what each one stores, where its browser runs, and what its verdicts are allowed to mean. In short, testRigor wins on platform breadth in one language, mabl is the most mature AI cloud, TestMu AI has the biggest published device fleet and best export, Testim the strongest flake governance, Katalon the widest all-in-one IDE, Testsigma the only open-source core, Functionize the deepest enterprise scale, Autify the most complete private-network menu, and Rainforest QA the only human-backed model. Our own TestAutomate is built for testing web apps from the browser that’s already logged in.

How we graded the best AI testing tools

Disclosure first. We build TestAutomate, this list lives on our site, and no roundup that includes the author’s product should be read as neutral. So instead of pretending, we’re publishing the grading frame and applying it to everyone, ourselves included. Four questions decide every entry. What does the tool store when you author a test? Where does the browser that runs your suite live, and what can it reach? What is a verdict allowed to mean beyond red and green? And how is the meter priced?

Two more things you should know before trusting a word of it. TestAutomate is pre-launch, and every entry below says plainly where a shipping rival beats us today. And every competitor claim on this page links to a full head-to-head where each fact traces to that vendor’s own documentation and pricing pages as of a stated review date, September 1, 2026 for all nine. Where a vendor’s docs are silent, those pages say “not documented” rather than guessing, and each one carries a standing correction invite. If we got a fact wrong, tell us and we’ll fix it.

The verdict question is the one I care about most, because it decides who does your morning triage.

That’s a real run from our recorded QA dataset. Run 36 ended blocked in 1m 3s because setup couldn’t establish the precondition, so the app was never exercised and nothing got reported as a failure. On a red/green dashboard, that same minute produces a false red for a human to investigate. The deepest split in this whole category is between tools that heal a stored artifact and tools that never store one, and we’ve written a fuller piece on self-healing versus selectorless testing if you want the mechanism.

AI testing tools compared at a glance

The table compresses each tool to the four grading questions. Vendor rows come from their own docs and pricing pages as of September 1, 2026, via the linked comparisons below.

Tool	What a test is	Where it executes	Verdict model	Pricing model
TestAutomate (ours, pre-launch)	Plain-English intent in versionable YAML, no selector layer	Your own Chrome, plus self-hosted or cloud runners for unattended runs	Pass / fail / blocked, advisory flags, labeled retry passes	BYOK at provider cost, or managed keys at provider price + 15%
testRigor	Plain English over an element association recorded on first successful run	Their cloud, on-premise offered	Pass or fail, with retries	Free public tier, private from $300/mo
mabl	Recorded or generated steps on a learned element model	mabl Cloud (local and CI runs are limited)	Passed / failed / passed-with-warning	Quote-only, expiring credits
TestMu AI (KaneAI)	Versioned natural-language steps synced to a code view	HyperExecute cloud, free local Chrome via Kane CLI	Pass or fail, plus test muting	Credits, from $17/mo annual
Testim	Recorded steps with Smart Locator fingerprints per element	Local editor runs, scheduled runs on their grid only	Pass/fail with flaky states and quarantine	No public prices, parallel slots
Katalon	Object Repository locators plus Groovy scripts	Studio locally, paid Runtime Engine for CI, TestCloud	Pass/fail with retries and flake scoring	Studio $180/seat/mo, KRE $182/license/mo
Testsigma	Templated English bound to stored element records	Their cloud grid, or a local Java Agent	Passed / failed / not executed	Quote-gated, open-source core is free
Functionize	Ordered steps bound to ML element models built in their cloud	Their cloud, one VM per test	Green / red / yellow (healed) / purple	Studio $0 to $200/user/mo, enterprise quote
Autify	Recorded steps (NoCode), compiled steps (Nexus), plain English (Aximo)	Their cloud, Nexus can run locally	Pass/fail plus “Recovered with AI”	Nexus $400/mo, Aximo credits $99 to $450/mo
Rainforest QA	Fixed actions targeting stored screenshots of your UI	Rainforest-hosted VMs	Pass or fail, then human triage	No public prices

The 10 best AI testing tools in 2026

1. TestAutomate (that’s us)

TestAutomate is our product, it’s pre-launch, and it sits at the top of this page because we built the page, not because we out-graded nine shipping rivals. Judge us with the same frame as everyone else. A test stores plain-English intent, a prompt plus an expected outcome in versionable YAML, with no selector layer anywhere in the artifact. The agent drives your actual Chrome, so real logged-in sessions behind VPN and SSO are the default rather than a tunnel project, and how execution works covers the unattended paths. Verdicts are graded pass, fail, or blocked, so a dead staging server can never be a false fail, and no failure is reported until a stronger model re-drives the run and agrees. Model spend is bring-your-own-key at your provider’s list price with a live dollar meter, and a real graded run report shows what lands after a run.

Where we lose today, plainly. We’re web-first, so mobile and API coverage belongs to others on this list. Our execution model needs a browser extension or a headless runner, and some orgs forbid both. We can’t hand you a SOC 2 attestation this quarter. A dev-only team that wants code-native output is better served by Kane CLI’s free local runs or Autify Nexus’s Playwright export. And if your team has no tracker discipline, our close-the-loop bug workflow has nothing to close.

2. testRigor

testRigor is the most credible name in natural-language testing, and it earned that. One English grammar drives web, native iOS and Android, Windows desktop, API checks, email and SMS flows, even mainframe, which is breadth nobody else on this list matches from a single language. The free-forever tier for public test suites is a generous on-ramp, and when its AI repairs a step, the step is labeled fixed-by-ai rather than silently patched. The architectural catch sits in their own self-healing page: on the first successful run the system internally records an element association, and later runs replay that binding until it breaks and AI re-binds it. The English is the label, the binding is the test. Execution defaults to their cloud, with IP whitelisting, tunneling, or on-premise as the documented routes into a VPN’d environment. Pick it for mobile-native or desktop coverage this quarter, or for a manual QA team that wants a recorder. Pricing starts free for public suites, with a private tier from $300 a month per their sign-up page as of September 1, 2026. Our full testRigor comparison carries every source.

3. mabl

mabl is the most mature trained-model platform in the category. Browser, native mobile, API, performance, accessibility, and visual testing share one platform and one credit pool, concurrency is documented up to 1,000 browser tests, and it holds a SOC 2 Type II attestation. Its healing is the sophisticated version of the idea, with per-environment element histories and confidence gates, so a low-confidence match fails rather than guesses. The structural limits come from the same design. Natural-language authoring generates steps that a learned element model must then maintain, the custom CSS and XPath escape hatches can’t auto-heal by mabl’s own docs, and the full product lives in mabl’s cloud, with local and CI runs limited to Chrome-only, pass/fail-only artifacts. VPN’d staging is reached by deploying the mabl Link tunnel, every cloud run starts logged out, and pricing is request-a-quote with expiring credits, per their pricing page as of September 1, 2026. Pick it if you want breadth plus vendor attestation and your org is happy with cloud execution. The full mabl comparison has the receipts, including the lossy export list.

4. TestMu AI (KaneAI)

TestMu AI is not a challenger brand. It’s LambdaTest, renamed in January 2026, with three million users and 18,000 enterprise customers by its own count, plus a published fleet of 10,000+ real devices and 3,000+ browser environments. KaneAI drafts a test plan you review before anything executes, and its export story is the best in the category, generating Selenium, Playwright, Cypress, or Appium code and opening the pull request into your repo itself. Kane CLI even runs a local Chrome for free. What the English sits on is still a binding, though. KaneAI stores versioned natural-language steps synced to a code view, and self-healing updates affected steps and surfaces the diff for review, which is exactly the hygiene you need once a stored artifact mutates with your app. VPN’d apps are reached through their Tunnel client. Pricing is credit-metered from $17 a month on annual billing as of September 1, 2026, with operations like auto-heal priced per step. Pick it for a real-device matrix or a dev team that wants code-native output. Sources live in the full TestMu AI comparison.

5. Testim

Testim has spent over a decade making recorded tests break less, and since 2022 it’s part of Tricentis. Its flake governance is the strongest we found documented in the category, with a precise flaky-test definition, a formal Draft, Evaluating, Active, and Quarantine state machine, and retries visibly marked in run history. Its Salesforce tooling, from metadata locators for Lightning to testing against upcoming Salesforce releases, is a real moat, including against us. The artifact is a recorded step sequence with a proprietary Smart Locator fingerprint per element, auto-corrected by attribute re-scoring until a human repairs it in the editor, and Copilot’s natural-language authoring emits JavaScript steps. Scheduled runs execute only on their cloud grid, and their pro-plan tunnel is documented as unsupported for scheduled runs, which is precisely the run you care about most. Every pricing tier says contact us as of September 1, 2026. Pick it for Salesforce, for mobile grids, or when procurement wants a twelve-year vendor. The full Testim comparison quotes their docs verbatim.

6. Katalon

Katalon is the widest single IDE on this list, covering web, API, mobile, and Windows desktop, with TestOps layered on top for genuine manual test management and a rare two-way Jira sync. Studio is free for local authoring and execution, and the recorder-to-Groovy ramp works well for mixed-skill QA teams. What a recording saves is an Object Repository of locators plus a Groovy script, and Katalon’s self-healing swaps in backup locators, then has an LLM propose replacements that you approve after the run. The maintenance gets batched, not removed. The other catch is the meter. Running tests from the command line, which is what CI is, requires the paid Katalon Runtime Engine at $182 per license per month, on top of Studio seats at $180, per katalon.com/pricing as of September 1, 2026. Reaching VPN’d staging from TestCloud means configuring their tunnel, with per-suite setup as their stated best practice. Pick it for a mixed manual-plus-automation org that needs all four surfaces in one place. Details sit in the full Katalon comparison.

7. Testsigma

Testsigma is the broadest platform on this list, with web across 2,000+ browser and OS combinations, 800+ real mobile devices, API, Salesforce, SAP, and Windows desktop coverage, plus the only genuinely open-source core here, Apache-2.0 on GitHub and self-hostable via Docker. Its plain English is a templated grammar. Every step begins with a predefined action word and binds to a saved element record with a DOM locator underneath, per their elements documentation, which is why auto-healing exists to update broken locators during execution. Verdicts come in red and green plus a not-executed status, so a broken environment and a product bug wear the same color. The private-network Tunnel is an Enterprise line item, and both paid tiers are quote-gated with no public numbers as of September 1, 2026. Pick it for real-device mobile reach, or when an open-source evaluation path is the trust signal your team needs. The full Testsigma comparison traces every claim.

8. Functionize

Functionize has been building ML-driven testing for years, and the infrastructure shows it. Every test runs on its own cloud VM, their FAQ pitches running all 10,000 in parallel, the debugging suite captures four screenshots per action with Live Debug and breakpoints, and the company carries SOC 2 plus packaged-app claims for Salesforce, ServiceNow, Workday, and SAP. The artifact is ordered steps bound to ML element models built by a cloud modeling process that can take up to a day before a test is runnable. Their own FAQ concedes the sharp edge, that self-healing can cause a test to pass, guarded by verifications you write and corrected by a manual Force Fail. Private apps need the ACS tunnel after a questionnaire, a security review, and a DevOps onboarding meeting. To their credit, Studio pricing is public, running $0 to $200 per user per month on their pricing page as of September 1, 2026, with enterprise quote-only. Pick it for enterprise parallel scale or packaged enterprise apps. The full Functionize comparison quotes their docs throughout.

9. Autify

Autify is four products, and any comparison that doesn’t say which one it means is waving its hands. NoCode is the classic record-and-heal recorder. Nexus is the Playwright-based flagship that compiles your English into structured steps and honestly exports them as editable Playwright code. Genesis generates test cases from requirements. Aximo, the agentic tester now leading their homepage, stores plain-English scenarios and is the closest architectural neighbor to our own storage model that we profiled, though what it stores underneath isn’t publicly documented, and neither is its private-network reach. Autify’s private-network menu is the most complete in the category, spanning static IPs, the Connect tunnel, a private runner, and on-premise deployment, and its verdicts admit when AI touched them, through Nexus’s Recovered-with-AI status and NoCode’s Review-Needed flag. Published pricing puts Nexus at $400 a month for one user and Aximo credits at $99 to $450 a month, per autify.com/pricing as of September 1, 2026. Pick it for a proven recorder on-ramp, or when Playwright code is the durable artifact you want. All four products are mapped in the full Autify comparison.

10. Rainforest QA

Rainforest QA is the one entry that isn’t purely software. Its VMs test past the browser’s edge, covering Windows and Mac desktop apps, built-in email, file uploads, and legacy browsers, while a trained human tester community executes written tests on demand and a services tier embeds test managers in your Slack and Jira. Their open-source CLI pulls every test down as plain-text RFML for your git repo, a better exit door than most. The automation itself is pixel-based. Each action targets a stored screenshot of your UI, with DOM and AI matching as fixed-order fallbacks, so dynamic regions need masking and drift triggers healing that rewrites the test. Failures get up to four attempts, and after a real red a human sorts it into one of six triage buckets. Everything runs on Rainforest-hosted VMs, with IP allowlisting or a beta tunnel as the VPN routes, and no public pricing as of September 1, 2026. Pick it when you want to buy QA labor along with the tooling. The full Rainforest QA comparison has the doc trail.

How do you choose between AI testing tools?

Choose by your binding constraint rather than by feature counts, because in this category the architecture decides what’s even possible for you. Ask where your app lives, what coverage you need this quarter, what your organization allows on its machines, and whether anyone will act on filed bugs, and most of the ten options eliminate themselves before a demo call.

If your staging sits behind a VPN with enterprise SSO, count the tunnels first. Every cloud vendor on this list reaches your private environment through added infrastructure, whether that’s mabl Link, Functionize’s ACS, Testsigma’s Tunnel, or testRigor’s IP whitelisting, and that infrastructure is a standing security exception someone owns. TestAutomate exists for this constraint, since your own Chrome is already inside.

If you need native mobile coverage today, we’re not your tool. testRigor, Testsigma, TestMu AI, mabl, Katalon, Testim, and Autify all ship it now, and Rainforest covers it with humans.

If your org forbids browser extensions on managed machines and won’t run a headless runner, our execution model has no path in, and a cloud vendor’s model fits your policy cleanly. If your developers want code-native output, Kane CLI’s free local runs and Nexus’s Playwright export are the honest answers. If procurement gates on a vendor SOC 2 attestation this quarter, mabl and Functionize clear it and we don’t yet. And if your team files bugs nowhere, skip any tool selling a bug loop, ours included.

Last, weigh price legibility as its own feature. Four of the ten publish no numbers at all, and several meter in credits whose dollar value only the vendor knows. We’ve written up how BYOK pricing works and a wider survey of the AI test automation landscape if you’re still mapping the space, and the full comparison hub holds every head-to-head in one place.

Best Test Case Generation Tools in 2026

Tue, 08 Sep 2026 00:00:00 GMT

Every release week of my testing career ended the same way, with a manual regression pass eating the evening and the last stretch of the checklist quietly cut when time ran out. What I wanted on those evenings was a tool that could draft the coverage for me. In 2026 there are at least eight serious tools claiming exactly that, and I’ve spent enough time inside this category to know how to tell them apart.

Test case generation tools differ less in their AI than in their raw material. Every tool here derives tests from one of four sources: prompts and recordings a human supplies, the codebase, recorded user traffic, or written requirements and documentation. That source decides what the generated suite can catch. A suite derived from the implementation can only check that the app still does what it did. A suite derived from stated intent can catch what it never did right.

How we graded every tool here, ourselves included

We build TestAutomate, so this list lives on our site and you should read it that way. TestAutomate is listed first because it’s ours, not because we ran a bake-off and won one. We don’t claim a measured number-one spot for a pre-launch product, so grade our entry with more suspicion than any other, and hold everyone, us included, to the same four questions we used throughout. Where do the generated tests derive from? Where do they execute? What is a verdict allowed to say? And what does the meter actually count?

Every competitor fact below is condensed from our full head-to-head pages, each of which was reviewed against that vendor’s own public docs and pricing pages as of September 1, 2026, says “not documented” where their docs are silent, and carries a standing correction invite. If we’ve got something wrong about your product, tell us and we’ll fix it on a dated edit.

Eight test case generation tools at a glance

Tool	Tests derive from	Where tests execute	Verdict model	Pricing model
TestAutomate (ours, pre-launch)	Docs + repo + the live running app, or typed and voice-dictated plain English	Your own Chrome; self-hosted runner or cloud browsers for unattended runs	Pass / fail / blocked; a stronger model re-drives every failure first	BYOK at provider cost; managed keys at provider price + 15%
testRigor	Typed, recorded, or AI-generated English, with an element association recorded on first successful run	Their cloud (on-premise offered)	Pass or fail	Free public tier; private from $300/mo; mid tier unpublished
Functionize	Recordings or plain English, compiled into ML element models by a cloud modeling step	Their cloud, one VM per test	Green / Red / Yellow (healed) / Purple (incomplete)	Credit tiers, $0 to $200/user/mo published; enterprise quote-only
Katalon (StudioAssist)	Recorder plus Groovy scripts, with StudioAssist AI assist; stored as Object Repository locators	Local Studio free; paid Runtime Engine for CLI/CI; TestCloud hosted browsers	Pass/fail with retries and post-run flakiness scoring	$180/seat/mo Studio; $182/license/mo Runtime Engine; $197/session/mo TestCloud
Testsigma	Templated English (predefined action words) bound to saved element records	Their cloud grid or a local Java Agent	Passed / Failed / Not Executed	Quote-gated, no public numbers; open-source core self-hosts free
Autonoma	A planner CLI that reads your codebase and drafts markdown tests, with review checkpoints	Their cloud builds a per-PR preview; free self-host	Six verdicts with published trust-rate math	100K credits free; $100 per 150K credits; free self-host
Checksum	Repo analysis, live-app scanning, and an optional SDK recording real user sessions; output is Playwright	Your CI runners or their cloud	Pass / fail / healed	Sales-led, priced per maintained workflow; no public dollar figures
TestMu AI (KaneAI)	Natural language, a recorded click-through, or an imported Jira ticket, PRD, or PDF	Their HyperExecute cloud; Kane CLI runs a local Chrome free	Pass/fail with configurable step semantics, plus test muting	Credit-metered; KaneAI from $17/mo on annual billing

Competitor prices are as published on each vendor’s public pages as of September 1, 2026, per the head-to-head reviews linked in each entry below. Where a vendor publishes no number, the table says so instead of guessing.

Where do test case generation tools get their test cases from?

From four places: a human’s prompts and recordings, the codebase, recorded user traffic, or written requirements and documentation, and each source carries a characteristic blind spot into every test it produces.

Prompt- and recording-derived tools, which is most of the market, preserve what a person stated or demonstrated. testRigor, Functionize, Katalon, Testsigma, and TestMu AI all work this way, and it’s a fine way to capture the flows your team already knows matter. The blind spot is authorship. The suite contains exactly what somebody thought to write down, and the flow nobody wrote is the flow nobody tests.

Code-derived tools like Autonoma read your repository and enumerate the surface that actually exists, which makes their coverage of shipped functionality genuinely hard to match by hand. The blind spot is inheritance. A spec inferred from an implementation inherits the implementation’s mistakes, so a feature that has computed the wrong thing since day one produces a test that enshrines the wrong thing as expected.

Traffic-derived tools like Checksum add what users really do, which is powerful for prioritization. The same inheritance problem applies, since recorded behavior of the current app is still the current app grading itself.

Requirements- and docs-derived generation is the only family whose expectations are independent of the implementation. Its blind spot is the input: requirements docs are usually missing or stale on mature products, which is why TestAutomate reads the product documentation you already publish instead, and why our generation pipeline spends most of its effort filtering that input before drafting anything. The full landscape lives on our AI test case generation pillar.

The best test case generation tools in 2026

The entries below run in the table’s order. Ours comes first because it’s ours, and the other seven are grouped roughly by derivation source, prompts and recordings first, then code and traffic.

TestAutomate: generation from docs, repo, and the running app

TestAutomate is our agentic web-testing tool, currently pre-launch, and its generation model is the reason this article’s framing exists. Draft suites derive from three sources at once. Your documentation supplies the expected behavior, your repo enumerates the routes and components that exist to test, and the live running app supplies what is actually on screen. Because expectations come from the docs rather than the implementation, a verdict can catch a feature that never did what the docs promised, and none of the commercial tools we researched ingests documentation as the expectation authority. Every generated test is the same artifact as a hand-written one, with exactly three required fields (an id, a prompt, and an expected outcome) and a generation rule that forbids code, JSON, YAML, and selectors inside them; the stored artifact is the portable, plain-English YAML that our authoring page shows, with no selector layer anywhere in the file. Tests execute in your own Chrome, behind your VPN and SSO, and verdicts are pass, fail, or blocked, with no failure reported until a stronger model re-drives the run and agrees. Where we lose, plainly: no mobile or API coverage today, no SOC 2 attestation yet, and our execution model needs an extension or a headless runner. Judge the claims against a real graded run.

testRigor: plain English across the widest platform surface

testRigor is the most credible name in natural-language test authoring, and its breadth is real. One English grammar drives web, native and hybrid iOS and Android, Windows desktop, API checks, email and SMS flows, and even mainframe, with a recorder-to-English on-ramp that genuinely works for manual QA teams and a free-forever tier for public test suites. Tests derive from what you type, record, or let its AI draft. The architectural catch sits in their own self-healing documentation: on the first successful run the system records an internal element association, and later runs replay that binding until it breaks and AI re-binds it. The English is the label, and the recorded binding is the test, which is why healing exists at all. Execution happens on their cloud servers, with VPN reach via IP whitelisting, tunneling, or on-premise deployment, and verdicts are pass or fail. Pricing lists a free public tier and a private tier from $300/month on their sign-up page as of September 1, 2026, with the mid tier unpublished. Pick testRigor if you need mobile, desktop, or mainframe coverage from one language this quarter. Full head-to-head: TestAutomate vs testRigor.

Functionize: ML element models at enterprise scale

Functionize has built ML-driven testing for years, and its infrastructure is not marketing. Every test runs on its own cloud VM, the debugging suite captures four screenshots per action, and the enterprise toolchain reach spans Jira Cloud and Data Center, Azure DevOps, TestRail, and packaged apps. Tests derive from an Architect recording or plain English (or chat, in their July 2026 Studio product), which a cloud modeling process compiles into machine-learning models of every element, a build step their FAQ says completes in less than a day, sometimes a few hours. The limitation is that the model is still a stored binding with a lifecycle: replayed each run, healed on drift, and manually Force-Failed when healing guesses wrong, which their own FAQ concedes can cause a test to pass. Private apps are reached through a WireGuard-based tunnel that requires a questionnaire, a security review, and a meeting with their DevOps team. Their pricing page publishes credit-based tiers from free to $200 per user per month as of September 1, 2026, with enterprise quote-only. Pick Functionize for massive parallel scale, cross-browser coverage, and enterprise attestations today. Full head-to-head: TestAutomate vs Functionize.

Katalon StudioAssist: AI assist inside the broadest IDE

Katalon covers more surfaces from one IDE than almost anything else, with web, API, mobile, and Windows desktop automation plus real manual test management in TestOps, and its two-way Jira sync is rare at that depth. StudioAssist is the AI layer, and to its credit it supports a personal API key for OpenAI, Azure OpenAI, Gemini, or Bedrock, which is unusual flexibility for an authoring assistant. Tests derive from recordings and Groovy scripts, and what gets stored is an Object Repository of locators with the steps compiled to code. Self-healing tries backup locators, then has an LLM propose replacements that a human approves in a post-run insights tab, so the maintenance gets batched rather than removed. The commercial catch is that CLI and CI execution requires the paid Runtime Engine at $182 per license per month on their pricing page as of September 1, 2026, on top of $180/seat/month Studio. Pick Katalon if you run a mixed manual-plus-automation QA org that needs all four surfaces and a mature ecosystem this week. Full head-to-head: TestAutomate vs Katalon.

Testsigma: templated English with an open-source core

Testsigma is the broadest platform in this list, spanning web across 2,000+ browser and OS combinations, 800+ real mobile devices, API, Salesforce down to CPQ, SAP, and Windows desktop, all by their published counts. Its server, agent, and UI are Apache-2.0 on GitHub and self-host via Docker, a trust signal most codeless vendors simply lack. Tests derive from a templated NLP grammar in which every step begins with a predefined action word like Click, Verify, or Enter, and each step binds to a saved element record with a DOM locator underneath, which is why auto-healing exists to update broken locators during execution. Execution runs on their cloud grid or a local Java Agent, with private staging reached through a Tunnel that sits on the Enterprise tier. Verdicts are Passed, Failed, or Not Executed, and pricing publishes no numbers as of September 1, 2026, with both paid tiers quote-gated. Pick Testsigma if mobile or packaged-app coverage gates your decision, or if an open-source core you can read matters to your security team. Full head-to-head: TestAutomate vs Testsigma.

Autonoma: tests generated from your codebase

Autonoma is the most transparent of the agent-testing platforms, and the honesty shows up where it counts. Its planner CLI reads your codebase and drafts natural-language markdown tests into your repo, with review checkpoints at each stage, and the planner itself is MIT-licensed. Its verdict taxonomy is the best documented in the category, six named verdicts with published trust-rate math in which environment noise counts as noise rather than failure. Every PR gets an isolated, seeded preview environment, and self-hosting the whole platform is free with no feature limits. The structural limitation is the source: their own launch post says the engine reads your codebase, so expectations are inferred from the implementation under test and your review is the safeguard against inheriting its bugs. VPN reach is not documented, and the metered credit’s buying power is not published, at 100K credits free and $100 per 150K on their homepage as of September 1, 2026. Pick Autonoma if you live in GitHub PRs, want ephemeral seeded environments instead of live staging, or need zero-license self-hosting. Full head-to-head: TestAutomate vs Autonoma.

Checksum: tests generated from code and recorded user traffic

Checksum generates standard Playwright tests from three implementation-side inputs: source-code analysis of your connected repo, direct scanning of the live app, and an optional JavaScript SDK that records real user sessions. That last input is its distinctive strength, because coverage grounded in what production users actually do prioritizes itself, and its export story is among the best anywhere, with real Playwright in your repo atop an open-source runtime. The generation pipeline verifies tests by running them before opening a PR. Two limitations follow from the architecture. All three inputs observe the implementation, so behavior that shipped wrong becomes the expected behavior. And the verdict grammar is pass, fail, or healed, where healed means the agent edited the test until it passed, assertions included, with their auto-healing docs describing fixes that update expected values to match current app behavior, checked by whoever reviews the heal PR. Pricing is sales-led and priced per maintained workflow, with no public dollar figures as of September 1, 2026. Pick Checksum if your team is Playwright-native and wants generated coverage at CI volume. Full head-to-head: TestAutomate vs Checksum.

TestMu AI: KaneAI on the biggest device cloud

TestMu AI is LambdaTest renamed in January 2026, so treat it as the incumbent it is, with 10,000+ real devices and 3,000+ browser environments by its own count. KaneAI, its test agent, derives tests from typed natural language, a recorded click-through, or an imported Jira ticket, PRD, or PDF, and stores them as versioned natural-language steps synced to a code view. Its export is the best in the category, generating Selenium, Playwright, Cypress, or Appium code and opening the pull request into your repo itself, and Kane CLI runs a local Chrome for free. The limitation worth understanding is that document import happens at authoring time only. The ticket becomes steps, and from then on the steps are the truth, maintained by self-healing that updates them and surfaces a diff for review, rather than a standing expectation the app keeps getting graded against. Verdicts are pass or fail with configurable step semantics and a test-muting feature for noisy tests. Pricing is credit-metered from $17/month on annual billing as of September 1, 2026. Pick TestMu AI for a real-device matrix, one-vendor consolidation, or free code-native output. Full head-to-head: TestAutomate vs TestMu AI.

How do you choose between AI test case generation tools?

Choose by your binding constraint, not by feature count, because in this category the architecture decides what’s even possible for you. These are the decision rules I’d actually apply.

If your app lives behind a VPN or enterprise SSO on real staging data, favor tools that execute where your access already exists. TestAutomate drives your own Chrome, Checksum runs on your CI runners, Katalon Studio runs locally, and Testsigma offers a local Java Agent. Every pure-cloud option on this list reaches your network through whitelists, tunnels, or an on-premise deal, and each of those is security work you’ll own.

If you need mobile coverage today, we’re not your tool and neither is Checksum. testRigor, Testsigma, TestMu AI, and Katalon cover mobile now, and Autonoma drives Appium. If it’s API coverage you need, those same four cover it, and Checksum’s top tier adds an API agent by their claim; we don’t cover API either.

If your docs are thin or stale, docs-grounded generation starves, ours included. Code-derived (Autonoma) or traffic-derived (Checksum) generation will produce a fuller first draft from what you do have, and you can see what a docs crawl really yields before betting on the docs-first path, since we recorded a full run against a real docs site.

If your team has no tracker discipline, skip close-the-loop bug workflows entirely, ours included, because there’s nothing for them to close. If your org forbids browser extensions and won’t run a headless runner, Checksum’s CI-native model or a cloud grid fits where our execution model doesn’t. And if you need SOC 2 on the vendor’s letterhead this quarter, the established vendors win that race against a pre-launch product, no contest.

One last honesty note on verdicts, since generation quality means nothing if you can’t trust the grades. Our own runner’s verdict grammar is pass, fail, or blocked, with skipped runs counted separately, so a broken environment is never counted as a failure, and the verdicts page documents exactly what each verdict is allowed to mean. Ask every vendor on this list the same question, and read their answer as carefully as their generation demo.

AI Testing Cost: Engineering the Bill Down

Wed, 02 Sep 2026 00:00:00 GMT

While I was building TestAutomate’s agent loop, I kept a provider usage dashboard open in a second tab the whole time. Nobody asked me to. Watching real spend while your own code runs just changes how you write the code. The lesson from those weeks is that the meter doesn’t climb smoothly. It jumps, and every jump has a cause you can name and remove.

AI testing cost is token cost, almost entirely. An agent-driven test spends tokens on three jobs: observing the page, deciding the next action, and verifying the result. The bill comes down through four mechanical levers. Observe pages as compact text instead of screenshots, match model tiers to step difficulty, cache the stable prompt prefix, and let each run make the next one cheaper. Credit-metered pricing hides all four, which is a big part of why vendors in this category rarely explain them.

So I’ll explain them, from the inside of building the thing. No benchmarks, no invented percentages, just the mechanisms, plus the questions I’d put to any vendor in the category, including us.

What drives AI testing cost in an agent run?

The loop drives it. An agent testing your app runs a cycle of observe, decide, act, and every cycle sends a model a fresh picture of the page plus the history of the run so far. That means the way the tool observes a page sets the slope of the whole bill, because observation is the input context you pay for on every single turn.

There are two ways to show a model a web page. The expensive way is a screenshot. Image tokens scale with pixel area, per Anthropic’s vision documentation, so a full screenshot of a dense dashboard is a large input purchase every time you take one, and an agent that screenshots after every click is buying that picture dozens of times per test. The cheap way is text. A page’s interactive structure, the buttons, links, and fields with their labels, is the same information accessibility tooling reads, and rendered as text it’s a tiny fraction of the input a screenshot produces while being more useful to an agent that needs to pick its next click.

TestAutomate’s runtime treats the text read as the default and the screenshot as an exception that must earn its place. After every navigation and click, the runner automatically captures the page’s interactive elements as text, capped at 1,500 characters, and appends the snapshot to the action’s own result, waiting 300 milliseconds after clicks so the page has settled. The agent never spends a turn asking to look, because looking already happened, and no image tokens were involved. Its instructions say the quiet part explicitly: don’t screenshot to check what happened, use a screenshot only to judge visual state. Escalating to a fuller page read or an actual screenshot is a deliberate step taken for a named reason, never a vibe. All of this happens while the agent drives your real Chrome against your real app, so the pages being observed are the ones your users actually see.

The third spender is verification. When a run finishes, a separate model reads the recorded trajectory and grades each written expectation, and that pass is bounded by construction. The trajectory is compacted to its first 20 and last 60 events and capped at 24,000 characters before the verifier sees it, so the grading call can’t balloon with the run. The acting loop itself is capped at 120 turns by default, which is a correctness guard against a confused agent looping forever and, in the same stroke, a cost ceiling.

Which levers actually cut LLM testing costs?

Four levers, all mechanical: model tiers matched to step difficulty, prompt-prefix caching, escalation that spends premium capability only where it changes a verdict, and run memory that makes repetition cheaper. None of them is exotic. They’re ordinary engineering, applied to a bill most of the category would rather you didn’t examine line by line.

Match the model tier to the step’s difficulty

Not every step in a test deserves the same brain. In TestAutomate a mid-tier model drives the browser, because clicking through a flow is steady work that rewards speed and economy. The strongest tier is reserved for the two places where quality compounds: grading verdicts, where a wrong judgment poisons your trust in every green and red, and authoring generated tests, because spec quality directly drives every future run’s correctness. And the cheapest tier handles genuinely mechanical jobs, like the live ping that verifies a pasted API key actually works before it’s saved. The decision rule I applied while wiring this up is simple to state. Pay for capability where the output is load-bearing or long-lived, and refuse to pay for it where the work is procedural.

Cache the prompt prefix, and then defend it

Providers bill a repeated prompt prefix at a steep discount once you mark it cacheable, and Anthropic’s prompt caching documentation spells out the mechanic that matters most: caching is a prefix match, so any changed byte invalidates everything after it. That turns caching from a checkbox into a discipline. The stable content, the system prompt and the tool definitions, has to come first and stay byte-identical across turns, and nothing volatile may sit ahead of it. A timestamp rendered into the system prompt, an ID that varies per request, a tool list that assembles in a different order, any of these silently zeroes the discount while everything looks fine.

In our agent loop the system prompt is marked cacheable and held stable across the turns of a run, precisely so the growing part of each request is only the new trajectory. The audit rule that follows from the mechanism: if your provider’s cache-read counters sit at zero across a run’s turns, something in the hot path is breaking the prefix, and it’s nearly always a small, innocent-looking piece of dynamism.

Spend premium certainty only where it changes a verdict

The strongest model is the most expensive place to run a browser, so the honest question is where its judgment actually changes an outcome. Our answer is escalate-on-fail. When the verifier judges a test a genuine failure, the runner re-runs the act phase exactly once on the strongest tier and lets the second verdict stand. A pass on retry is recorded as exactly that, evidence the first failure was agent error rather than an app defect. A second failure is recorded as high confidence that the issue is real. The retry never triggers on a blocked or skipped run, because an environment problem is not a failure and burning premium tokens on one proves nothing.

The cost shape this produces is the point. Premium spend scales with your failures, not with your suite size. A healthy suite of passing tests runs entirely on the economical tier, and the expensive model appears only at the moments where its certainty is the product. The grading side of this arrangement, why a separate judge reads the evidence at all, is covered in how an LLM judge grades software tests.

Make the second run cheaper than the first

The first time an agent tests an app, it pays a discovery tax. It has to find the login flow, learn which menu hides which page, and trip over the app’s quirks one by one, and every bit of that exploration is billed in tokens. TestAutomate keeps what it learns. Logins, navigation paths, and app quirks are learned once and scoped to the organization, so the knowledge is shared across your team’s runs rather than rediscovered per person. The second run walks a known path instead of exploring, and cost falls with use.

I’d flag this lever especially when you evaluate ai test automation pricing, because it’s the one that inverts the usual curve. Per-execution pricing makes the thousandth run cost what the first did. A tool that remembers makes the thousandth run the cheapest one yet.

Why does credit-based pricing hide the real cost?

Because a credit is an abstraction with a conversion rate inside it, and the conversion rate is where the markup lives. When a vendor buys tokens at provider prices and sells you credits, the spread between those two numbers is their margin, and the credit meter is what keeps you from computing it. You can’t reconcile a credit balance against a provider’s published per-token rates. That’s not a quirk of presentation. It’s what the abstraction does.

The deeper problem is the incentive it creates. A vendor whose revenue is the markup earns more when your runs burn more, and every lever in the previous section cuts their revenue directly. Text observation instead of screenshots, cheaper tiers for easy steps, cached prefixes, memory that shrinks repeat runs: each one is money out of their pocket the moment they implement it. I’m not claiming any particular vendor refuses to optimize. I’m pointing at the structure, which rewards them for not trying very hard. The check is simple enough to run yourself: ask a credit-priced vendor for the published token-to-credit conversion, and note how the conversation goes.

BYOK testing inverts the incentive. When you bring your own API key, model spend passes through at cost, the vendor’s revenue comes from the product rather than the metering, and every optimization we ship lands in your pocket instead of ours. That’s precisely why I get to write a post like this one. Making runs cheaper costs us nothing, so the engineering can be honest about where the money goes. The full comparison against credit-metered AI testing SaaS walks the pricing structures side by side.

In fairness, credits buy one real thing: simplicity. No provider account, no key management, swipe a card and start. That convenience has a legitimate price. The test of a fair pricing model is whether the price of the convenience is printed where you can read it, or buried in a conversion rate where you can’t.

What cost visibility should a testing tool give you?

Dollars, at three moments: an estimate before the work starts, a live meter while it runs, and a hard ceiling you set that pauses the work before the line is crossed. Anything less, and you’re discovering your spend from an invoice, which is the most expensive possible place to learn it.

That before-the-line detail is worth dwelling on, because it’s where I put real implementation effort. In TestAutomate’s issue re-verification jobs, the shipped defaults are a ten-dollar job budget with a rough per-issue estimate of about thirty-five cents that the live meter refines as real usage arrives, and a 25-turn cap on each issue. Before picking up the next item, the job checks whether the spend so far plus that next item’s estimate would cross the budget, and if it would, the job pauses right there, stops cleanly, and reports everything it graded up to that point. Those numbers are defaults you can change, not benchmarks, and your spend depends on your app and your provider’s prices. The mechanism is the promise: the ceiling is respected before the overspend happens, never reconstructed after.

Two more visibility details round it out. Retrieval, the part of the system that recalls what earlier runs learned, runs locally and adds nothing to the API bill, so the meter reflects model work and only model work. And for teams that start without a key, managed keys show the math in the open, the provider’s price plus a stated managed fee, printed next to the price rather than dissolved into a conversion rate. The economics page lays the whole structure out, meters and ceilings included.

What should you ask any AI testing vendor about cost?

Ask about mechanisms, because vendors who have them love this conversation and vendors who don’t will answer a different question than the one you asked. This list works against TestAutomate too. That’s deliberate, and honestly it’s the best reason to trust it. Bring it to us and expect specific answers.

How does your agent observe a page? Text-first or screenshot-first, and what exactly triggers an escalation to a fuller read or an image?
Which model runs which step? Can you name the tiers and say what each one is trusted with?
Do you cache prompt prefixes, and what sits in your hot path that could silently break the prefix?
When a test fails, what re-runs, on which model, and how many times? What does a pass-on-retry mean in your reporting?
Is the second run cheaper than the first? What does the tool remember about my app, and who inside my org shares that memory?
Can I see spend in dollars before, during, and after a run? Can I set a ceiling, and does it pause work before the line or notify me after?
If you price in credits, what is the token-to-credit conversion, and will you publish it?
When you make the runtime more efficient, whose bill goes down, mine or yours?

That last question is the whole post in one line. Every mechanism I’ve described here, the text observation, the tiering, the caching, the escalation, the memory, exists because in a pass-through model the customer’s bill and the vendor’s engineering pride point in the same direction. Ask the question, listen for which direction the answer points, and you’ll know most of what the pricing page wasn’t going to tell you.

Automated Testing Behind Login: SSO and MFA

Wed, 02 Sep 2026 00:00:00 GMT

In twenty years of QA I have tested enterprise service platforms, a telephony product, and health-tech applications, and none of them kept anything worth testing in front of the login page. Tutorial apps are public. The apps people are paid to test are walls first and features second, and health-tech builds the tallest walls of all, because what sits behind them is medical data.

Automated testing behind login means getting a test runner past authentication and keeping it authenticated for a whole suite. It is where most end-to-end automation dies. SSO redirects leave your app’s domain, MFA wants a code the script doesn’t have, magic links detour through an inbox, sessions expire mid-run, and login pages carry a site’s heaviest bot defenses. What holds up is a browser that is already signed in, plus stored auth recipes for unattended runs.

Both halves of that answer need unpacking, and so do the workarounds most teams are living with instead.

The login page is the one page every single test must pass through, and it is also the page most deliberately hardened against software acting like a person. That combination is lethal to a scripted suite. A sign-in that only occasionally fails reads as noise on any individual test, and as a red build every morning once every test in the suite rolls those dice at the same door.

I learned this the slow way. The suites I have watched rot did not rot on the features. They rotted at the front door. A session that expired partway through a long run took every remaining test down with it, and the failures said nothing about the app. Parallel workers signing into the same account tripped lockout policies that existed exactly to stop credential stuffing, so the better our security posture got, the redder our dashboards got. And once single sign-on became the norm, the sign-in page stopped being our page at all. The redirect left our domain, bounced through an identity provider we didn’t control, and came back with opinions about whether our test runner looked trustworthy.

None of that is a bug in any one tool. It is a structural mismatch. Login exists to distinguish humans from software, and a test script is software insisting otherwise, at scale, on a schedule.

Each wall breaks automation in its own way, and the fix for one is useless against the next, which is why “handle auth” is never a single work item. This is the taxonomy I sort by when someone tells me their suite can’t get in the door, roughly from mildest to hardest.

Auth wall	What it does to a scripted test
Email + password	Works until it doesn’t. Password rotation invalidates stored credentials, lockout policies punish parallel logins, and session TTLs sign the suite out mid-run.
Google OAuth	The flow leaves your app for Google’s domain, where automated and embedded browser contexts violate policy. Scripted sign-ins get challenged or refused outright.
Okta / SAML SSO	A redirect chain across domains you don’t control, plus conditional-access rules and device checks. A fresh headless browser is what those policies exist to block.
Magic links	Sign-in requires an email round trip. The test now depends on an inbox, and the link is single-use and expiring.
TOTP	The code changes every 30 seconds and is computed from a secret your script was never given.
VPN-only staging	Not an auth problem at all. The runner cannot reach the app unless it is inside the network.

A few of these deserve expansion, because the failure modes are less obvious than the table makes them look.

Google OAuth is the clearest case of the wall being deliberate. Google’s OAuth policies require sign-in to happen in a secure, standard browser context, and the sign-in page actively refuses contexts it judges automated. If you have ever watched a run stall on “This browser or app may not be secure”, you have met that policy in person. It is not a flake. It is Google doing its job, and your test being collateral.

Magic links move the problem out of the browser entirely. The test has to receive an email, extract a link, and follow it before it expires, so automating magic-link sign-in puts an email delivery dependency in front of every authenticated test. TOTP at least stays local, since RFC 6238 codes are just math over a shared secret and a 30-second time step, but that only helps if you are permitted to hold the secret somewhere a runner can use it.

And VPN-only staging outranks everything above it, because it decides whether the conversation even starts. In health-tech and enterprise work, staging environments carrying realistic data do not sit on the public internet, and no cleverness at the login page helps a runner that can’t route a packet to the app.

Every team testing a real app has a workaround in place, usually several, and each one buys access by making staging less like production or the perimeter less like a perimeter. I have sat in the meetings where all of the usual ones got proposed, and in the reviews where some of them got regretted.

Static-IP allowlists come first because cloud vendors suggest them. You punch the vendor’s published IP ranges through the firewall so their runners can reach staging. Now your perimeter has a standing hole shaped like a third party’s data center, your security team owns a rule it never wanted, and the rule silently breaks the day the vendor reshuffles its ranges.

Tunnel clients are the same trade in a different coat. An agent inside your network bridges connections out to the vendor’s cloud. It works, and it is one more daemon to install, monitor, and explain at the next audit, one more moving part that can be the real reason a run failed.

Test-only auth backdoors are the workaround security teams hate most, and they are right to. An endpoint that mints authenticated sessions for test users is a skeleton key, and the distance between a staging convenience and a production incident is one misconfigured deploy. It also removes login itself from coverage, which is a strange thing to give up on purpose.

Long-lived sessions are the gentlest version. Sign in once by hand, save the cookies and storage, and reuse that state across tests. Playwright’s authentication docs describe the pattern well, and it genuinely works, right up until the session hits its TTL, the credentials rotate, or IT shortens session lifetimes for compliance and every suite in the company logs out on the same morning.

Disabling MFA on staging is the most common workaround and the most expensive in disguise. Staging now authenticates differently from production, so the flow every real user walks first is the one flow your suite never exercises. The MFA prompt, the redirect handling around it, the conditional-access behavior, all of it ships untested while the sign-off email says the regression suite passed. If you want testing behind MFA rather than around it, the honest floor is provisioning the TOTP secret to the runner so codes can be computed, which keeps the prompt in play but concentrates a secret in one more place.

My decision rule after years of watching these. If a workaround makes staging behave less like production, write down what it un-tests and treat that as a cost. If it weakens the perimeter, let the security team price it rather than the QA team. Most of the five fail one of those checks.

Why can’t cloud grids test staging behind a VPN?

Because the browsers live in the vendor’s network, not yours, and no amount of product polish moves them. A cloud grid’s machines cannot route to a VPN-only environment without an allowlist hole or a tunnel agent, which means the workarounds above are not optional extras for that model. They are prerequisites.

The trouble goes past reachability. A grid browser is a fresh, headless profile in a data center, with no history, no cookies, and an IP address that reputation systems already know by heart. That is precisely the anonymous profile login defenses are tuned to challenge, so the fingerprint that makes grid browsers cheap to scale is the fingerprint that makes them expensive to sign in. Enterprise SSO compounds it, since conditional access can require a recognized device or network, and a headless container in someone else’s cloud is neither.

None of this makes cloud grids useless. For public apps with tame authentication they are a workable execution model, and plenty of teams run them happily. But if your staging sits behind a VPN and your login sits behind an identity provider, you are not choosing between products, you are choosing between architectures. I have written up the fuller comparison of cloud AI testing platforms separately.

How does TestAutomate handle authenticated session testing?

Two ways, and the split between them matters. For attended runs, TestAutomate executes tests in your own Chrome, through an extension, in the browser where you are already signed in. There is no session to construct because the session already exists. You are behind the VPN because your machine is. The SSO handshake already happened, MFA was satisfied by you, and the identity provider sees the same recognized device it saw this morning. No tunnel, no allowlist, no backdoor. The real-browser execution model page walks through the mechanics.

That covers the runs you watch, which leaves the obvious question of what signs in for the Saturday 6 a.m. run. Scheduled and unattended runs can’t lean on a live session, and that is what the Auth Catalog is for. It stores named auth entries per app, covering email and password, Google, Okta, magic links, and TOTP, so a run that wakes up without you knows how this particular app signs in. Entries are seeded from run memory, which learns each app’s login path and quirks from real runs and keeps them scoped to your org, so the catalog starts from what the agent has already done rather than from a blank form. Entries are encrypted per org, with the same AES-256-GCM sealing the platform uses for bring-your-own-key credentials. Results from those runs arrive by Slack or email, wherever the team already lives.

What I appreciate most, as the person who spent years triaging reds that were really login failures, is what happens when authentication fails anyway. The agent’s standing instructions are blunt. If it hits a login, SSO, or MFA page with no credentials provided, it stops and reports “the app required login and no credentials were available”, and it is explicitly forbidden from inventing credentials or attempting password resets. A login that can’t complete during setup ends the run blocked rather than failed, because the app was never exercised, and a blocked run never counts as a failure and never files a bug.

Here is what that verdict looks like on a real recorded run in our dashboard. Run 36 ended blocked in 1m 3s because setup could not establish the test’s precondition, and the verdict says in as many words that this is a test-environment problem, not an app defect. A login that will not complete gets exactly the same treatment, because the failure sits in front of the app, not inside it.

The runner is just as literal about the reverse case. Tell it the browser is already authenticated, then hand it a test whose whole purpose is the login flow, and it refuses to pretend. The run records as skipped, with reasoning that the test “exercises the login/authentication flow” and so cannot run in already-logged-in mode, because you can’t demonstrate signing in while already signed in. It is a small behavior, and it is the kind of honesty that keeps an agentic testing suite trustworthy behind a wall.

Hardware security keys, some bot detection, and judgment, and I would rather list them here than have you discover them mid-rollout. A WebAuthn hardware key is designed around a test of user presence, a physical gesture on a physical device, and no honest automation supplies that gesture on your behalf. If staging mandates hardware keys for every sign-in, unattended runs need a policy conversation with your security team before they need any tool.

Bot detection is the second boundary. TestAutomate does not solve CAPTCHAs or defeat bot-detection challenges. Running in a real, recognized browser means those challenges appear far less often, not never, and when one does appear the mechanism is a person. Unattended runs stream a live view you can click into and take over, exactly for the SSO prompt or challenge that needs human hands for a moment, and attended runs can ask you a question mid-run the way a careful colleague would.

And runs go where you point them. The target environment is configured per app, staging is the only place a test agent should be learning an app’s login quirks in the first place, and production is a target you would have to configure deliberately.

Two decades of testing behind other people’s walls left me with one conviction. The login page is not an obstacle standing in front of the real testing. It is the first piece of your app every user meets, and the first place your automation should be honest about what it can and cannot do. Pick an architecture that respects the wall instead of pretending it is not there.

How a Test Run Gets Graded, Step by Step

Sun, 30 Aug 2026 00:00:00 GMT

The run I keep going back to in our QA environment is a failure. Run 33 on our recorded timeline drove two tests against an HCM test app, took 12 minutes and 10 seconds, failed both, and the verdict on one of them names the exact wrong turn. The agent operated on a review cycle called 360-Review-Fiscal-Year-2026 when the test required 360-QA-Regression-2026. Nobody re-watched the session to learn that. The grading pipeline produced it, and this post walks through that pipeline one stage at a time, because how TestAutomate grades every run is easier to trust once you’ve followed a single run all the way through it.

The short version fits in a paragraph. Every run moves through the same fixed sequence. A setup phase must declare the test’s preconditions ready or the run records as blocked. An agent then drives the flow in a real browser while every action and result is recorded as a trajectory. A separate verifier model reads that trajectory against the expected outcome and issues one of four verdicts, passed, failed, skipped, or blocked, and a judged failure earns one retry on a stronger model before the grade stands.

I’ve written separately about why the judge must be a separate model, and our verdicts page makes the case for the philosophy. This is the operational companion to both. Not why the pipeline is shaped this way, but what actually happens to your run inside it.

What has to be true before the browser opens?

The runner won’t start a run it can’t grade. A test here is three required plain-language fields, an id, a prompt saying what to do, and an expected outcome saying what must be true afterward, and a test missing its prompt or expected outcome doesn’t load at all. That gate exists for the verifier’s sake. The expected outcome is the checklist the whole pipeline will eventually be graded against, so a run without one would be motion with no way to score it. Most of the craft in writing expected results for automated tests is really about making this stage strong enough to carry the rest.

Two exits can end a run before the browser does anything interesting. The first is a skip. If you run a suite in already-logged-in mode and a test’s whole purpose is to exercise the login flow, the two are mutually exclusive, so the runner records the test as skipped instantly, with a plain-words reason and neither a pass nor a failure. On our recorded timeline that costs literally nothing. Run 31 shows one skipped test in a run rendered at zero seconds.

The second early exit is the setup gate. A test can carry an optional setup prompt whose only job is to establish preconditions, idempotently, inside its own fixed time budget. Setup must end by declaring exactly one of two outcomes in a form the runner reads mechanically, either ready, with a one-line description of the precondition state that now exists, or blocked, with the reason it couldn’t be established. Anything short of ready and the run is recorded as blocked on the spot. The test’s own steps never execute, no verdict about the application is issued, and the record says so explicitly.

That’s run 36 from the same dataset, blocked in 1 minute and 3 seconds because the precondition proved unreachable. Under the hood a blocked run is recorded as neither a pass nor a failure and tagged as a precondition problem rather than a test result, which is what keeps it out of failure counts everywhere downstream. I’ve argued the full case for why tests get blocked instead of failed elsewhere. Operationally, the thing to know is that a blocked run stops early, runs cleanup to revert any partial state, and never reaches the verifier at all.

What does the act phase actually record?

Once setup declares ready, the act phase begins, and this is where the evidence gets made. A browser-driving model receives the test’s prompt plus the one-line description of what setup established, and it operates your real application in your own Chrome. That’s not a simulation layer or a stripped-down clone of a browser. It’s the same Chrome you use, carrying your real logged-in sessions, which is why the run sees your app the way your users’ browsers do. The execution page covers that architecture in more depth.

What matters for grading is that every step becomes a record. Each turn, the agent picks a browser action, the action executes, and the result comes back and is appended to the trajectory. After every navigation or click, the agent also reads the page that resulted, the way a person would glance up to check their work, and that observation lands in the record too, so the trajectory fills up with written observations instead of image spam. Screenshots still happen when visual state is genuinely the question, and they live in the trajectory too.

The act phase is bounded, not open-ended. It runs inside its own fixed time budget, and the agent loop carries a hard cap on turns so a confused agent can’t wander forever. Inside those bounds, speed is never a grading input. A flow that takes ninety seconds on a warm environment and four minutes on a cold one produces the same verdict if the same things ended up true, which is a large part of how this design sidesteps the timing brittleness I covered in the flaky tests post.

When the agent finishes, it writes a closing summary of what it believes it accomplished. That summary goes into the trajectory clearly labeled as the agent’s own account. It is a claim, and the next stage exists because claims aren’t evidence.

How does a recorded trajectory become a verdict?

First the trajectory gets condensed into something a judge can actually read. A long run can produce hundreds of events, and a verifier drowning in context produces garbage verdicts, so on a very long run the runner keeps the opening stretch and a larger closing stretch, inserts a note that the middle was elided, and bounds the total size. The head preserves how the run began, the tail preserves where the outcome was decided, and the bound guarantees the judge reads all of what it’s given.

Then a separate verifier model, one that took no part in driving the browser, receives three things. The test’s expected outcome. The established precondition from setup, presented as guaranteed so the judge credits it rather than demanding the run re-prove it. And the condensed trajectory. Its standing instruction is to judge only on evidence in that record, and it’s warned specifically about the agent’s closing summary, that the agent may have claimed success without verifying, so the agent’s word alone never counts as evidence. The same instructions set a deliberate asymmetry, that a false pass is worse than a false alarm, so a doubtful core expectation gets marked missing rather than waved through.

The verifier can’t answer with a vibe. Every verdict comes back in the same fixed structure, carrying six things:

the call itself, whether every core expectation held
the written reasoning behind that call
each expectation the trajectory clearly satisfies, itemized
each core expectation it doesn’t, also itemized
any flags, changed-but-equivalent observations passed on a stated assumption
any extras, actions the agent took that nobody asked for

Two escape valves keep that strictness from turning into noise. An expectation the author prefixed with [incidental] can never fail the run, and behavior that changed but still satisfies the intent becomes an assumption flag on a passing verdict instead of a failure. A flagged run is still a pass, and a flag may never excuse an unmet core expectation.

That’s run 33 again, the failure I opened with, exactly as the dashboard renders it. The reasoning names the wrong cycle and the right one, and the itemized checklist behind it says which expectations died with that wrong turn. This stage is the one that turns a browser session into something you can act on without replaying it, and it’s also the clearest place to see how differently a red gets made here than in the usual alternatives.

Approach	What produces a red	What arrives with it
Scripted, selector-based suites	An assertion or selector threw at one step	A stack trace, sometimes a screenshot, and a triage job done by hand
Record/replay tools	Playback diverged from the recording	The step that diverged, with no judgment on whether the app is actually wrong
Managed QA services	A person judged the behavior wrong	An accurate written report, on the service’s timeline
TestAutomate	A separate verifier judged recorded evidence against the expected outcome, and the failure survived a stronger-model retry	Written reasoning, itemized unmet expectations, and the full trajectory

What happens when the verdict comes back failed?

One thing, once. The runner re-runs the act phase on a stronger model than the first attempt used, and that retry produces its own fresh trajectory and its own fresh verdict from the same skeptical verifier. Nothing is averaged between attempts. The second verdict simply becomes the authoritative one.

Both outcomes are recorded in words a human can use. When the retry passes, the verdict’s reasoning states that the first failure was an agent-execution issue, not an app defect, and the record is annotated as escalated and passed-on-retry so the intermittent behavior stays visible instead of being laundered into a plain green. When the retry fails too, the reasoning states high confidence that this is a real issue, which is the sentence that earns a failure your attention. I walk the whole path in the post on false positives in automated testing, including the honest caveat that a genuinely intermittent app bug can pass the retry and wear the agent-error label.

Just as important is when this stage refuses to run. Blocked and skipped runs never escalate, because re-running a missing precondition on a smarter model fixes nothing. And if the verifier itself couldn’t be reached, the run records as failed with a verdict saying the trajectory needs manual review, and no retry fires, because escalating a run that no judge actually graded would manufacture confidence out of an outage. The retry budget exists for exactly one situation, a judged failure where a second attempt could genuinely change the conclusion.

Where does a graded run end up?

Cleanup runs first, whatever the verdict was. It reverts what the run created, follows the same idempotency rule as setup, gets its own time budget, and has no way to touch the grade. A cleanup hiccup can never turn a pass into a failure, and a doomed run still tidies up after itself so the next run doesn’t trip over its debris.

Then the record persists, and it’s the whole case file, not a letter grade. Verdict, reasoning, the itemized met and missing lists, any flags and extras, the full trajectory including any screenshots the agent took, the escalation annotation if one fired, and the duration. The suite’s progress streams to the dashboard while it runs, and the finished timeline renders each run’s verdict mix as separate counts rather than collapsing everything into red and green. On our recorded timeline that’s the difference between run 33, two failures in over twelve minutes that deserve an investigation, and runs 34 through 36, three blocked runs finished in about a minute or less apiece that deserve an environment fix. Same color on most dashboards. Different work, and the report says which.

One more door opens off this stage, deliberately gated. A judged failure’s evidence can become a bug candidate, and the verifier’s reasoning plus its unmet expectations ride along as the reproduction context. Candidates are never auto-filed. A human reviews each one in the dashboard and clicks to file it to Jira, after a strict duplicate check, and blocked runs never produce a candidate at all because there’s no application behavior to report. The trajectory and its screenshots stay attached to the run record itself, which is where anyone triaging the filed issue goes to see what the agent saw.

That’s the full journey. A three-field test either skips, blocks at the setup gate, or gets acted out in a real browser while every step is recorded, and the record gets read by a skeptical judge whose failure calls survive one stronger-model retry before reaching you. Each stage exists to make the next one’s output trustworthy, and the sum is the property I care most about in AI test automation, that by the time a red result asks for a human’s attention, the pipeline has already done the triage a human used to do by hand.

Testing in Plain English: How It Works

Sun, 30 Aug 2026 00:00:00 GMT

When we built TestAutomate’s authoring layer, I held one line I wouldn’t trade away. A stored test would have nowhere for code to live. No selector slot, no script block, no step-definition hook, nothing. Every test would be a few English sentences, and those sentences would be the thing that runs. The product page calls that plain-English authoring. Searchers call it testing in plain English. What follows is my attempt to pin the phrase down properly, because people use it to mean at least three different things, and only one of them survives contact with a changing UI.

Testing in plain English means the English is the executable test itself, not a readable label on top of code. You write what a user does and what must be true afterward, the way you’d brief a careful teammate. At run time an AI agent performs that intent in a real browser, and a separate model grades what happened against your written expectations. No selectors, scripts, or step definitions exist underneath the words.

That definition draws some hard boundaries, so let me walk them one at a time.

What does testing in plain English actually mean?

Three different pitches wear the phrase, and they behave nothing alike, so the first job is telling them apart. The weakest sense is cosmetic. A framework gives tests readable names and prints friendly reports, and every assertion underneath is still code. The middle sense is translation. You type an English sentence and the tool generates a script from it once, selectors included, and from that moment on you maintain the script, not the sentence. The strongest sense, the one worth searching for, is execution. The English is the stored artifact, and something interprets it fresh against the live app every time the test runs.

TestAutomate sits in the third camp, and the honest disclosure is that I built it, so hold my definition to the same scrutiny I’m applying to everyone else’s.

The probe I’d use on any plain-English testing tool is simple. Take a stored test, change one English word, and ask what has to happen before the next run reflects the change. If a human or a generator must touch code first, the English is documentation with good posture. If the answer is nothing, because the runner reads the English itself, you’re looking at natural language test automation in the strict sense. A second probe works from the other side. Ask to see the stored artifact and check whether a selector could live anywhere in it. I proposed that check in the piece on test automation without selectors, and it transfers to this vocabulary unchanged, because intent-based testing and selectorless testing are two names for the same structural bet.

Is plain-English testing the same as Gherkin?

No, and the difference sits exactly one layer down from the surface. Gherkin, the language behind Cucumber, has made tests read as Given, When, Then sentences for nearly two decades, and those sentences are genuinely plain English. But each line works by matching a step definition, a function somebody on the team wrote in a programming language, and the step definition is what actually drives the browser. Cucumber’s own reference is straightforward about the mechanism. The English selects code. It doesn’t replace it.

That design decides who can really write tests. The vocabulary is closed. If no step definition exists for “When I archive the oldest project”, the line can’t execute until an engineer implements one, so the set of sentences a non-programmer can compose is bounded by the glue code the team has accumulated. And the glue code inherits every maintenance burden the plain English was supposed to remove. The selectors live inside the step definitions, and they go stale there just as fast as they would anywhere else.

Plain-English execution removes the matching layer entirely. There is no fixed vocabulary, because there are no step definitions to match against. Any instruction you could hand a competent teammate is a legal sentence, and the agent works out the mechanics against the page it actually sees on that run.

I want to be fair to Gherkin on the way past, because its specification discipline holds up. Martin Fowler’s description of Given-When-Then as a way to separate precondition, action, and check is exactly how a good prompt and expected outcome divide their labor in an agentic test. The grammar survives. What goes away is the requirement that every sentence be pre-implemented before it means anything.

What does a plain-English test actually store?

Three required fields, all natural language. An id names the test. A prompt describes what to do, end to end. An expected outcome itemizes what must be true afterward for the run to count. A test can also carry optional setup and cleanup prompts written the same way, for establishing preconditions before the run and removing what the run created. The runner refuses to load a test that’s missing any of the three required fields. The optional fields around them are housekeeping, a name, tags, a timeout, and nothing in the stored test is ever executed as code or resolved as a selector.

Here’s a complete test in that shape:

id: reviews.cycle.create
prompt: sign in as the demo HR admin, create a review cycle named 360-QA-Regression-2026, and add one participant from the employee list
expected outcome: the cycles list shows 360-QA-Regression-2026 with exactly one participant, no other cycles were modified, and [incidental] a confirmation message appears

The bracketed prefix does real work. Every unprefixed bullet is a core assertion that must hold for the run to pass. A bullet marked [incidental] is a nice-to-have, and a missing incidental can never fail the test, which stops a cosmetic toast from outvoting the behavior the test exists to check.

The plain-language rule binds machines as well as people. When TestAutomate drafts tests itself, the generation rules state it verbatim: “Do NOT include code, JSON, YAML, or selectors.” A generated test meets the same bar as a hand-written one, sentences only.

Notice what the artifact never says. Nothing about where the create button sits, what the form’s fields are called in markup, or how the participant picker is implemented. Finding all of that is the agent’s job at run time, on every run. The product page on plain-English authoring makes the pitch for why this is a pleasant way to write tests, so I’ll leave that side to it. What matters for the definition is narrower. The English above is not a description of the test. It is the test.

What happens when the UI changes?

Nothing in the test goes stale, because nothing in the test referred to the old UI in the first place. This is the property most people are actually shopping for when they search for plain-English testing tools, so it deserves precision rather than a slogan. There are three cases, and they end differently.

First, the change is cosmetic. A button gets renamed, a form moves into a modal, a redesign reshuffles the layout. This is why automated UI tests keep breaking after frontend deploys that changed no behavior at all, since a scripted suite fails the moment its stored strings stop matching the markup. A plain-English test carries no stored strings. The agent reads the live page on this run and finds the control as it exists now, so the rename is absorbed without anyone editing anything.

Second, the change is real but compatible. The flow still satisfies the written intent through different behavior, say an extra confirmation step before the record is created. When a run succeeds through behavior that differs from what the test described, the verifier can attach an assumption flag to the pass, recording what it observed, what was expected, and the assumption it made, for a human to review. A flagged run is still a pass, and a flag never excuses an unmet core expectation.

Third, the change breaks the behavior. The record never appears in the list, and the test fails. That’s the outcome you wanted, because the red is now about your app instead of your locators.

One boundary is worth drawing plainly here. This is not self-healing. A self-healing tool keeps stored selectors and patches them when they stop resolving, so the artifact still contains the thing that breaks. A plain-English test has nothing to patch. The practitioner’s comparison between the two approaches gets its own post, written by someone who carried a locator repair queue for years.

How does an English sentence become a pass or a fail?

Through two deliberately separated jobs, acting and judging. On the acting side, the agent drives a real browser, and specifically your own Chrome, through an extension that binds to a tab. It signs in, clicks, and types the way a person at the keyboard would, reading the live page to find each control, against your real app rather than a cloud replica of it. Everything the agent does, and everything the page did in response, is recorded as the run’s trajectory.

On the judging side, the agent that drove the browser never grades its own work. A separate verifier model reads the recorded trajectory against the expected outcome and judges each written expectation strictly on evidence. It’s skeptical by instruction. A success the trajectory can’t substantiate counts as an expectation not met, however confidently the agent declared victory. The verdict comes back itemized, each expectation met or missing, and every run lands in exactly one of four outcomes, passed, failed, skipped, or blocked. Blocked means the test’s preconditions never held, which is an environment problem rather than evidence against your app, and it’s counted as neither pass nor failure.

Strictness needs a check on itself, so a judged failure isn’t final on the first attempt. The runner re-runs the failed attempt once on a stronger model, and the second verdict stands. A pass on retry is recorded as evidence the first failure was agent error. A second failure is recorded as high confidence the issue is real.

Here’s a real graded run from our QA environment, on an HCM test app:

That test required adding participants to one specific cycle, 360-QA-Regression-2026. The agent operated on a similarly named cycle instead, 360-Review-Fiscal-Year-2026, and the verifier failed the run with reasoning that names exactly that mismatch. That failure is the discipline plain English needs before you can trust it. The sentence allowed a wrong-but-plausible reading, and the grader caught it because the expected outcome wasn’t loose.

A failure that survives all of that can become a bug candidate, carrying the verifier’s reasoning as context. Nothing files itself. A candidate reaches Jira only after it’s checked against existing issues for duplicates and a person reviews it and clicks to file. The verdicts page walks this grading pipeline end to end.

Can you dictate a test by voice?

Yes, and it turned out less gimmicky than I expected when we wired it up. The mic opens a live voice session, you talk, and the prompt box fills in as you speak. Editing works by voice too, in the register you’d actually use, telling it to replace one phrase with another or to scratch the last sentence, and the draft updates live. The fastest way to spec a flow is often to walk through the app while narrating what you’re doing, and dictation matches how that knowledge actually arrives.

Two honest notes on the plumbing, because billing claims deserve exactness. Voice capture is the one part of the product that doesn’t run on Anthropic models. The live session runs on OpenAI’s Realtime API, billed to your own OpenAI key, and the handshake is proxied through your local server so the key never reaches the browser extension. The spoken edits themselves are applied to your draft by Claude, on the same bring-your-own-key terms as everything else. The Claude calls that drive and grade tests bill your own Anthropic account directly, with no markup and no per-seat fee, and the server itself is self-hosted, running on your machine against your own database. The economics page lays out that cost structure in full.

What doesn’t plain English fix?

Ambiguity, lazy specifications, and nondeterminism, for a start, and pretending otherwise is how the phrase gets a bad name.

Ambiguity first. A prompt that would confuse a new teammate will confuse an agent, and the wrong-cycle failure above is what that looks like in practice. Plain English moves your rigor, it doesn’t retire it. The effort that used to go into locator hygiene now goes into prompts that name their targets precisely and expected outcomes that could actually fail. A test whose expected outcome would be satisfied by nearly any run can’t lose, and a test that can’t lose verifies nothing. I keep a separate checklist for writing expected results that hold up, and it matters more in this world, not less.

Determinism is a real tradeoff too. A script clicks the same element in the same order on every run. An agent interpreting intent may take a slightly different path to the same outcome, which is precisely what makes it resilient to UI change and precisely what a team needing step-for-step reproducibility should weigh honestly before switching.

The layer matters as well. Plain-English tests don’t replace unit tests, which are fast, cheap, and belong in CI on every commit. This approach earns its cost at the end-to-end regression layer, the multi-step flows through a real UI that are the most expensive tests to write by hand and the first ones abandoned when a suite starts rotting.

So, my decision rule, compressed. If your reds cluster after frontend deploys and the fixes are locator edits, the English-as-artifact bet removes that entire failure class. If you need bit-identical replay of exact steps, keep scripts on those paths. Either way, hold every tool claiming testing in plain English to the two probes from the top. Change one word and see what must be regenerated. Ask where a selector could hide. The phrase means something only when the answers are nothing and nowhere.

Why Regression Suites Break: 5 Failure Modes

Sun, 30 Aug 2026 00:00:00 GMT

The first version of this post went up in early June and blamed three things. Selectors that encode markup, waits that encode luck, and red results that arrive with no explanation attached. The months since, spent building TestAutomate’s runner and reading the recorded verdicts it produced against a real HCM test environment, convinced me that list was short by two, and that I’d been treating why regression suites break as the wrong kind of question. The interesting part isn’t the inventory. It’s which entries discipline can fix and which are baked into the test artifact itself.

Regression suites break in five distinct ways. Selectors couple tests to markup instead of behavior. Fixed waits couple them to one machine’s timing. Shared data couples tests to each other. Reporting that can’t tell a defect from an environment problem makes every red an investigation. And maintenance cost compounds across the first four until repair can’t keep pace with change. Three of the five are architectural. Discipline can’t fix them, because the brittleness lives in the artifact.

That sorting matters because teams spend years applying discipline to architectural problems, and the discipline keeps almost working. So let me take the five one at a time, with the test I use to tell one kind from the other.

Why regression suites break: the five failure modes

Here’s the map, in the order I meet them when I read a decaying suite:

Selector coupling. Tests assert markup, so refactors fail them without a defect.
Timing assumptions. Fixed waits encode one machine’s speed and lose the race everywhere else.
Shared test data. Tests lean on state other tests created, so order and history decide the verdict.
Verdict-free reds. The harness reports exceptions, not classifications, so every failure costs an investigation.
Maintenance economics. The repair bill compounds across the first four until neglect becomes rational.

The first four are causes. The fifth is what they add up to, and it’s usually the one that actually kills the suite. Teams rarely decide to abandon a regression suite. They fall behind on it, one skipped red at a time.

Why do selectors break tests nobody touched?

Because a selector is a claim about the DOM at the moment of authoring, and the DOM is the part of an application that changes most freely. A test that clicks .btn-primary-2.css-1x9f isn’t testing your product. It’s testing your markup. That example survives from the original version of this post because nothing about the pattern has changed since. Rename a class, restructure a component, upgrade the UI library, and the test fails even though no behavior a user could notice is different.

The direction of the coupling is the whole problem. The application is allowed to change its implementation at will. The test froze one implementation detail and made it load-bearing. Modern frontends make this strictly worse, since CSS-in-JS tooling hashes class names per build, which turns routine deploys into locator invalidation events.

My test for this mode is simple. If a diff that changes zero user-visible behavior can fail a test, that test asserts implementation, and it will keep breaking for as long as engineers keep refactoring. Page objects are the discipline answer, and they’re worth having, but look at what they actually do. They centralize the coupling so each repair is cheaper. The breakage rate doesn’t move, because the selectors are all still there, just tidier. That’s why I file this mode as architectural. The artifact has to stop containing selectors, which is the case I make in full in selectorless testing, explained.

Why do automated UI tests keep breaking on timing?

Because a scripted test can’t observe, it can only wait and then act blindly, so every wait is a guess about how fast the app will run on a machine the author never sees. The empirical study of flaky tests by Luo and colleagues found async waits to be the single most common cause in the projects they analyzed, ahead of concurrency and test order. That matches every suite I’ve read. sleep(2000) is a bet that the network settles inside two seconds, and the bet’s loss rate changes with the runner’s CPU, the state of a cache, and the time of day.

Waiting on conditions instead of clocks is real discipline and really helps. Martin Fowler’s essay on eradicating non-determinism in tests laid that playbook out years ago. But conditions are authored guesses too. The author has to predict what “ready” looks like for every step, and applications are inventive about looking ready while they’re not, with spinners that unmount and remount and content that renders before its data arrives.

I call this mode architectural because the durable fix requires the runner to perceive rather than pause. TestAutomate’s agent works in a real browser, through your own Chrome, and observes the page as it acts, proceeding when the app has actually responded rather than when a timer expires. Runs stay bounded, so nothing hangs forever. The execution model is built this way precisely so the pacing can belong to the app instead of the author.

How does shared test data break a suite?

Quietly, and in dependency order. The classic shape is a chain. Test 12 creates a record, test 30 edits it, and everything works until someone runs test 30 alone, or in parallel, or after test 12 failed, at which point test 30 goes red with no defect anywhere in the app. The suite’s pass rate now depends on execution history, which is a property nobody ever meant to test. I walked through the full anatomy of these data cascades separately, because the data slice deserves its own post.

What makes this mode different from the previous two is that it genuinely yields to discipline. Give every test its own data, never let one lean on another’s leftovers, and the mode disappears. The honest caveat is that the discipline is a permanent tax, paid on every test forever, and suites under deadline pressure default on it fast.

Structure can lower the tax. In TestAutomate a precondition lives in an explicit, idempotent setup phase that has to end by declaring itself ready or blocked before the test’s own steps are allowed to run. When the precondition can’t be established, the run records as blocked rather than failed, because the application was never exercised and a failure would be a lie about it. The assumption still exists, but now it’s written down, checked on every run, and incapable of masquerading as a regression.

Why does every red cost an investigation?

Because most harnesses report exceptions, not verdicts. An element-not-found error might mean a real defect, a lost timing race, or a missing fixture, and all three produce the same red X with the same stack trace. The classification work still has to happen. It has simply been shifted onto whichever human reads the report, per red, per morning. That tax is what breaks trust, and broken trust is fatal. Google reported that almost 16% of its tests showed some level of flakiness, and once a team learns that red usually isn’t a defect, rerun-until-green becomes the culture and real regressions ride out the same door.

That’s a real capture from our QA environment, and the durations tell the story on their own. Run 33 failed 2 of 2 tests in 12 minutes 10 seconds, a genuine investigation. Runs 34 through 36 each ended blocked in about a minute, 36 seconds, 49 seconds, and 1 minute 3 seconds, which says the environment needed fixing and says nothing bad about the app. On a conventional dashboard all four rows would be the same color.

The structural fix is to make the run produce a classification instead of an exception. Every TestAutomate run ends in one of four verdicts, passed, failed, skipped, or blocked, and the grading is done by a separate verifier model that reads the recorded evidence of the run and itemizes which expectations were met and which weren’t, instead of taking the acting agent’s word for it. When a test genuinely fails, the runner re-runs it once on a stronger model before standing behind the failure. And a failure’s evidence only ever becomes a Jira issue after a human reviews the candidate and clicks to file it. None of that comes from writing more careful assertions in a framework whose only outputs are green and an exception.

Why does test maintenance eventually eat the suite?

Because the repair bill is a product of three factors, how fast the app changes, how much coupling each test carries, and how many tests there are, while the capacity to pay it is a fixed slice of team attention. Every mode above is a coupling multiplier. Selectors break on refactors, waits break on infrastructure changes, shared data breaks as the suite grows, and verdict-free reds multiply the cost of every break by the triage it takes to understand it.

The death is rarely a decision. It’s rational neglect. On any given day, skipping the red test is cheaper than repairing it, so skips accumulate, quarantine becomes a one-way door, and the suite’s effective coverage shrinks while its nominal size keeps growing. Six months later there’s a folder of tests nobody believes, which costs more than no tests at all, because it still burns CI minutes and still interrupts people with false alarms.

The diagnostic I trust is the repair audit from the FAQ below. Pull a month of test-repair commits and look at what they touched. Repairs that mostly changed expected behavior mean the app changed and the suite did its job, which is the normal price of coverage. Repairs that mostly changed selectors, waits, and fixtures mean the budget went to incidentals, and no future month will be better, because the artifact guarantees the same repairs recur.

Which failure modes are architectural, and which are discipline?

Three of the five live in the artifact and the harness, one yields to discipline, and one is downstream of the rest. Laid out honestly:

Failure mode	Where the brittleness lives	Does discipline fix it?
Selector coupling	In the test artifact	No. Page objects make repairs cheaper, but the breakage rate stands
Timing assumptions	In the artifact’s static pacing	Partly. Condition waits help until “ready” itself must be guessed per step
Shared test data	Between tests	Yes, with isolation and setup hygiene, paid as a permanent tax
Verdict-free reds	In the harness’s reporting model	No. An exception can’t classify itself
Maintenance economics	Downstream of the other four	Only by shrinking the per-break cost, never by resolve

Two caveats on that table. Disciplined teams do run long-lived scripted suites, and I won’t pretend otherwise. They manage it by paying the tax continuously, staffing suite ownership like production ownership, and treating every red as tracked work. That’s a real option with a real payroll cost. And “architectural” doesn’t mean hopeless. It means the fix has to change the artifact, not the team’s habits.

What does an architectural fix actually look like?

Like removing the coupled artifact instead of patching it. A TestAutomate test is three required fields, an id, a prompt, and an expected outcome, written in plain natural language, and the generation rules flatly ban code, JSON, YAML, and selectors from appearing in them. You can type or dictate a test the way you’d brief a colleague. There’s no locator for a refactor to invalidate and no sleep to mistune, because there’s nowhere in the format to put either. The agent finds the controls on the live page at run time, in a real browser, and the timing mode dissolves for the same reason, since the agent acts when the app has responded.

The verdict layer covers the fourth mode, with the four outcomes and the skeptical second-model grading described above. What’s left, honestly, is the tradeoffs. An agentic run spends real model tokens on every execution, and it’s slower than a bare scripted assertion, which is part of why the economics of the product start from keys you bring yourself, billed at cost. A verdict is a judgment on recorded evidence, not a mathematical certainty, and the escalation retry exists exactly because a first failed attempt can be the agent’s error rather than the app’s. And nothing here removes nondeterminism that genuinely lives in your app. What it removes is the five ways a suite manufactures failure on its own.

The question I’d put to any regression suite, ours included, is the one this post kept circling. When it breaks, what changed, the product’s behavior or the test’s assumptions about implementation? A suite that mostly breaks for the first reason is an asset. One that mostly breaks for the second is a countdown, however disciplined the team, because the artifact sets the breakage rate, and the artifact is the one thing discipline never touches.

AI Bug Reports With Reproduction Steps

Tue, 25 Aug 2026 00:00:00 GMT

The slowest part of finding a bug was never finding it. It was writing it up. Repro steps someone else could follow, screenshots captured and attached, and a search of the tracker first, because a duplicate came straight back to me.

AI bug reports with reproduction steps earn a reader’s attention when they prove three things fast. That the defect is real, that the reader can reproduce it without asking the reporter anything, and that it isn’t already in the tracker. The steps have to be literal and ordered, the evidence checkable rather than described, and the duplicate check has to have run before the ticket existed.

I’ve been on both ends of that transaction. I’ve written the careful report and watched it sit unread, and I’ve picked up someone else’s report and given up before the second line. Filing bugs across several teams taught me that writing effort and reading value come apart badly. You can spend real time on a ticket and still produce something nobody can act on, usually because the effort went into describing the bug rather than into letting a stranger re-create it.

The three questions a reader asks in the first two lines

A developer opening a bug report is not reading it. They’re triaging it, and triage is a fast sort into act now, act later, or close. The sort runs on whatever fits in the first couple of lines, which in most trackers means the summary and the opening of the description. Everything you wrote below that gets read only if those two lines survive.

The three questions behind that sort are always the same. Is this the app’s fault or the reporter’s? Can I get to this state myself? Have I seen this before? A report that answers all three up top buys the rest of its length. A report that answers none of them gets closed as cannot reproduce, which is the polite version of I didn’t believe you enough to try.

That constraint is visible in the tooling, not just in habit. In TestAutomate’s dashboard, a pending bug candidate renders as a severity badge, a one-line summary, and the first couple of lines of the details, with File and Dismiss sitting right there. The reviewer decides from that much, which is an honest rendering of how bugs actually get read.

My working rule for the summary has three parts. Name the feature, name the action that failed, and name what the app did instead. “Participants not added to cycle” is a subject line. “Add Participants saves against the wrong review cycle” is a bug report someone can sort. The second version answers the fault question and the have-I-seen-this question in the same breath, which is also what makes it findable later by whoever searches before filing.

What makes reproduction steps actually reproducible?

Steps are reproducible when a stranger can follow them cold. That means naming the starting state, the exact records and accounts involved, every action in order, and the observed result at the end. The failure of most repro steps isn’t missing detail, it’s substituted detail, a description of what the reporter did instead of the literal sequence they performed.

The classic guidance here has barely aged. Mozilla’s bug writing guidelines still ask for precise steps, the expected result, and the actual result as three separate things, and the reason that structure works is that it forces the reporter to separate observation from interpretation. Most weak reports collapse those into one sentence and lose the observation.

Two habits fix most of it. The first is naming rather than describing. “Open the review cycle” is a description. “Open the review cycle named 360-QA-Regression-2026 from the Cycles list” is a step. Anything the reader has to guess is a place where their reproduction can quietly diverge from yours, and a diverged reproduction produces a false cannot reproduce, which is the worst outcome available because it discredits a real defect.

The second is ending at the observed failure and stopping. One bug per report, no speculation about the cause, and no bundled second issue you noticed on the way. Diagnosis in a bug report is a gift to nobody. If your theory is wrong, it sends the reader down your wrong path and they come back annoyed. If it’s right, they’d have got there anyway from a clean set of steps.

An agentic run’s real advantage is that it never has to remember. A recorded run already holds the ordered sequence of actions and the state at each point, so the steps come from the record rather than from recollection at the end of a long afternoon. Human reporters reconstruct. That’s where detail goes missing, and it was the part of the job I was worst at.

What do good AI bug reports with reproduction steps look like?

They read like a run report with a point of view. When a regression test fails in TestAutomate and survives triage, the candidate carries a one-line summary, the name of the test that found it, the verifier’s reasoning for the failure, and the itemized list of expectations that went unmet, plus a rough severity and the context of where it was observed. Unmet expectations are the machine-readable version of “actual result”, and they’re only as good as the expectations a human wrote in the first place, which is why writing a sharp expected outcome is upstream of every readable report the system will ever produce.

Here’s a real failure rendered on the dashboard, from a recorded suite run against a test app.

The reasoning names the specific thing that went wrong. The run acted on a cycle called 360-Review-Fiscal-Year-2026 when the test required 360-QA-Regression-2026, so the expectations tied to that cycle came back unmet. Read it as a bug report and it does the reader’s first sort for them. There’s a named record, a named divergence, and a stated consequence. Notice also what this view doesn’t do. It shows the reasoning summary, not the full itemized met and missing checklist, which lives in the run report rather than in the timeline. A report that overstates what it shows you is its own kind of unreliable, so it’s worth knowing which surface holds which evidence.

The honest limit is that a generated report describes the run in the run’s own terms. It knows what it did and what the app did back. It doesn’t know that your team calls that screen something else, or that this area was rewritten last sprint, or that the customer who hit this is on a call right now. That context is the reviewer’s contribution, and it’s why the report is a draft for a human rather than a replacement for one. TestAutomate treats it that way. The candidate is prepared, and a person still decides.

Why most red tests should never reach a bug report at all

Because a red result is not a defect claim, it’s a signal that something didn’t go as written, and the something is often the test or the environment. Filing every red fills a backlog with tickets nobody trusts, and once a queue has taught its readers that most entries are noise, the good report loses too.

TestAutomate puts three filters in front of the reader, and each one removes a different kind of noise. A run whose preconditions were never established comes back as blocked rather than failed, so it’s an environment problem and never counted as a failure or turned into a bug candidate. A run that genuinely failed under judgement is re-run once on a stronger model first, because a first attempt can fail for agent reasons rather than app reasons, and treating those as defects is how false alarms get manufactured. What survives both then goes to a triage step that decides whether the failure describes a real application defect or a test problem, a missing precondition, a timing wobble, a test written for a build that doesn’t ship that feature. AI bug triage here decides what enters the queue, not how to sort it afterwards. Only defects become candidates, and the judgement is deliberately conservative about that call.

The reason to care as a reader rather than as an engineer is that this is what makes the queue mean something. Every card in the candidate list traces back to a run where the app itself did something wrong, and that property is what earns the review. A webhook that fires on red never establishes it, because it has no way to tell a broken feature from a broken fixture.

If you’re doing this by hand, the same filter works without any of the machinery. Before you file, ask whether the app misbehaved or whether your setup did, and whether you can make it happen again. Two questions, and they catch most of what would have been bounced.

A duplicate ticket wastes the reader you were trying to help

A duplicate doesn’t just fail to help. It costs someone a triage slot, a search, a link, and a close, which is a worse trade than never filing. That’s the part of bug filing I resented most, because the search takes real time whether or not it finds anything, and skipping it was how I earned tickets bounced straight back at me.

So the check belongs before the ticket exists, not after. When someone clicks File on a candidate, TestAutomate searches live Jira for issues whose summaries overlap, folds in similar entries from a cached inventory of the board, and hands that shortlist to a judge instructed to be strict about what counts as the same defect rather than merely the same area of the app. If it finds a likely match, nothing gets created. The dashboard shows the existing issue’s key, its status, and its summary, and asks whether to file anyway, so forcing a genuinely new ticket is still one click. Preventing duplicate Jira tickets from test failures is mostly this, a check that runs at the moment of creation instead of a cleanup pass someone runs later.

The failure direction is chosen on purpose. If the duplicate check can’t run at all, the result is treated as no match and the create path is offered, because a duplicate is recoverable by linking and closing while a silently swallowed defect is just gone. Jira’s own issue linking makes that recovery cheap. The wider filing pipeline this check sits inside is walked through in how mid-run defects become reviewed Jira candidates.

Dedup also rewards good writing, which is the part teams miss. A vague summary matches nothing and produces a false clean bill of health. The feature-plus-action-plus-result summary from earlier is what makes both the machine check and the human search work, so the writing rule and the duplicate rule are the same rule.

Where a report comes from changes what a reader gets

The same defect produces very different tickets depending on what generated the report, and the differences show up in exactly the places a reader cares about. Here’s the honest comparison across the ways a red check becomes something in a tracker.

Report source	What the first two lines carry	Can a reader reproduce without asking?	Duplicate check before it lands	What the reader still has to do
CI webhook auto-filing on a red test	Test name and a job status	Only if they can read the job’s logs and re-run it themselves	None by default, so retries arrive as clones	Reconstruct the scenario from the pipeline output
Record and replay playback failure	The step where playback diverged	Partly, by watching the recording back	None	Decide whether the divergence is a defect or a stale recording
Managed QA service report	The vendor’s written summary	Usually, since a human wrote the steps	The vendor’s internal process, which you don’t see	Wait for the reporting cycle, then ask follow-ups through the vendor
A careful human write-up	Whatever the tester chose to lead with	Usually yes, when the tester had time to be literal	The tester’s search, if the day allowed one	Little, which is why this is still the benchmark
TestAutomate bug candidate	Severity, a one-line summary, and the opening of the evidence	Yes for the sequence and the records involved, from the run’s own record	Runs at the moment of filing, strictly judged, human-confirmed	Supply team context and decide whether it’s worth the backlog

Two caveats belong with that table. A careful human write-up remains the best bug report in any tracker, and nothing here beats it on judgement. What it competes with is the rest of that person’s day, and it loses often enough that most trackers are full of the hurried version instead. CI auto-filing is also a perfectly reasonable trade for a small suite whose failures are rare and nearly always real. My objection isn’t the automation, it’s that the duplicate guard and the defect-versus-environment question are both left as work somebody has to bolt on afterwards.

What I check before I let a report out

Four things, in order, whether a person or an agent drafted it.

Does the summary name the feature, the failing action, and the observed result? If not, rewrite it, because that line does the sorting, the searching, and the duplicate matching all at once.

Can someone follow the steps on a fresh session without asking me anything? Read them as a stranger, and anywhere you’d have to guess a record, an account, or a starting state, name it instead.

Is the evidence checkable rather than described? For a functional defect the ordered steps and a precise account of what the app did instead usually beat a picture. For layout and rendering, the picture is the evidence, and the same principle holds for tooling that keeps a run’s own record so the failure can be inspected after the fact rather than re-narrated.

Did anything actually check for a duplicate? A search that only happens when the reporter has time isn’t a check, it’s a hope.

None of this makes bug reporting glamorous work. It does make it cheap enough to keep doing properly, which is the whole point. The reason I care that an agent drafts the mechanical parts is not that machines write better prose. It’s that a recorded run doesn’t get tired late in the day and start summarizing, and the pipeline still hands the judgement, the context, and the filing decision to a person, which is the division of labour worth keeping.

Agentic Testing Guardrails That Matter

Thu, 20 Aug 2026 00:00:00 GMT

Every real design argument we had while building TestAutomate’s runner collapsed into one question: what is this agent allowed to conclude on its own? That question is more useful than it sounds, because an autonomous tester that can reach any conclusion it likes is not a testing tool. It’s a machine for generating confident-sounding results.

Agentic testing guardrails are the constraints that decide which conclusions an autonomous test agent may reach by itself and which ones stop and wait for a person. Four of them carry most of the weight: a result vocabulary that separates environment problems from real failures, a grading pass independent of the agent being graded, runs that are bounded instead of open-ended, and a human click before anything leaves the system and lands in someone else’s queue.

Most writing about agentic AI in QA borrows its oversight thinking from agents in production, where the worry is an agent moving money or emailing a customer. The QA version of the conversation usually ends at “keep a human in the loop,” which is advice with no shape to it. A loop has to have a location. What follows is where we put ours, and why I’d ask the same questions of any tool in this category.

What are agentic testing guardrails?

They’re the limits on an agent’s authority, not limits on its capability. The distinction matters because the two get conflated constantly. A capability limit says the agent can’t reach a part of the app. An authority limit says the agent can reach it, act on it, and still not be the thing that decides what the result means. Nearly all the safety in an agentic test system comes from the second kind.

This maps onto the oversight language regulators have settled on. The EU AI Act’s Article 14 frames human oversight as the ability to correctly interpret a system’s output and to decide not to use it, which is a much sharper requirement than “a human is nearby.” The NIST AI Risk Management Framework makes the same move under Govern, treating accountability as something you assign in advance rather than reconstruct after an incident. Neither is written for QA, but both describe the failure mode a testing agent actually has, which is producing an output nobody can interpret and nobody owns.

The practical version is a short list of questions. What can this agent declare true without a second opinion? What does it do when it can’t tell? What can it send outside the system? If a tool can’t answer those three, its guardrails are marketing.

The verdict vocabulary is the first guardrail

Before anything else, a run has to be able to say it proved nothing. Most automation can’t. A script throws an exception, the report paints it red, and that red carries no information about whether the app misbehaved or the test never got started. Teams learn that red is ambiguous, and once red is ambiguous it stops being urgent.

Every run in TestAutomate ends in one of four outcomes: passed, failed, skipped, or blocked. Blocked is the one doing the guardrail work. It means the preconditions the test depends on could not be established, so the flow under test never executed and the run is not evidence about your application. It’s recorded as neither a pass nor a failure and counted separately, which is the whole point. I’ve written before about why a blocked run is not a failure, and the reason it belongs in a guardrails discussion is that a system without this vocabulary has no way to be honest about its own uncertainty. It has to call something a pass or a failure, so it guesses.

That capture is from our own QA environment. Reading down the timeline, the three most recent runs each ended blocked, one blocked outcome out of one test, finishing in 36 seconds, 49 seconds, and 1 minute 3 seconds. The run before them failed 2 of 2 and took 12 minutes 10 seconds. Those numbers describe two different situations, and a report that folded them together would have shown five consecutive failures and taught the team to stop looking.

The same principle runs one level deeper. A verdict doesn’t arrive as a single word. It arrives with each expectation you wrote marked met or missing, so a failure tells you which specific claim didn’t hold. Expectations can also be marked as incidental when they’re genuinely nice-to-have, and a missing incidental expectation can’t fail the test. That’s a guardrail against a different problem, the suite that goes red because a confirmation toast changed wording.

Should the agent that runs a test also grade it?

No, and this is the structural decision I’d defend hardest. The agent that drove the browser has a record of what it was trying to do and a natural read on whether it succeeded. Asking it to grade itself is asking the least objective available party. In our runner the grading is a separate pass over the recorded trajectory, and it’s told plainly not to take the agent’s word for it: an expectation with no supporting evidence in the trajectory is marked missing, not assumed satisfied.

The bias this defends against is specific. An agent that clicks Save, sees a spinner, and moves on will summarize that as saving the record. A separate pass looking for evidence that the record actually saved will not find it and will say so. The instruction that governs this is blunt about the asymmetry, that a false pass is much worse than a false negative, because a false pass is a silent hole in your coverage while a false negative is an annoying morning.

There’s a matching guardrail on the other side, since a strict grader will occasionally fail a run over the agent’s own clumsiness rather than a real defect. When a genuinely judged failure comes back, the act phase is re-run exactly once on a stronger model. If the stronger attempt passes, the first failure was an execution problem and gets recorded as such. If both attempts fail, that’s high confidence something is actually wrong. It’s one retry, not a loop, and it never fires on a blocked or skipped run, because re-running a test whose preconditions were never met would just waste time and produce another non-answer. I covered the reasoning in more depth in the post on false positives in automated testing.

How do you stop an agent from inventing a bug?

You stop it from filing. Everything upstream of filing is cheap to get wrong and cheap to correct, and everything downstream is expensive: a ticket in a tracker consumes a triager’s attention, gets assigned, gets discussed, and has to be closed by a human even when it was nonsense from the start. Automatic filing puts the cost of the agent’s worst moments onto your team, and it does it fastest precisely when the agent is misreading the app most badly.

So a failed run doesn’t become a ticket. It becomes a candidate. A triage step first decides whether the failure looks like a real application defect or a test and environment problem, and only defects become candidates at all. Blocked and skipped runs never reach that step, which means a missing fixture record cannot turn into a bug report about your product. The candidate then sits in a pending queue with its reasoning and its unmet expectations attached.

Filing happens when a person clicks File, and not before. At that moment the candidate is checked against existing issues, and if it matches something already open, the duplicate comes back for review instead of a new ticket being created. The dedup check is a guardrail on the guardrail, since the fastest way to lose a team’s trust is to file the same defect four times. The full path from a mid-run failure to a reviewed ticket is in the post on autonomous bug reporting.

Where the human decision point sits

The useful comparison between testing approaches isn’t which one is most autonomous. It’s where the human judgment lands, and whether the tool tells you enough to exercise it. Here’s how the common approaches differ on that, in my reading of them.

Oversight question	Scripted and selector-based suites	Record and replay	Managed QA service	TestAutomate
What does a red result mean?	An exception was raised, from either an app defect or the environment	The replay diverged from the recording, cause unspecified	Whatever the vendor’s engineers decided it meant	One of four outcomes, where blocked explicitly means the app was never exercised
Who decides a failure is a defect?	A human reading stack traces after the fact	A human comparing the replay against the recording	The vendor, before you see it	A triage step proposes it, a person confirms it
Who files the bug?	A human writes and files it	A human writes and files it	The vendor files on your behalf	A person clicks File on a prepared candidate
What happens on an ambiguous run?	Surfaces as a red result to be investigated	Usually resolved by re-recording	Absorbed by the vendor and typically invisible to you	Recorded as blocked, or as a pass carrying a flag
What does oversight require of you?	Reading traces and maintaining locators	Re-recording flows as the UI moves	Trusting a process you can’t inspect	Writing expectations, then reviewing verdicts and candidates

No row there is free. Ours moves the human work from repairing and triaging toward specifying and reviewing, which is a better trade for most teams but not all of them. A team with a large stable scripted suite and a UI that rarely changes shape is not obviously better off switching. The honest claim is that the review work is smaller and earlier, not that it disappears.

The other thing worth naming is the managed QA row. Handing triage to someone else genuinely reduces your workload, and it also removes your ability to inspect the judgment. That’s a real oversight tradeoff, not a knock on the model, and teams choose it deliberately all the time.

What these guardrails still don’t cover

Plenty. The grading pass reads a recorded trajectory, so a defect that leaves no trace in that record is a defect it can’t see. Expectations are written by people, and a vague expectation produces a vague verdict, which is why writing a good expected outcome matters more than any setting in the product. If you write “the page looks right,” nothing downstream can rescue you.

Runs are bounded so a confused agent stops rather than grinding indefinitely, a scoping choice grounded in the open-ended autonomy numbers, but a bounded run that stops early still needs a person to read why it stopped. The dedup check compares against a pool of existing issues rather than every ticket ever filed, so it catches the duplicates that matter most and not every possible one. And none of this addresses coverage, the oldest problem in testing. An agent that faithfully runs the eleven tests you wrote will tell you nothing about the twelfth thing your users do.

I’d rather state those limits than let a guardrail list imply completeness. The four constraints above are about making each result mean something specific, so a green suite is a claim you can actually check and a red one points at a real thing. If you’re evaluating anything in this category, including ours, the questions to ask are the ones I opened with. What can it declare true alone, what does it do when it can’t tell, and what can it send outside the system without asking. The answers tell you more than any feature list. For the wider picture of how these runs work end to end, the complete guide to agentic testing covers the mechanics this post assumes.

Self-Healing vs Selectorless Testing

Thu, 20 Aug 2026 00:00:00 GMT

More than once in my testing years, a frontend redesign has taken a big slice of the automated suite down with it. Not because the product broke. The features worked fine when I clicked through them by hand. What broke were the locators, the stored CSS and XPath strings each script used to find its buttons and fields, and until we re-mapped them the suite had nothing true to say about the product.

Self-healing vs selectorless is the choice between two answers to that repair bill. A self-healing tool keeps the stored selector and patches it when it stops matching, using fallback attributes, nearby text, or element history to relocate what the test probably meant. A selectorless tool never stores a selector at all. The test states intent in natural language, and an agent finds the element on the live page, fresh, every run. One repairs the artifact. The other deletes it.

Full disclosure before we start: TestAutomate, whose blog this is, sits on the selectorless side, and so do I. But I carried a locator repair queue for years, so I want to make the self-healing case properly before I argue with it. The real question isn’t which pitch sounds smarter. It’s where you want your maintenance work to live, and which kind of mistake you can least afford.

What does a self-healing selector actually do?

It patches a broken locator at run time. When the stored selector stops matching, the tool hunts for the element the test probably meant, scoring candidates by fallback attributes, position, visible text, or a remembered profile of what the old element looked like, then swaps its best guess in so the step can proceed. Most implementations follow the same three-beat shape: notice the dead selector, relocate the element, update the stored locator so the next run uses the patch.

For the breakage class I met most often, that genuinely works. A build ships, submit-order becomes checkout-submit, and a script that would have died on the stale string sails through because the healing layer matched the button by its text and role instead. Nobody loses an afternoon to re-mapping. That’s real value, and I won’t pretend otherwise.

Self-healing test automation is also the newest chapter of a very old story. A selector is a pattern matched against markup, so the honest answer to why selectors break tests is that every stored selector is a bet that the markup will hold still. Teams have known that bet was fragile for as long as UI automation has existed, which is why the Page Object pattern told us to quarantine element location in one layer long before anyone put machine learning near a test suite. Page objects centralized the repair. Self-healing automates it. Neither questions whether the test should be betting on markup at all.

Why do automated UI tests keep breaking even with self-healing?

Because healing repairs identity, not structure. A heal can find the renamed button. It can’t invent the extra confirmation step your checkout flow just grew, retarget a journey whose pages were merged into one, or decide what “the submit button” means now that the redesign put two of them on screen. When a UI changes shape rather than spelling, the healing layer is out of its jurisdiction, and you’re back to editing scripts by hand.

It helps to separate two words that get blurred in the AI test maintenance pitch: this breakage is brittleness, not flakiness. A brittle test fails deterministically when the markup changes. A flaky one fails nondeterministically with no change at all, from timing, data, or environment, and that failure class has its own playbook. Self-healing addresses the brittle half only. If your suite goes red for timing reasons, a locator patch has nothing to offer it.

There’s also a quieter cost inside the healing wins. Every heal is a heuristic guess, and a wrong guess doesn’t fail. It passes. The script clicks a plausible but wrong element, the assertion happens to hold, and the suite reports green against a flow no user takes. The better self-healing tools are candid about this, which is why they log every patch and queue the uncertain ones for a human to approve. Read that arrangement plainly. The bargain trades repair work for review work, and the broken selectors I used to fix by hand become a heal log somebody has to audit with the same care, because an unreviewed heal is a test whose meaning changed while nobody was watching. A test that needs a fresh heal every run isn’t being maintained by the tool anymore. It’s being resuscitated by it, and the locator strategy underneath needs a rethink, not another patch.

Self-healing vs selectorless: where does the maintenance go?

It doesn’t disappear in either model. It changes shape, and choosing between the two is really choosing which shape of work you’d rather staff.

With self-healing, the suite remains a codebase of scripts and stored locators wearing an automated repair layer. The recurring work is reviewing heals, hand-editing the structural changes healing can’t reach, and keeping the locator strategy sane enough that the repair layer isn’t permanently busy.

Selectorless removes the stored artifact instead of protecting it. The full definition of what selectorless testing means is on this blog already, so I’ll keep to the practitioner’s view. In TestAutomate a test is three required fields, an id, a prompt stating the intent, and an expected outcome, plus optional setup and cleanup prompts written the same way. That’s natural language test automation in the literal sense: there is no field in the artifact where a CSS class or an XPath could live, and even the tests the product generates for you are barred from embedding code or selectors. At run time the agent reads the live page and finds what it needs the way a person would, so when the UI changes there’s no stored string to go stale and nothing to update. Rename the button, and the next read simply finds the renamed button.

That’s the real trade in test automation without selectors. You stop maintaining how elements get found and start maintaining what the product is supposed to do, which means keeping expected outcomes honest as features evolve and reading the verdicts that come back. Intent-based testing is a fair name for it. You write the intent once, and the ongoing work is editorial, not archaeological.

Here’s the contrast in one table, alongside the neighbors people usually shortlist with these two:

Approach	Where element identity lives	A renamed control	A redesigned flow	The recurring work
Scripted, selector-based	Author-written CSS or XPath in test code	Test fails on a stale selector	Hand-rewrite the affected scripts	Locator upkeep on every UI change
Scripted plus self-healing	The same selectors, plus fallback data the tool maintains	Usually healed, and the patch joins a review queue	Healing can’t add or reorder steps, so hand edits remain	Reviewing heals, fixing what they miss
Record and replay	A recording captured while a person clicked through	Playback halts where the recording mismatches	Re-record the flow end to end	Re-recording and re-approving flows
Managed QA service	A human tester’s understanding of the app	Absorbed by the human	Absorbed, at human turnaround and per-cycle cost	The invoice, and the wait
TestAutomate, selectorless	Nowhere. The agent finds elements on the live page each run	Absorbed on the next read	The same stated intent still runs, and outcome changes are graded rather than guessed at	Reading verdicts, keeping expected outcomes true

Honest caveats, because tables flatten things. A disciplined team with a large, stable scripted suite gets real mileage from a healing layer, and rewriting that investment as intent-based tests isn’t free. Record and replay starts fast and stays fine for flows that rarely move. Managed services genuinely absorb UI change too, humans are excellent at that, but you’re renting the absorption per cycle instead of owning it.

What breaks a selectorless test instead?

Ambiguity, genuinely missing elements, and lazy expected outcomes. No-selector testing doesn’t repeal failure. It moves the failure surface up from string matching to intent matching, and you should know exactly what lives up there before you commit.

If a page offers two controls that both plausibly satisfy “the export button”, an agent can pick the wrong one. That’s the same species of mistake as a wrong heal, relocated from repair time to run time, and pretending otherwise would be marketing. If an element is genuinely gone or never renders, no amount of fresh page reading will find it, and the run fails for the same underlying reason a stale selector would have failed. That one is a feature. A control that vanished is exactly what you want a red result for.

The failure mode that’s truly yours to own is the expected outcome. An adaptive executor plus a vague assertion produces tests that can’t lose, and a test that can’t lose isn’t testing anything. In TestAutomate the expected outcome is written as itemized assertions about the end state, including negative ones like no other records being modified, and any bullet prefixed [incidental] is barred from failing the run, so a missing success toast can’t outrank the point of the test. Writing those assertions well is a craft with its own checklist for expected results that hold up. The short version is that selectorless moves your rigor from selectors to specifications, which is where it should have been living all along.

Can you trust a test that adapts on its own?

Only if something other than the adapting agent grades the run. That’s the structural answer, and it’s the question I’d put to every tool in this category, ours included.

The worry is symmetric, after all. A self-healing suite can quietly heal onto the wrong element, and a selectorless agent could quietly adapt its way around a real regression. What separates the trustworthy versions of both is whether adaptation gets graded and disclosed or assumed correct by default.

In TestAutomate, the agent that drives the browser never grades its own work. A separate verifier reads the recorded run against the expected outcome and returns an itemized verdict, every expectation met and every expectation missing, under a standing instruction to stay skeptical and treat unevidenced success claims as not met. Every run lands in exactly one of four outcomes, passed, failed, skipped, or blocked, so an environment problem is never dressed up as a failure. Redesigns get a mechanism of their own. When the app’s behavior no longer matches what the expected outcome implied but the change looks deliberate, the verdict carries an assumption flag recording what was observed, what was expected, and why the change is probably intended, a mechanism explained properly in its own post. A flagged pass is still a pass. The drift is just on the record for a human to see, which is the disclosure a silent heal never makes.

Failures get a second look before anyone gets paged. A run that fails on the first attempt is retried once on a stronger model before the red is treated as real, because first attempts can fail for agent reasons rather than app reasons, and false alarms are how teams stop reading red at all. Hold all of that against the heal review queue and the shape of the difference is clear. Self-healing asks you to audit repairs to the finding mechanism. Selectorless with a separate judge asks you to review verdicts about the outcome, with the evidence itemized. I know which review I’d rather staff, partly because it’s the same review you’d want even if your UI never changed again.

How do I decide between self-healing and selectorless?

By the shape of your churn and the size of your sunk cost. Four rules I’d actually use, having sat on both sides.

If you own a large scripted suite and your churn is mostly attribute-level, renames and moved nodes rather than redesigns, a self-healing layer earns its keep. Treat heals like code review. Audit the log, and treat any test that heals repeatedly as a design smell to fix rather than a save to celebrate.

If your churn is structural, redesigns, component library migrations, variants that reshuffle whole flows, the heal reviews will pile up faster than they pay off, because the layer can’t follow structural change. That’s the world intent-based tests without stored locators were built for, and it’s the world my re-mapping years were spent in.

If you’re starting from zero, weigh who does the upkeep. A selector suite needs someone who repairs code. A selectorless suite needs someone who writes sharp expected outcomes and reads verdicts, which is a QA skill rather than an engineering one. The no-code AI test automation pitch is honest here as long as you hear it as no code with real editorial work, not no work.

Whichever way you lean, interrogate the trust story before the feature list. Ask a self-healing vendor to show you the heal log and who approves a patch. Ask a selectorless vendor what grades a run and what happens when the app changed on purpose. If either answer amounts to the system that acted also approved itself, keep walking.

The redesigns aren’t going to stop, and product teams are right to keep shipping them. The choice is what’s left holding the bet when the markup moves: a selector string somebody has to keep alive, or a stated intent the next fresh read can still satisfy, graded by a verifier that doesn’t take the agent’s word for it. The full selectorless loop, generation through verdict, is mapped in the agentic testing guide. As for me, I re-mapped locators after enough redesigns to know which side of self-healing vs selectorless my desk is on.

Generate Test Cases From Requirements AI

Fri, 14 Aug 2026 00:00:00 GMT

Release week had a shape, wherever I was working: an evening lost to a manual regression pass through everything the release note didn’t already cover, done by hand because there was never enough runway to script it properly first. Whatever we couldn’t get through before the cutoff got cut, quietly, and nobody went back to check what that cost later.

People search “generate test cases from requirements ai” expecting a shortcut, but most tools that answer the query need a requirements document to read, and mature products rarely have a current one for every feature under regression coverage. TestAutomate skips that input. It crawls the product documentation you already publish, turns what a page actually describes into a test, and hands you something you can run immediately, not a document someone still has to automate.

That’s not a small substitution. A requirements document describes intent, written before a feature shipped, and on a mature product it’s usually gone, buried in a ticket nobody can find, or describing a version of the flow that changed twice since. Documentation describes what the feature does right now, for the person using it, which is exactly what a regression pass on release week actually needs covered.

Why does “generate test cases from requirements AI” assume a document that isn’t there?

Because most AI test-case generators are built around a single input: a requirements document, a product requirements doc, or a user story, fed in by a person before generation starts. The tools that rank for this query mostly live inside test-management platforms, and their workflow begins from an artifact already sitting in the tracker. You select a requirement record, the generator reads its description and attachments, and structured test cases come back for review. Every step of that flow presumes the artifact exists, is current, and describes the whole feature rather than one long-ago slice of it.

That works fine for a feature being built right now, while its ticket is still open. It breaks down the moment you’re testing something that shipped a year ago, whose original requirements doc was never updated, or never existed as a discrete artifact in the first place. The standard advice for a requirements spec is to maintain it as the product evolves, and in practice almost nobody does once the feature ships, because the document’s audience was the team building it, and that team has moved on to the next thing.

On release week the coverage that got cut first was never the brand-new feature, the one with a fresh ticket and a spec somebody could hand to a generator. It was the older corners of the product, the ones that had shipped so long ago nobody remembered where the original write-up went, if one was ever produced instead of just discussed out loud in a meeting. Those are exactly the flows a requirements-first generator can’t help with, because there’s no requirements document left to read. There’s usually still a docs page, though, because someone has to tell users how the feature works even after the people who built it have moved on.

Can AI generate test cases from user stories and acceptance criteria?

Yes, and while the story is still in flight it’s arguably the best input available. A user story describes one change from the user’s point of view, and its acceptance criteria pin down what done means, often already phrased in the given-when-then shape. That structure maps almost one-to-one onto test steps and expected results, which is why asking an AI to generate test cases from user stories during a sprint works so well. The artifact is current, precise, and sitting right next to the code it describes.

AI test case generation from acceptance criteria inherits that strength along with an expiry date. Acceptance criteria capture behavior at the moment the ticket closed. Three releases later the same flow has picked up a confirmation dialog, a renamed button, and a permissions check, and none of those changes reopened the original story. Stories also describe deltas, not products. “Add bulk archive to the projects list” gives a generator everything about bulk archive and nothing about the projects list it landed in, so a suite generated purely from stories skews toward whatever changed recently and thins out across the accumulated older surface, which is the part release week actually puts at risk.

The other gap shows up after generation. The road from user stories to automated tests has two conversions in it. The first turns a story into test cases, and generators handle it well. The second turns those test cases into something a machine can execute, and most generators leave that half to you, handing over preconditions, steps, and expected results as text for a person or a scripting effort to act on later. A perfectly generated manual test case still lands in a backlog column labeled automate later.

So my decision rule runs on the age of the surface. Use story-based generation for the feature being built this sprint, where the artifact is fresh and the feature is too new for anything else to describe it. For the shipped surface, treat story archaeology as a last resort and reach for an input that’s still being maintained.

Which input source should you actually point a generator at?

The freshest one that exists for the surface you need covered, and that answer is different for a feature in flight than for one that shipped two years ago. Here’s how the four common inputs compare across their lifecycle, from the day they’re written to the release week when you actually need them to be true. Their executability trade-offs get a fuller treatment in the docs-to-suite walkthrough.

Input source	Written when	Who maintains it after ship	State by the release week you need it
Requirements document	Before the feature was built	Usually nobody, ownership dissolves at ship	Describes the plan rather than the product, and the two have diverged
User stories and acceptance criteria	While each ticket was open	Nobody, a closed story is archived rather than maintained	Scattered deltas, each true only of the sprint it shipped in
Screen recordings of the flow	The day somebody captured one	Whoever remembers to re-record	Exact until the first UI change after capture, then misleading
Published product docs, which TestAutomate reads	At ship, for the feature’s users	The docs owner, because users keep reading it	Current for anything users still depend on

Some honest caveats on that table, because documentation isn’t a perfect input either. Docs lag releases too, just usually by less than a spec nobody owns anymore. They over-describe the happy path and under-describe edge cases, so a docs-generated suite starts breadth-first and needs a human to add the sharp corners. An internal admin tool with no published docs gives a docs-first pipeline nothing to read at all. And for a feature that hasn’t shipped yet, requirements-first generation is genuinely the right call, since the feature doesn’t exist for any docs to describe. The point of the comparison isn’t that one input wins everywhere. It’s that the input should match the age of the surface you’re covering, and the older that surface gets, the more often its docs are the only written source still telling the truth.

How does test generation work when the input is documentation?

In TestAutomate it runs in four stages that never ask for a requirements document: crawl, extract, classify, emit. The crawler walks the docs site you already maintain within a bounded page budget, pacing itself politely between fetches, deliberately bounded rather than exhaustive. Extraction is where the requirements-document problem actually gets solved. An LLM reads each page and pulls out the discrete procedures it describes, keeping the doc’s own button and field wording rather than paraphrasing it. Pages that don’t describe a task get rejected outright, marketing copy, conceptual overviews without steps, API references, pricing pages, and changelog entries among them, so a page never becomes a candidate test just because it exists. Pages too thin to describe a real procedure are skipped, while very long ones get truncated with an explicit note instead of being clipped without a trace. The full filter stack that decides what becomes a test has its own write-up, and what matters here is narrower. Every one of those filters runs against a documentation page, never against a requirements doc that has to exist first. The one real run we’ve recorded backs this up. Pointing the pipeline at Linear’s published docs turned into roughly 40 skills without anyone writing a single requirements document for the features being tested, because there wasn’t one to write.

What survives extraction is stored as a skill, a description of one thing a user can do in the product. A second stage, auto-suite, reads those skills and drafts test flows from them, triggered the first time someone clicks Run suite on an app with no plan yet rather than running eagerly on every ingest. The skills get clustered into topics first, and the clustering needs no predefined categories. It reads every skill’s description and returns its own groupings. A drafting step then writes a handful of flows per topic, and the generation rules bear down hardest here, because the spec drafted once decides the correctness of every future run against that flow. One filter runs at this stage rather than at ingestion. Skills purely about logging in or signing up get dropped before generation, because the runner assumes you’re already authenticated in your own browser profile, though admin-side flows like SSO setup survive, since those configure the app rather than authenticate a user.

What comes out the other end is a runnable test with three required fields, an id, a natural-language prompt, and an expected outcome, and the generation rules explicitly forbid code, JSON, YAML fragments, or selectors inside the prompt and outcome. That plain-language contract is the same one behind every test in our agentic testing setup. An agent re-derives how to execute each step against the live page at run time, so nothing brittle from generation time gets baked into the flow. A lint pass then runs over every drafted flow before it’s kept, on the principle that a bad test is worse than a missing one, dropping flows that assume data a prior step created or demand a specific named resource nothing guarantees exists. The same rules push any resource that genuinely must already exist into an idempotent setup prompt, checked before the test’s own steps run, so a missing precondition surfaces as a blocked run rather than a false failure. Nothing in that chain, crawl through lint, ever asks for a requirements document.

What still needs a human?

Judgment about what’s worth testing, and a read-through of what got drafted. TestAutomate’s lint pass drops a flow outright rather than ship something that quietly assumes a state nothing guarantees, which is a different failure mode from the one release week taught me. My old regression passes cut coverage under time pressure, and whatever got skipped just stayed skipped, with no record anywhere of what had been left out. A dropped generated flow at least says so, in a warning that names the topic and the reason.

That doesn’t make the output finished. A generated test still deserves the same read I’d give one I wrote by hand: does the expected outcome actually pin down success, or does it just gesture at “works”? Loose ones exist, and the recorded Linear run produced one, a drafted cycle test whose outcome passed on a run that created the wrong date range until it was tightened by hand. I’d rather tighten one by hand once than trust a vague assertion on every future run. Writing expected results that hold up matters just as much for generated tests as for hand-written ones. A separate verifier model grades every run against those expectations afterward, skeptically, so the outcome you approve today decides the honesty of every verdict you read later.

What’s different from my release-week evenings isn’t that the review step disappeared. It’s that the tests in front of me for review are already runnable, drafted from documentation that describes the product as it exists today, instead of from a document I’d have had to go looking for first.

Browser Agent Testing Catches What APIs Miss

Thu, 13 Aug 2026 00:00:00 GMT

At the health-tech platform where I test today, we had a stretch where the API suite was reliably green while real users were genuinely stuck. The endpoints returned exactly what the contract promised. The page those responses fed did not hold up its end, and the thing that finally told us wasn’t a test. It was people.

That gap is what browser agent testing — the end-to-end lane of agentic testing — exists to close. An AI agent drives a real browser through the same flow your user takes, reading the rendered page and acting on what it finds, so the test can only pass if the whole stack cooperates. An API suite proves your backend keeps its promises. Browser agent testing verifies the product in front of the user, with no test scripts to write and no selectors to maintain.

Why do API tests pass while the UI is broken?

Because an API test verifies the contract and then trusts everything above it. Between a correct JSON response and a working screen sit rendering, client-side state, permissions logic, and event handlers, and every one of them can fail while the contract holds. A submit button whose handler was detached in a refactor. A form that validates but never sends. An error swallowed by a component so the user sees a spinner forever. None of these move an API assertion, and I say that with affection, because I run API tests every week and value them. They’re fast, precise, and wonderful at pinning down which layer a bug lives in. What they can’t do is stand in for the user’s path.

The failure that finally exposed our gap was exactly this shape. Data arrived correctly, the page consumed it wrongly, and every layer we measured reported success while the one layer we didn’t measure, the rendered product, failed people quietly. The decision rule I took away from those weeks is blunt. If the only signal that a flow is broken is a user saying so, that flow has no test, whatever the dashboards claim. Coverage isn’t the number of assertions you run. It’s the set of failures you’d hear about from a test before you hear about them from a person.

The testing-pyramid tradition treats end to end testing as the expensive top layer you minimize, and when end to end meant hand-maintained scripts, that was fair advice. I’ve lived that math, where every UI change came with a tax of script repair and the honest response was to keep the end-to-end layer thin. Browser agents attack the cost side of that trade rather than the value side, which is why the research community now benchmarks them on realistic multi-step web tasks like the WebVoyager evaluation instead of toy pages. Once walking a flow no longer requires owning a script, the pyramid’s economics stop forcing the layer users actually touch to go nearly untested.

What does browser agent testing actually verify?

The end state a user would see, reached the way a user would reach it. In TestAutomate a test is three plain-language fields, an id, a prompt describing what to do, and an expected outcome describing what must be true afterward, and the runner refuses to load a test that’s missing its prompt or its expected outcome. No scripting layer hides underneath, which makes this web app testing without writing test scripts in the most literal sense. The artifact you author is a paragraph, not a program.

The expected outcome carries the verification weight, and its shape is where the discipline lives. It reads like acceptance criteria, a bulleted list of end-state assertions including negative ones, such as nothing else being modified. When TestAutomate drafts tests from your product docs, the generation rules require plain natural language, forbid code, JSON, YAML, and selectors outright, and reject vague outcomes. A page merely loading doesn’t qualify as an assertion. A named table rendering with at least one row does. A concrete example makes the shape obvious. For an invite flow, the prompt might say to invite a new member with the viewer role from team settings, then open the members list, and the expected outcome would carry bullets like these.

The members list shows the invited address with a pending status
The invitation carries the viewer role, not an elevated one
[incidental] A confirmation toast appears after sending
No existing members were modified or removed

That prefix on the third bullet is a deliberate escape valve. An [incidental] bullet marks a nice-to-have that can never fail the run, so a reworded toast doesn’t burn anyone’s afternoon while the load-bearing assertions stay strict. The habit I’d push for writing outcomes is to describe the evidence a user could point at to prove the task worked, the same bar the guide to writing expected results that hold up sets in more detail.

Execution is wider than clicking, too. The agent can navigate, click and type, fill forms, read the page, and watch console messages and network requests, so a run can notice a JavaScript error or a failing request as it happens rather than inferring it from the final screen. And because the test is written without selectors, the UI refactors that used to snap end-to-end scripts don’t threaten the artifact. What breaks a run is the flow genuinely not working, which is the only thing you ever wanted a red result to mean.

When my API-green incident happened, the fix for our coverage wasn’t more endpoint assertions, it was a check that walks the flow. That’s exactly the shape of what an agent runs.

How does a browser agent read the page?

Structurally first, visually only when structure isn’t the question. The agent reads the page’s elements as data and finds what it needs by description rather than by authored CSS or XPath, and after each action it observes the page again, so it always acts on the page as it now is, not on a memory of the page as it was. Screenshots exist in the toolbox but their place is deliberately narrow, reserved for judging visual state such as layout, since reading text or values straight off the page is cheaper and more reliable than squinting at pixels.

The practical consequence is the one that matters to a tester. Pages built with proper names and roles on their controls are easy for the agent to operate for the same reason they’re easy for assistive technology to operate, and that’s why I stopped calling these tools vision gimmicks. The agent isn’t guessing at pixels. It’s working from what the page itself declares, which I’ve written more about in how computer-use agents read a page. There’s a free testability rule hiding in that. If a control has an honest name and role, both your screen-reader users and your test agent can find it, and when the agent keeps struggling to locate something, the page’s declaration is usually vague before the agent is dim.

What does each test layer verify for the user?

Each layer answers a different question, and choosing between them honestly means naming that question instead of arguing about tooling. This is the comparison as I’d draw it for the layers most teams actually run, scored on the only axis users care about, which is what a green result actually proves.

Test layer	What a pass actually verifies for the user	What it takes on trust
Unit tests	One function or component behaves correctly in isolation	All the wiring between components, and the entire browser environment
API suite	The backend returns contract-correct responses to the requests you thought to send	That rendering, client state, and handlers turn those responses into a working screen
Scripted end to end	The exact scripted path still works, as long as its locators still match the page	That the selectors describe today’s UI and someone updated the assertions after each redesign
Managed QA	A human saw the real product work, with human judgment attached	The vendor’s turnaround, and coverage bounded by the cycles you pay for
Browser agent (TestAutomate)	The flow’s user-visible end state, graded from the recorded run by a separate verifier	That the expected outcome was written to describe the end state honestly

Now the caveats, because that table flatters the bottom row if you read it lazily. The lower layers aren’t lesser, they’re differently aimed. Unit and API tests run in milliseconds, pin a failure to a specific layer, and belong in every build, none of which a browser session can match. A browser agent run takes minutes and involves a model that can misread a page in ways a compiled assertion never will, which is exactly why the grading can’t be left to the agent itself, and I’ll get to that. A disciplined team can also push any layer past its default. A well-owned scripted suite with carefully chosen locators approaches agent coverage, it just pays a standing maintenance tax to stay there. The table describes what each layer verifies by default, not what determined engineering can bolt on top.

Can AI replace manual testing?

No, and browser agent testing doesn’t need that claim to justify itself. What an AI QA agent genuinely absorbs is the repetitive half of manual work, the release-day walk through flows that were fine last week and should still be fine today. Some people have started calling that agentic manual testing, a machine performing the checking pass a human used to grind through, and the name is fair as long as you notice what it leaves out. Deciding what’s worth testing, probing the odd paths no document describes, and judging whether technically-correct behavior is actually acceptable all stay human, and nothing I’ve watched agents do changes my mind on that.

The trust question is the real one, because an agent that ran a flow will happily believe it succeeded. In TestAutomate the verdict doesn’t come from the agent at all. A separate verifier model reads the recorded trajectory and grades the expected outcome bullet by bullet, itemizing which expectations were met and which are missing, and it treats any expectation without clear evidence in the trajectory as not met rather than taking the agent’s word for it. Runs land in one of four outcomes, passed, failed, skipped, or blocked, and a blocked run is an environment problem, never a bug candidate. When the app did something different that still satisfies the intent, say a renamed button or an extra confirmation step, the verifier records a flagged assumption for a human to review instead of a failure, and a flag is never allowed to excuse a genuinely unmet core expectation. A failure isn’t final on the first attempt either. The failing attempt is re-run once on a stronger model before the suite stands behind the failure, so a clumsy first try doesn’t masquerade as a product defect.

A recorded run from our QA test environment shows the skepticism working. The agent performed a participant-management flow against the wrong review cycle, one named for the fiscal year instead of the required regression cycle, and the verifier failed the run on exactly those grounds, spelling out that the expectations were tied to a cycle the agent never touched. An end-to-end check that grades itself would have called that a pass.

There’s a bonus that comes from testing at this layer, and it’s the part my past self would have wanted most. Because the agent is inside the app while it works, a genuine defect it stumbles into mid-run, an error page, a broken control, a console or network error, becomes a recorded bug candidate, logged for a human to review with the agent’s account of what happened. Nothing files anywhere automatically. Candidates wait for a person to click, a duplicate check runs before anything is created, and the agent is explicitly told not to report its own navigation mistakes or features that simply aren’t configured. So the human never left the loop. The human moved to the two positions where judgment actually pays, deciding what the suite should verify and deciding which caught defect becomes a filed bug.

My API-green weeks taught me to distrust any single layer’s testimony about the whole product. The API suite told the truth about the backend and stayed silent about the user. Browser agent testing gives that silent layer a voice, and with agentic testing maturing fast, it’s a voice you can afford on every release rather than saving it for the flows you had time to script. I still keep my API suite. I’ve just stopped letting it speak for the product on its own.

Test Data Dependencies in Test Automation

Wed, 12 Aug 2026 00:00:00 GMT

My first QA job was at a services firm, on a team that ran overnight cycles of batch-dependent test scripts. A real part of my week was planning which scripts could run in which cycle, because one script needed the records an earlier one created. When an upstream script died in the night, everything downstream of it died too, and the next morning was spent re-planning the cycle instead of testing anything.

That’s the shape of test data dependencies in test automation, and it hasn’t changed since. A dependent test doesn’t fail on its own merits. It fails because a sibling didn’t leave the world the way it expected, and one upstream break invalidates every result downstream. The durable fix isn’t smarter sequencing. It’s tests that are self-contained by construction, plus a verdict system that refuses to call a missing precondition a failure.

Those are two separate disciplines, and you need both, whatever AI test automation layer sits on top. Self-containment stops tests from failing for each other’s reasons, and honest verdicts stop the leftover environment problems from masquerading as bugs. The same lens also answers a question every team eventually asks, which is why a suite that’s green on one machine dies on another.

Why do chained tests collapse together?

Because a chain turns one failure into many, and the copies carry no information. If twenty tests depend on data from one creation flow, a bug in that flow doesn’t cost you one red, it costs you twenty-one, and twenty of them tell you nothing about the app. This is the pattern the xUnit testing literature catalogs as interacting tests under the erratic test smell, and Martin Fowler’s piece on non-determinism in tests names lack of isolation as one of its main causes.

Sequencing feels like a fix because it makes the chain reliable when everything works. But ordering only schedules the dependency, it doesn’t remove it. My batch cycles were exquisitely ordered and they still collapsed weekly, because ordering does nothing about the morning when script twelve writes half its records and script forty consumes the halves. The failure I actually spent hours on was rarely the upstream bug. It was working out which of the downstream reds were echoes.

The economics are lopsided in the worst way. The chain saves minutes at authoring time, one shared record instead of twenty setup steps, and pays it back at triage time with interest, every single time anything upstream wobbles. And triage is the expensive kind of time, because until you’ve traced the cascade you can’t even say how many distinct problems you have. There’s a quieter cost too. Correlated failures train a team to assume that a big red morning means one upstream hiccup, so the day a cascade hides a second, genuine regression among the echoes, nobody goes looking for it.

Why does a green local suite die in a fresh CI environment?

Very often because the local environment has quietly accumulated the data the tests depend on, and the CI environment starts from nothing. A workstation database carries months of records left behind by manual poking, demo prep, and earlier runs. A test that opens “the existing project” always finds one there. In a freshly provisioned CI environment there is no existing project, and the same test dies on its first step with zero change in the application between the two runs. Google’s testing blog made the case years ago for hermetic servers, test environments that own every piece of state they need, and the argument lands hardest right here. State a test merely inherited is state nobody guaranteed.

I treat “passes locally, fails in CI” as a test data dependency report until proven otherwise. The dependency just isn’t on a sibling test this time. It’s on the residue of your own past sessions, which is harder to see, because no test created it and no plan mentions it. Two cheap experiments will expose it. Run one test alone against a fresh environment and watch what it assumes into existence. Then run the whole suite in shuffled order, because a suite that only passes in one order is confessing that order was doing invisible work.

This is also where I’d aim most test data management effort, at the level of the single test rather than the dataset. Refreshing and masking shared datasets has its place, but a test that checks for the data it needs and creates it when absent is portable by construction. It reuses the lived-in state on your machine, builds from nothing in CI, and behaves identically in both. That property makes the local-versus-CI question boring, which is exactly what you want it to be.

How do you find test data dependencies in test automation?

Read the test’s steps and hunt for the phrases that reach outside it. “The record created earlier.” “The previously added user.” A named resource that no step in this test creates. Any definite article pointing at something the test didn’t make is a dependency wearing camouflage, and the order-sensitivity tell from the CI section applies here too, since a test that passes in sequence but fails solo has just named its dependency for you.

When TestAutomate generates a suite it applies this hunt mechanically, as a lint pass over every generated flow before it becomes a test. The contract behind the lint is blunt. Each test runs independently and in any order, no test may assume a resource was created in a prior step, and there is no shared state between tests. The lint rejects three shapes of violation. A flow that assumes prior-step data. A flow that hedges with “or use any existing one” while nothing guarantees one exists. And a flow whose expected outcome demands a specific named resource that neither its setup nor its steps promise to create. Offenders get dropped with the reason recorded, on the theory that a bad test is worse than a missing one. Suites are generated the first time someone hits Run suite on an app with no plan yet, and I’d rather have that lint delete a test at generation time than triage its echo failures every week after.

It helps that the artifact being linted is small enough to read. A test here is three required fields, an id, a prompt, and an expected outcome, all in plain natural language with the generation rules explicitly forbidding code, JSON, YAML, or selectors in them. In script-based suites I’ve maintained, data dependencies loved to hide in fixture files and helper functions three imports away from the test. A dependency can’t hide in a one-paragraph intent. If the prompt says “the cycle created earlier”, it’s right there for the lint, or for me, to catch.

Which approach actually guarantees the data a test needs?

Only one of the common strategies checks the guarantee at the moment it matters, which is the moment the test runs. Here’s how the four usual answers compare on where the guarantee actually lives:

Approach	Where the guarantee lives	When the data is missing	Blast radius of one gap
Shared fixtures	A dataset loaded before the suite, trusted by every test	Failures surface mid-test, indistinguishable from app bugs	Every test that reads the fixture
Chained tests	A sibling test’s side effects, moments earlier	Downstream tests fail as echoes of the upstream break	Everything downstream in the chain
Manual seeding before a run	A human or script priming the environment	Depends on whether anyone re-seeded recently	The entire run
Idempotent per-test setup (what TestAutomate generates)	The test itself, checking then creating at run time	Setup creates it, or the run reports blocked with a reason	One test

Some honest caveats belong next to that table. Shared fixtures remain the right call for large read-only reference data, where per-test creation would be absurdly slow and nothing mutates the records anyway. Chained tests are cheaper to author and genuinely mirror multi-step user journeys, so their appeal is real even though the bill arrives later. Manual seeding is fine for exploratory sessions where no recorded verdict depends on it. And idempotent per-test setup pays a small runtime toll on every run, plus it deliberately tolerates variation by reusing whatever matching record exists, so a test that needs pristine, exactly shaped data still has to create its own clearly marked records and remove them afterward rather than borrow whatever it finds.

The mechanics of that fourth row are worth spelling out, because idempotent test setup is a discipline, not just a phase name. In TestAutomate a generated test can carry a setup prompt whose whole job is to guarantee any data the test assumes already exists. Check whether the resource exists, use it if it does, create a minimal valid one if it doesn’t. Setup contains state-establishing actions only, none of the test’s own steps and no assertions. Cleanup mirrors it, removing only what the run created, idempotently, and it can never affect the verdict.

The honesty extends into the assertions. An expected outcome may only require a specific named resource if the test itself guarantees that resource exists, and a test that operates on whichever record it finds must phrase its assertions as “the record operated on” rather than a hard-coded name. There’s even a hardening pass that rewrites older generated tests into this self-contained shape without changing what they exercise or watering down their assertions. Roles get the same explicit treatment, since a signed-in identity is test data too. A test that needs an HR admin says so in a one-line note in its prompt rather than generating steps to switch accounts, which keeps identity a stated precondition instead of a hidden dependency on whoever the previous test left signed in. Writing a good expected outcome covers the assertion side in detail, and the docs-to-suite pipeline shows where these generated flows come from in the first place. None of this is exotic. It’s the discipline I tried to impose on batch scripts by hand, made a property of the artifact instead of a property of my vigilance.

When is blocked the right verdict?

When setup fails, because a test that never touched the app has produced no evidence about the app. In TestAutomate the setup phase must end by declaring its own outcome, ready or blocked, with a one-line reason. A blocked setup short-circuits the run before the test proper, and the verdict is recorded as blocked, explicitly neither a pass nor a failure, never counted among failed runs and never the source of a bug candidate. Cleanup still runs afterward, reverting anything a partial setup managed to create, so a blocked run doesn’t leave debris for the next run to trip over. Setup and cleanup even operate under their own time bounds, sized to their jobs, which lets setup fail fast and honestly so the test only ever runs against a world that’s actually ready.

A real run from our QA test environment makes it concrete. Run 36 ended blocked in 1m 3s because setup couldn’t establish the precondition, and the recorded reasoning states the run is “neither a pass nor a failure” and calls it a test-environment problem, not an app defect. That sentence is the whole philosophy. On my batch mornings, the downstream scripts that starved for data were reported as failures, and I burned hours proving they weren’t.

What happens downstream of the verdict matters as much as the label. When a test genuinely fails, the runner re-runs it once on a stronger model before standing behind the failure, and blocked runs never enter that path, because retrying a missing precondition on a smarter model fixes nothing. Bug reporting gets the same wall. Only failed runs can produce a bug candidate, every candidate waits for a human to review it and click file after a strict dedup check, and a blocked run never generates a candidate at all. Missing data can cost you one test’s verdict, but it can’t burn a retry, inflate the failure count, or plant a phantom bug in your tracker.

The quiet part of this problem is that dependency never announces itself. Nobody writes “this test is fragile” in a plan. It arrives as a helpful reuse of existing data and gets discovered months later, at cascade time, or on the first run against a clean CI database. Generated suites have one real advantage here. The discipline gets applied every time, by a lint that doesn’t get tired, which is exactly the sort of grunt work agentic testing should be spending its patience on so I don’t have to spend mine. Twenty years after those batch cycles, the best mornings are still the ones where every red in the suite is a fact about the app and nothing else.

Flaky Tests AI Can Fix and the Ones It Can't

Tue, 11 Aug 2026 00:00:00 GMT

I spent close to five years testing a VOIP telephony product, and our suite was red most mornings. Calls connected half a second late, presence indicators lagged behind a reconnect, notifications landed after an assertion had already given up. Plenty of those reds weren’t bugs at all. They were the network being the network, and it was my job to sort one from the other before standup.

So here’s my honest read on the flaky tests AI test automation can absorb and the ones it can’t. No layer of intelligence makes a nondeterministic app deterministic, and anyone promising zero flakes is selling something. What an agentic layer can do is execute tests adaptively instead of through brittle waits, judge every red on recorded evidence, retry the genuinely suspicious failures on a stronger model, and refuse to count environment problems as failures.

What causes flaky tests and how do you prevent them?

Four cause families cover nearly everything, and each one wants a different prevention. The widely cited empirical study of flaky tests in open-source projects by Luo and colleagues found the three most common causes to be async waits, concurrency, and test order dependency, and Martin Fowler’s essay on eradicating non-determinism in tests covers the same ground from the practitioner side. Here’s how I sort them when a suite starts getting noisy.

Timing and asynchrony come first because they dominate. The test asserts before the app finishes doing the thing, so the verdict depends on a race the test doesn’t know it entered. Prevention means waiting on conditions instead of clocks. A test should proceed when the confirmation appears, not after two seconds have politely elapsed.

Shared state comes second. Two tests touch the same record, or one leans on data another created, and the suite’s pass rate starts depending on execution order. The prevention is isolation, with every test owning its data and never assuming a prior step left something behind.

Environment problems are third. Missing fixtures, a misconfigured tenant, a service that isn’t up yet. These reds say nothing about your app, and preventing the damage is partly hygiene and partly reporting that refuses to count them as failures, which I’ll come back to.

External dependencies are the fourth family, and the one I know best. My VOIP years were a live demonstration. The app under test talked to carriers, push services, and other clients we didn’t control. Some mornings the reds told me about a regression. Most mornings they told me about the weather between our lab and the nearest data center, and no amount of selector hygiene was going to change that. You can stub externals in lower layers, but an end-to-end test of a real-time product eventually touches the real world, and the real world doesn’t do determinism.

Size amplifies all four. Google’s analysis of its own test corpus found flakiness climbing steadily with test binary size, which matches intuition, since a bigger test crosses more async boundaries, holds more state, and touches more of the environment per run. The end-to-end tests that matter most are structurally the flakiest kind you can write. And a real-time app concentrates every family at once, because its correct behavior is itself timing-dependent. A call that connects in 300ms today may take 900ms tomorrow over a congested link, and both are correct. The test that hard-codes the wait is the thing that’s wrong.

Why do tests pass locally but fail in CI?

Because CI is a different machine keeping different time, and most suites quietly encode their author’s machine. Your laptop has a warm cache, an idle CPU, and one test running at a time. A CI runner is a shared, throttled container where services boot cold and parallel workers fight over the same fixtures. A wait that comfortably covers a local render loses the race on a loaded runner, and a data assumption that holds when tests run alone breaks when they interleave. Nothing about the app changed between the two runs. The assumptions around it did.

My sorting rule for these has three branches. If it reproduces locally too, treat it as a candidate defect or a test-logic bug. If it appears only in CI, suspect timing and environment before you suspect the app, and go read what the environment did. If it comes and goes in both places, think concurrency or shared data.

An agentic runner shifts this picture in one specific way. A TestAutomate test carries no machine-tuned waits at all, because there’s nowhere in the format to put one. The agent observes the page after each action and proceeds when the app has actually responded, at whatever pace the app produces, so the artifact that breaks under CI timing in a scripted suite simply doesn’t exist here. Runs are still bounded, so nothing hangs forever, but a run that finishes inside its budget is judged purely on the evidence of what happened, never on how long it took. Slow is not a defect unless slowness itself violates an expectation you wrote down.

The flaky tests AI can absorb, and the ones it can’t

The absorbable kind is flakiness the test inflicted on itself, and in my experience that’s most of it. A scripted check goes red when a selector string stops matching or a fixed wait loses a race, and neither event says anything about the app. TestAutomate removes both failure modes from the test artifact. A test is three user-authored fields, an id, a prompt, and an expected outcome, and the runner refuses to load a test missing any of them. The generation rules require the prompt and expected outcome to be plain natural language, flatly banning code, JSON, YAML, and selectors from appearing in them. There’s no locator to go stale and no sleep to mistune, because there’s nowhere in the format to write one. The agent finds elements on the live page by reading it the way a person would.

The expected outcome does its own de-flaking work. It’s written as itemized end-state assertions, including negatives like “no other records were modified”, so the verdict has concrete things to check instead of a vibe. And any bullet you prefix with [incidental] is explicitly a nice-to-have. A missing success toast marked incidental can never fail the run. That gives you a dial for rigidity, where the load-bearing assertions stay strict while the cosmetic ones stop generating noise.

What no AI absorbs is nondeterminism that really lives in the app or its dependencies. A race condition that drops a message one run in fifty is invisible to any single execution, and an adaptive runner sails through the other forty-nine as honestly as a scripted one. The claim worth holding any tool to isn’t zero flakes. It’s that every remaining red gets classified correctly, which is where the verdict layer earns its keep.

How does an evidence-first verdict separate noise from signal?

By splitting the actor from the judge. In TestAutomate the model that drives the browser never grades its own work. A separate verifier model reads the recorded trajectory of the run, every action the agent took and what came back, and judges each expectation strictly on that evidence. It’s skeptical by design. If the trajectory doesn’t contain clear evidence for an expectation, that expectation is not met, however confidently the agent declared success. And the verdict itemizes, listing each expectation met and each one missing, so a red arrives with its reason attached instead of a bare X. I’ve written before about how this verifier grades runs, and for flakiness, the itemizing is the part that pays. In my VOIP days that combination would have deleted a whole category of morning triage, the reds you investigate for an hour just to learn what the failure even was.

Here’s a real one from our recorded QA dataset. The verifier failed a test because the agent had operated on the wrong review cycle, a similarly named one instead of the cycle the test required, and the verdict’s reasoning says exactly that, naming the cycle mismatch as the reason the expectations weren’t met.

The vocabulary matters as much as the strictness, and this is where the false positive vs false negative distinction in software testing does real work. A false positive is a red with no defect behind it, the kind that burns trust. A false negative is a green that let a defect through, the kind that burns customers. Tolerance applied naively trades the first for the second. The verifier’s answer is assumption flags. When the app did something different that still satisfies the intent, a renamed button or an extra confirmation step, the test passes with a flag recording the observed behavior, the expected one, and the assumption made, for a human to review. But a flag can never excuse an unmet core expectation. Cosmetic drift gets absorbed without behavior ever getting a pass it didn’t earn.

What happens when a red might not be the app’s fault?

It gets classified, with separate machinery for the two innocent explanations. This is the mechanism I appreciate most as the person who used to do this triage by hand, because it encodes exactly the question I asked about every VOIP red. Was this the app, or was this everything around the app?

The first innocent explanation is the agent itself. When the verifier judges a test failed, TestAutomate re-runs the act phase once on a stronger model and lets the second verdict stand. A pass on retry is recorded as exactly that, evidence the first failure was agent error rather than an app defect. A second failure is recorded as high confidence that the issue is real. The escalation is deliberately narrow. Only genuinely judged failures qualify, a blocked or skipped run never burns the retry, and if the verifier itself was unavailable, the runner won’t manufacture a confident double-failure out of a fallback verdict. False alarms and why they get re-run walks the full path.

The second innocent explanation is the environment. Every run ends in one of four outcomes, passed, failed, skipped, or blocked. A blocked run means setup couldn’t establish the test’s precondition, so the app was never exercised. It’s recorded as neither a pass nor a failure, never counts toward failure totals, and never becomes a bug candidate, because there’s no app behavior to report. Skipped gets the same neither-nor treatment when a test can’t meaningfully run in the mode you chose. Bug candidates themselves only ever come from judged failures, and even then a human clicks to file each one after a strict dedup check. Environment noise has no path into your bug tracker.

Should you rerun, quarantine, or triage a flaky red?

Triage always happens somewhere, so the real question is who does it and what it costs. The standard plays for a nondeterministic red are rerun-until-green, quarantine, and human triage, and most flaky test detection and quarantine tooling exists to automate the first two.

Approach	What happens to the red	What you learn about the app	Where it goes wrong
Rerun until green	The test repeats until a pass shows up, and the pass is what gets reported	Nothing, since a real intermittent defect produces the same eventual green	Races and intermittents ship, since attempt three counts as passing
Quarantine	The test leaves the blocking suite while it stays unreliable	Nothing while it sits out, and coverage quietly shrinks	Quarantine turns permanent, tests check in and never leave
Human triage	An engineer reads the logs and decides app, test, or environment	The most of any row, when it actually happens	It taxes every morning, and under deadline pressure it decays into rerun-and-hope
TestAutomate verdicts	A skeptical verifier classes the red on evidence, environment problems record as blocked, and a judged failure gets one re-run on a stronger model	Every red arrives pre-classified, with its unmet expectations named	A true app intermittent can pass the retry and be logged as agent error

Some honest caveats on that table. Rerun-until-green is defensible as a stopgap while a real fix is in flight, and quarantine done with an exit date is what Fowler’s essay linked above actually recommends, a holding area with a short fuse rather than a dumping ground. Purpose-built flaky test detection and quarantine tools also do something TestAutomate doesn’t attempt, watching pass and fail patterns across long run histories to surface statistical flakiness. The bet here is on classifying each red at the moment it happens instead. The escalation retry costs a real extra run on the failing subset, which is a price. And the last row’s failure mode is genuine. A pass on retry means the stronger attempt succeeded, not that the app is innocent, so the same test earning that label repeatedly is worth a human look.

The recorded runs in our own QA test environment, an HCM app with 25 suite runs on record, show how much of that triage the verdict split does before you arrive. Run 33 failed 2 of 2 tests in 12m 10s, a real investigation. Run 32 finished with one failed and one skipped in 2m 11s. Runs 34 through 36 each ended blocked in about a minute, 36s, 49s and 1m 3s, which says the environment needed fixing and says nothing bad about the app. On the dashboards I grew up with, all six of those runs would have been the same color.

What I’d have given for that split on my telephony mornings. Flakiness didn’t go away when I changed domains, and it won’t go away from yours. The realistic goal is a suite where every red has already been cross-examined, the environment problems wear a different color, and the agentic approach to testing spends its adaptability on your app instead of on your patience.

LLM as a Judge Software Testing in Practice

Mon, 10 Aug 2026 00:00:00 GMT

The first reaction most QA people have to this corner of AI test automation is suspicion. A model drove the browser, and now another model decides whether the run passed? I designed TestAutomate’s verification layer around that suspicion rather than against it, because the suspicion is correct. An unexamined judge is a rubber stamp with extra steps.

LLM as a judge software testing means a language model, separate from whatever executed the test, grades the run against the expected outcome and returns a verdict. It works when three properties hold. The judge never drives the run it grades, it treats unevidenced success as failure, and it must name every expectation met and missing.

Most writing about LLM judges comes from the AI-evaluation world, where models grade chatbot answers for helpfulness or tone. Grading a functional test run is a different job with sharper edges, and it deserves its own treatment. So here it is. What the judge reads, why it never grades its own driving, when a deterministic check beats it, and where reliability and bias actually get decided.

What does LLM as a judge software testing actually mean?

It means a language model plays the test oracle, the mechanism that decides whether observed behavior counts as correct. The term migrated over from AI evaluation, where the study that popularized it found a strong model judge agreeing with human preferences more than 80 percent of the time, roughly the rate at which humans agree with each other. That literature grades essays and chat answers against rubrics. In software testing the judge grades something more concrete, a recorded run of a real application, against something more falsifiable, a written expected outcome.

The test oracle problem is much older than any of this. Every automated test needs a way to separate right behavior from wrong, and for end-to-end flows through a browser, that mechanism has historically been either a brittle coded proxy or a human. A skeptical model judge is a third option, and in TestAutomate it’s the only thing standing between an agent’s browser session and a verdict you’ll be asked to act on.

The shape of the test artifact decides whether the judge has anything real to check. A TestAutomate test is three required plain-language fields, an id, a prompt saying what to do, and an expected outcome saying what must be true afterward, and the runner refuses to load a test that’s missing its prompt or its expected outcome. The expected outcome isn’t a vibe. It’s itemized assertions about the end state, including negative ones like no other records being modified, and writing those assertions so they hold up is most of the craft. The judge’s entire job is checking that checklist against what actually happened.

Framed that way, the laziest version of the idea is already ruled out, the one where a model glances at a final screenshot and emits a thumbs-up. A checklist oracle has to show its work. Which items were satisfied, which weren’t, on what evidence. Everything else in the design follows from demanding exactly that.

Why shouldn’t the agent that ran the test grade its own run?

Because the agent’s account of its own run is a claim, not evidence. The same evaluation literature that validated model judges also catalogued their biases, and the one with real teeth here is self-enhancement bias, a model rating its own output more generously than a neutral party would. An agent that has just fought its way through a long flow is the least neutral party available. Its final summary reflects what it set out to accomplish, and intent is precisely what a verdict must not be built from.

So the split in TestAutomate is architectural rather than stylistic. One model drives the browser. A separate verifier model, which never participated in the run, makes one judgment per attempt. It receives the test’s expected outcome and the recorded trajectory of the run, meaning every action the agent took, every result that came back, and the agent’s closing summary presented as the agent’s own account. Its standing instruction is to judge only on evidence in that trajectory. If nothing in the record clearly shows an expectation was satisfied, the expectation is not met, however confidently the summary claimed success. AI test verification that takes the executor’s word for it isn’t verification. It’s transcription.

Two quieter details keep the separation honest. The judge grades one attempt’s trajectory, never a blend. When a failed first attempt gets retried once on a stronger model, the retry receives its own fresh judgment of its own trajectory, and the stronger attempt’s verdict becomes the authoritative one. Nothing is averaged across attempts. And the judge is told what the setup phase established before the test began, so it credits preconditions instead of demanding the run re-prove them. It grades the test’s own actions on top of that guaranteed state, which keeps verdicts about the flow under test instead of the scaffolding around it.

The blunt version is that we assumed the agent would sometimes be wrong about itself, then built the grading so that being wrong about itself can’t survive contact with the record.

When do deterministic checks beat an LLM judge?

Whenever the property you’re checking is machine-decidable and the interface it lives behind holds still. That isn’t a grudging concession. A coded assertion on a stable API returns the same verdict for the same input every time, costs nearly nothing to run, and fails in a way a stack trace can explain. Determinism is a real virtue in a test suite, and the classic case for eradicating non-determinism in tests applies with full force to any probabilistic judge, ours included. Run a model twice on a genuinely borderline trajectory and you can get two different verdicts. Anyone presenting LLM as a judge vs deterministic checks as a rout in either direction is skipping the interesting part.

The interesting part is what the check is actually checking. Unit and API layers state their expectations in machine terms, and there a deterministic assertion is strictly better. End-to-end flows state their expectations in human terms. The new cycle appears with the specified date range. No other participants were modified. The true oracle for sentences like those is judgment, and a deterministic proxy for judgment, a selector plus an equality check, encodes it badly and then silently drifts away from it as the UI changes. You don’t escape judgment by coding it. You freeze one day’s version of it and hope.

Here’s who grades a run under each approach, and on what evidence.

Who grades the run	Evidence it reads	Strongest at	Characteristic failure
A coded assertion in a scripted test	Values a locator fetched, compared against expected constants	Exact, machine-checkable properties behind stable interfaces	Blind to anything unasserted, and a dead locator ends the run before evidence exists
A screenshot diff in a record/replay tool	Pixels against a stored baseline image	Catching unintended visual change precisely	Reads intended change and regression identically, so every redesign forces re-baselining
A human reviewer in a managed QA service	Their own walkthrough of the application	Judgment, equivalence, and intent	Turnaround measured in cycles, and consistency that varies with the reviewer
TestAutomate’s separate verifier model	The full recorded trajectory, graded against itemized expectations	Natural-language outcomes of open-ended runs, with every item met or missing named	Probabilistic by nature, so it has to be deliberately engineered against leniency and bias

Honest caveats belong under that table. Keep deterministic layers deterministic, because nothing about a model judge belongs anywhere a coded assertion already decides the question cheaply. Screenshot diffing genuinely beats a trajectory reader at catching a subtle layout regression, which a judge reading actions and results will sail straight past. The human reviewer remains the standard every other row approximates. The verifier’s row earns its place in one specific territory, open-ended flows whose outcomes are written in natural language and whose interfaces refuse to hold still, and that territory happens to be where end-to-end suites spend most of their lives.

How do you handle LLM as a judge reliability and bias?

By engineering against named failure modes instead of trusting a capable model to be fair. The bias catalogue from the evaluation literature maps onto specific countermeasures in how our verifier is framed, and the mapping is more instructive than any reassurance.

Self-enhancement bias is handled structurally. The judge never grades its own work because it never does the work, which is the previous section in one sentence. Position bias, the tendency to favor whichever answer comes first, mostly afflicts pairwise comparison, and grading a single run against a checklist avoids that framing by construction. Verbosity bias, the tendency to reward long confident prose, would be the deadliest one here, because an agent’s closing summary is exactly that. The countermeasure is the evidence rule. Claims earn nothing. Only recorded actions and their results count toward an expectation, so a fluent summary with no supporting trajectory reads, to this judge, as a failure.

Leniency is the bias with the biggest cost in QA, the drift toward giving a plausible run the benefit of the doubt. Our verifier carries an explicit asymmetry instead. Claiming a pass when a core behavior broke is treated as much worse than a false alarm, and when in doubt about a core expectation, the judge marks it missing. On its own that rule would shred every run where the UI drifted harmlessly, so two pressure valves keep strictness from curdling into noise. An expectation the author prefixes with [incidental] is declared leniency, a nice-to-have whose absence can never fail the run. And changed-but-equivalent behavior, a renamed button, an extra confirmation dialog, becomes an assumption flag on a passing verdict rather than a failure, recording what was observed, what was expected, and why the change looks intended. The boundary holds in both directions. A flagged pass is still a pass, and a flag may never excuse a genuinely unmet core expectation.

Reliability also means behaving well at the edges. A very long run is condensed to what the judge can reliably read before grading, because a judge drowning in context produces garbage verdicts. And when the verifier itself can’t be reached, the system refuses to improvise. The run is recorded as failed with a verdict stating that the judge was unavailable and the trajectory needs human review, and that fallback never triggers the stronger-model retry, because escalating a run no judge actually graded would manufacture confidence out of an outage.

What should a verdict give you beyond pass or fail?

Named evidence you can act on without re-watching the run. Every verdict our verifier produces states whether the run passed, a reasoning summary behind the call, every expectation met, every expectation missing, any assumption flags, and any extras, meaning actions the agent took that nobody asked for, which is its own quiet category of signal. The dashboard surfaces the reasoning summary right on the test row, and the full run report itemizes the met and missing checklists.

That’s a real failed verdict from our QA environment. The test required the agent to work in a review cycle named 360-QA-Regression-2026. The verifier’s reasoning states that the agent operated on the wrong cycle, 360-Review-Fiscal-Year-2026, so the expectations tied to the required cycle were not met. Notice what that sentence does for whoever reads it next. A bare red X says investigate everything. A named unmet expectation says look at cycle selection first, and it turns the follow-up question, agent error or genuine app problem, into a reading exercise instead of a re-run.

The four-outcome vocabulary keeps those verdicts unpolluted. A run ends passed, failed, skipped, or blocked, and a blocked run never reaches the judge at all, because a precondition that couldn’t be established says nothing about the application. Every verdict the verifier issues therefore describes a run where the app was genuinely exercised, which is what makes the failure list worth triaging at all.

Downstream, the same structure feeds the bug workflow. Evidence from a failed run can become a bug candidate, checked against existing issues by a deliberately strict duplicate judge, and nothing is ever filed until a human clicks to confirm, with the verifier’s reasoning and unmet expectations carried into what does get filed. The judge grades runs. People decide what enters the tracker.

Where this leaves the human is reading verdicts rather than re-watching runs, which is the productivity claim of the whole agentic testing approach, and it only holds if the verdicts deserve the trust. None of what earns that trust is exotic. Separation, an evidence rule, an asymmetry, a checklist the judge must fill in item by item. We wrote the skepticism down where the model can’t ignore it, and that, more than any cleverness, is what makes an LLM judge safe to build a testing practice on.

Assumption Flags: UI Changed, Nothing Broke

Sun, 09 Aug 2026 00:00:00 GMT

Every UI suite eventually meets the same awkward run: the app changed, the test’s wording didn’t, and the flow still did exactly what it was supposed to do. Assumption flags are how TestAutomate’s verifier reports that run honestly. Instead of failing on a harmless difference or waving it through in silence, the verdict passes and carries a note saying what changed, what the test expected, and why the judge believed the change was deliberate.

An assumption flag is an annotation attached to a passing test run when the application did something different from what the expected outcome implied, but the difference still satisfied the test’s core intent. The flag records three things: what the app actually did, what the test’s wording expected, and the assumption the verifier made in letting it pass. A human reviews the flag later. The verdict itself stays green.

One clarification before anything else, because search results muddy it badly: assumption flags have nothing to do with feature flags. A feature flag is a switch that turns product behavior on or off. An assumption flag is a verdict annotation, and as far as I can tell almost nobody ships one, which is why I wrote the rule into our verifier instead of leaving the gap to a reviewer’s memory.

What do assumption flags actually record?

Three fields, and the third is the one I care about most. Every flag holds what the app actually did, what the test’s expected outcome implied, and why the verifier believes the difference was probably intended. On the record those fields are literally named observed, expected, and assumption. The first two establish the gap. The third is the judgment call, written down where a reviewer can accept it or reject it.

A worked example makes the shape obvious. Your test’s expected outcome says the user clicks Save and the record shows as saved. A release renames the button to Save as and adds a small confirmation dialog. The agent runs the flow, handles the dialog, and the record ends up saved. Nothing in the core intent failed, but two details of the app’s surface no longer match the test’s wording. So the verifier writes a flag instead of a failure. Observed, the button now reads Save as and a confirmation dialog appeared. Expected, a Save button and no dialog. Assumption, the rename and the dialog look like deliberate design changes because the save still completed. The run is green, and the note rides on it.

Notice how often a flag turns out to be a claim about what a control is now called. That’s no accident. The agent finds elements the way a person would, by reading what the page says they are, so the accessible name a page declares on its controls is exactly the layer where renames surface. A well-named control that changes its label produces a crisp, reviewable flag. An anonymous div that changes produces nothing readable for anyone.

On the dashboard, a flagged pass stays visually a pass. The test’s row keeps its green verdict and gains a small flag marker, the expanded view lists each assumption in full, and suite results count how many assumptions a passing run carried, tagged as worth a look. I sized that treatment deliberately. A flag shouldn’t shout like a failure, because it isn’t one, but it should be impossible to open a flagged run without seeing what the judge assumed on your behalf.

Why does a changed UI still produce a pass?

Because the run met every core expectation the test actually asked for, and a verdict should report on outcomes, not vocabulary. The sharper question is why the difference gets recorded at all rather than ignored, and the answer is that both easy defaults are corrosive. Fail the run and you’ve created a red result that means nothing, the classic false positive that trains a team to stop trusting red. Pass it silently and you’ve discarded the only interesting fact the run produced, which is that the app’s surface has drifted from what the test’s author believed. Somebody should confirm that drift was on purpose. The flag keeps the pass and keeps the fact.

The rule that makes this safe is the one we enforce hardest on the verifier. A flag applies only to a difference that still satisfies the expectation. If a core expectation genuinely went unmet, the action failed, the wrong thing happened, the data is wrong or absent, then the run is a failure with a missing expectation, and passing a broken run on the strength of a flag is the failure mode we designed the verifier hardest against. Flags annotate success. They can’t manufacture it. That boundary is what separates an assumption flag from the quiet leniency that ruins suites, where each accommodation makes a test a little harder to fail until it can’t fail at all.

It also helps to place flags next to the one other softening mechanism in the system, because the two point in opposite directions. An expectation the test’s author prefixes with [incidental] is declared leniency. You decide in advance that a nice-to-have can never fail the run. A flag is discovered leniency. The judge finds an unexpected difference at verdict time and documents the assumption it made. One is your call, made early. The other is the model’s call, made late, and surfaced precisely because it was the model’s call. The broader case for letting a model judge test outcomes rests on the same skepticism. Our verifier treats unevidenced success as failure, so the flag is the one narrow lane where it may accept a difference at all, and the lane comes with a paper trail.

One consequence is worth knowing before you see it in a run list. A flagged pass never triggers the retry machinery. When a run genuinely fails, TestAutomate re-runs the flow once on a stronger model before letting the failure stand. A flagged pass isn’t a failure, so it burns no retry and delays nothing. The extra scrutiny a flag requests is human and asynchronous, not computational and blocking.

How does each approach report an intentional UI change?

Mostly as breakage, which is odd once you notice what actually happened: the product changed on purpose and kept working. A rename that ships in a planned redesign isn’t a defect, yet most reporting pipelines render it as one. The comparison that matters is what lands in your report when the UI changes intentionally and the flow still works.

Approach	What the intentional change becomes	What the report shows
Scripted, selector-based automation	The stored locator no longer matches the renamed control	A red failure, element not found, indistinguishable from a real defect
Record/replay tools	Playback diverges from the recorded steps	A broken recording that asks to be captured again
Managed QA service	A tester notices the difference during a manual pass	An accurate human note, delivered on the service’s next cycle
TestAutomate	The verifier records the difference as an assumption	A flagged pass with observed, expected, and assumption on the record

Honest caveats, because that table flatters us if read uncritically. A scripted suite written against roles and visible labels rather than brittle CSS will survive many renames outright, and disciplined teams build exactly that. The catch is that surviving and reporting are different things, since the run that survives says nothing about the change at all. Managed QA humans are genuinely the best judges of intent on this list, and their eventual note is often better written than any machine’s. What you trade there is turnaround time and per-cycle cost. And our own row leans on a model’s judgment that the intent was still satisfied, which is exactly why the assumption is written down for review instead of trusted silently. A judge that can be wrong and shows its reasoning beats a judge that can be wrong and doesn’t.

What should you do when a run comes back flagged?

Answer one question: was the change intended? Everything else follows from the answer, and the flag exists to make the question cheap to ask.

If the change was intended, update the test’s wording so the next run doesn’t need the assumption. The flag’s expected field tells you which phrase went stale, so this is editing a sentence, not repairing a script. Change the expected outcome from a Save button to Save as, mention the confirmation dialog, done. It’s the same tightening discipline that writing good expected results demands in general. Expectations should describe the app you have, precisely, so that deviations mean something. Leave the wording stale and the verifier will keep making the same assumption on every run, which works, but every repeat is the judge guessing where your test should be stating.

This is also where most test maintenance after a redesign actually lands. When tests are plain-language descriptions of intent rather than selector scripts, a redesign doesn’t hand you a pile of broken locators to fix. It hands you a set of flags telling you which descriptions drifted, each with the new reality already transcribed in the observed field. The maintenance is reading and rewording, and I designed the flag record to double as that worklist.

If the change was not intended, the flag is one of the earliest honest signals you’ll get, precisely because nothing failed. An unplanned rename, a dialog somebody’s change introduced by accident, a control that moved when a layout refactor went further than intended. None of these break a flow, so a regression process keyed entirely to failures stays silent about all of them. The flag describes the drift in words a developer can act on directly. What happens next stays with you. If it deserves a ticket, bug candidates in TestAutomate are only ever filed by a human clicking to confirm, deduplicated against existing issues first, and nothing about a flag changes that rule.

Recurrence tells you which of the two readings you’re in. The same flag appearing on one test across several runs is almost always stale wording, and the fix is the one-sentence edit above. Similar flags appearing across many unrelated tests at once suggest a release shipped surface changes nobody catalogued, and that’s a conversation with whoever shipped it, with the observed fields as your inventory. Either way, don’t let flags pile up unread. Each unreviewed flag is a question the verifier asked and nobody answered, and the whole value of the mechanism is in the answers.

How are assumption flags different from self-healing tests?

They move the decision in opposite directions. A self-healing tool detects that a step no longer matches the app, patches the step, and keeps the run green. The patch is the tool’s decision, reported after the fact if at all, and the failure mode is well known. The UI changed but tests still pass, and the record of what changed lives in a healing log nobody reads, if it lives anywhere. When the change was an accident, the suite has just absorbed a regression on your behalf. There’s a fuller comparison in self-healing versus selectorless testing, but the short version is that healing edits the test and tells you second, while a flag edits nothing and tells you first.

Flags don’t need healing underneath them, because there’s nothing to heal. The agent reads the live page and finds elements like a person would, so a renamed button doesn’t break a locator. There isn’t one. The run proceeds against the app as it now is, and the flag captures the difference at verdict time rather than papering over it at runtime.

The flaky comparison fails for a different reason. A flaky test returns different verdicts against an unchanged app, which is why Martin Fowler treats non-determinism in tests as something to eradicate rather than manage. A flagged pass has a determinate cause. The app changed and the wording didn’t, so re-running it yields the same pass with the same flag until someone updates the test or the app. Nothing about it is noise. It’s the opposite, a signal precise enough to name the sentence you should edit.

Two boundaries complete the picture, so nobody buys more than we built. There is no watcher diffing your UI between runs. A flag is a judgment the verifier makes per run, at verdict time, from the trajectory and the expected outcome, so a test that never runs will never flag anything. And nothing rewrites tests in response to a flag. The review, the decision, and the edit are deliberately human work, because an assumption a model makes and a model then confirms is just the silent healing we set out to avoid.

The design bet underneath all of this is small and specific. When an automated judge makes an assumption, that assumption should appear on the record next to the verdict it shaped, in terms a person can accept or reject. That’s the entire feature. A flagged pass is still a pass, a flag can never excuse a genuine failure, and every flag ends its life the same way, with a human deciding whether the app or the test is the thing that should change. Suites earn trust by being caught reasoning honestly. Writing the assumptions down was the cheapest honesty we could ship.

Why Automated Tests Get Blocked, Not Failed

Sat, 08 Aug 2026 00:00:00 GMT

I’ll state the design opinion up front, because TestAutomate’s whole runner is built around it. Ask why automated tests get blocked instead of failed and you’re really asking what a failure is allowed to mean. A test gets blocked when its preconditions can’t be established: the required record didn’t exist, a needed control was missing, or the app errored before the flow under test ever started. A failure claims the application did something wrong. A blocked run claims nothing about the application, because the application was never exercised. Collapsing those two outcomes into one red X is how teams end up drowning in false positives in automated testing, and it was the single reporting decision I was least willing to compromise on when we built the runner.

That sounds like a semantic quibble until you’ve watched it play out in a real suite. So let me walk through what blocked actually means, the setup gate that enforces it, and the decision rules I’d use on a blocked run in any AI test automation stack, whether or not you ever use our product.

What does a blocked test actually mean?

It means the run couldn’t reach the starting state the test depends on, so no verdict about the application is possible. This isn’t our invention. The ISTQB glossary has long defined a blocked test case as one that cannot run because its preconditions aren’t fulfilled. What’s changed is that most automation frameworks quietly dropped the distinction: a missing precondition surfaces as an element-not-found exception, the exception marks the test red, and the report can’t tell you whether red means “the app broke” or “the fixture record was missing.”

TestAutomate keeps the distinction as a first-class verdict. Every run ends in one of four outcomes: passed, failed, skipped, or blocked, and the dashboard renders them as separate counts rather than mushing the last three into failure. Skipped and blocked are worth separating too, since they answer different questions. A skipped run means the test couldn’t meaningfully run in the mode you chose, the way a login-flow test can’t demonstrate logging in when the suite runs in already-authenticated mode, and its record says so in plain words along with what to change. A blocked run means the test should have been able to run and the environment wouldn’t let it. Skipped asks you to reconsider configuration. Blocked asks you to go fix something.

That’s a real capture from our QA test environment. The three most recent runs on that timeline each ended blocked, one blocked outcome out of one test, in runs lasting 36 seconds, 49 seconds, and 1 minute 3 seconds. The run before them failed 2 of 2 in 12 minutes 10 seconds. Those are different situations demanding different responses, and the report says so at a glance: the blocked runs finished fast because the runner stopped the moment the precondition proved unreachable, instead of grinding through a doomed flow and manufacturing a red result at the end.

Why automated tests get blocked: the setup gate

Blocked isn’t a judgment call the runner makes after the fact. It’s enforced by structure. A test in TestAutomate can carry an optional setup prompt alongside its three required fields, and when it does, the run starts with a dedicated setup phase whose only job is to establish the preconditions. Setup is required to be idempotent: check first whether the required state already exists, change nothing if it does, and create exactly what’s needed with minimal valid settings if it doesn’t. It’s explicitly forbidden from performing the test’s own steps, and it’s told not to loop or brute-force when something genuinely can’t be established.

Setup has to end by declaring one of two things: ready, with a one-line description of the state that now exists, or blocked, with the reason the precondition couldn’t be established. That declaration is the gate. If it doesn’t come back ready, the run is recorded as blocked right there, the test’s own steps never execute, and the record spells out that the application was not exercised, so the outcome is neither a pass nor a failure. Cleanup still runs afterward, reverting anything a partial setup created, because the other half of environment hygiene is not leaving debris for the next run to trip over. Cleanup follows the same idempotency rule and can never affect the verdict.

This is the classic test fixture discipline from the xUnit lineage, rebuilt for an agent that operates a real browser. The decision rule for what goes where is worth keeping even if your stack is entirely different. Setup establishes state and asserts nothing. The test acts and asserts, building on the state setup guaranteed instead of re-creating it. Cleanup removes only what the run created. Any step that violates one of those boundaries is in the wrong phase, and misplaced steps are where phantom failures breed.

Here’s what the enforcement looks like when it fires, from the same recorded dataset:

The reasoning shown there isn’t boilerplate. It names what setup couldn’t establish and states outright that this indicates a test-environment problem, not an app defect. Whoever picks up that run knows exactly which direction to start walking.

Why not just let missing preconditions fail the test?

Because a suite’s failures are only useful if every one of them means the same thing. The moment “failed” sometimes means the app broke and sometimes means the fixture data was missing, every red run needs a human to re-derive which kind it was before anyone can act on it. That triage tax is the quiet killer in most automation programs. Google reported that almost 16% of its tests showed some level of flakiness, and environment-dependent preconditions are one of the classic sources: the same test passes when the data happens to exist and fails when it doesn’t, with zero change in the application between the two runs.

Naming matters here. A run that goes red without a real app defect is a false positive, and a false positive doesn’t just waste the minutes someone spends investigating it. It erodes the suite’s authority, run by run, until people stop looking at red at all, and that’s the day a genuine regression sails through unexamined. The inverse error, a false negative, is a test that stays green past a real defect. Treating blocked as its own outcome attacks the false-positive side without inviting the false-negative side, because nothing about it makes a test more lenient. The application’s expectations are exactly as strict as before. The only thing that changed is that a run which never tested the application stops pretending it did.

Generated suites raise the stakes further. When TestAutomate drafts tests from your product docs, the generation rules push any “this resource must already exist” requirement into an idempotent setup prompt rather than letting it hide inside the test, and a lint pass drops drafted flows that silently assume data a prior step created. I’ve seen what those unexamined data assumptions do to a suite, and so has anyone who’s watched chained tests collapse together when the upstream link broke. A generated test with a hidden precondition is a false alarm on a timer. The lint exists because a bad test is worse than a missing one.

How do other testing approaches report a missing precondition?

Mostly as something indistinguishable from failure, which is the honest heart of the comparison. The table below is about the structural default in each approach, not about what a diligent team can bolt on top.

Approach	What a missing precondition becomes	What the report shows	Who untangles it
Scripted, selector-based automation	An element-not-found or timeout exception mid-script	A red failure, identical to an app defect	An engineer reading stack traces per run
Record/replay tools	Playback halts where the recording no longer matches the app’s state	A failed or broken recording, often prompting a re-record	Whoever owns the recording
Managed QA service	A human tester notices the environment problem and writes it up	An accurate note, on the service’s timeline	The vendor’s tester, then your team on receipt
TestAutomate	A blocked verdict with the setup phase’s stated reason	Its own amber outcome, separate from failures	Whoever reads the one-line reason

Two honest caveats on that table. Scripted frameworks can absolutely be engineered toward the same distinction, with guard steps and custom result states, and disciplined teams do it. The difference is that it’s opt-in engineering effort per test, where a blocked verdict here is the default behavior of every test with a setup prompt. And managed QA genuinely does distinguish environment problems from defects, because humans are good at that. What you’re paying there is the turnaround time and the per-cycle cost of humans doing triage that structure could have done upfront.

What should you do with a blocked run?

Read the reason, fix the environment, rerun. In that order, and without touching the test. The blocked record carries the setup phase’s one-line diagnosis, and in the runs I’ve watched it’s usually one of three things: a permissions problem for the account the agent operates as, a missing integration or configuration on the tenant, or required data that couldn’t be created. Each of those is fixable where it lives, and none of them is fixable by editing the test.

The tempting failure mode is loosening the test so it stops going amber, making it create its own data ad hoc or skip the precondition check entirely. Resist that. A test that manufactures whatever state it needs mid-flow isn’t testing your application’s behavior on realistic data anymore, and it will drift into asserting things about the mess it just created. Keep the precondition explicit in setup, where it’s checked, reported on, and reverted.

Recurrence is its own signal. A test that blocks once had an environment hiccup. A test that blocks three runs straight has a standing gap between what the suite assumes and what the environment provides, and that gap belongs to whoever owns the test data or the tenant configuration, as a tracked piece of work rather than a rerun button pressed on faith. The one-line reasons make that conversation concrete, since three identical reasons in a row read very differently from three different ones.

One nuance deserves its own paragraph, because it’s the case people miss. If setup was blocked because the application itself errored, a page that wouldn’t load, a creation form that threw, that blocked run is real signal about your app. It arrives labeled as an environment claim rather than as a false assertion that the specific feature under test regressed, and that labeling is correct: what you know is that the app broke before the flow started, not that the flow is broken. Investigate it as its own incident. The channel separation is doing its job precisely by not letting that incident masquerade as a scripted test failure with the wrong name on it.

While you’re fixing the environment, it’s also worth a minute on the test’s expectations themselves, since a precondition that keeps proving fragile sometimes means the expected outcome was written against an assumed state nobody guaranteed. I keep a separate checklist for how to write expected results for automated tests that holds up under exactly this pressure.

How does blocked interact with retries and bug reports?

It’s excluded from both, and both exclusions are deliberate. When a test genuinely fails, TestAutomate re-runs it once on a stronger model before standing behind the failure, because some first-attempt failures are agent error rather than app defects. Blocked runs never enter that path. Re-running a missing precondition on a smarter model fixes nothing, so spending the retry there would be pure cost. The retry budget is reserved for runs where a second attempt could actually change the conclusion.

Bug reporting gets the same wall. A failed run’s evidence can become a bug candidate that waits for human review, deduplicated against existing issues, and nothing is ever filed without a person clicking to confirm. A blocked run never produces a candidate at all, because there’s no application behavior to report. That single rule keeps the bug queue’s signal clean: every candidate in it traces back to a run where the application was actually exercised and actually misbehaved. When a candidate does get filed, the verifier’s reasoning and its list of unmet expectations ride along as the reproduction context, which is only trustworthy because environment noise was filtered out two gates earlier.

Stack those exclusions up and the outcome taxonomy stops being cosmetic. Blocked runs can’t dilute your failure count, can’t burn retries, and can’t pollute the bug queue. Failures, in turn, stay expensive in exactly the right way: each one has survived a precondition gate, a skeptical verdict, and a retry on a stronger model before it asks for your attention.

The pass rate you report at the end of a release is a promise about what the suite actually verified. Every blocked run that gets miscounted as a failure waters down that promise, and every one that gets miscounted as a pass inflates it. Keeping blocked as its own honest column is how the other two columns stay worth reading. That’s the whole design, and it’s the part of our runner I’d defend last if I had to give everything else up.

Computer Use Agents for QA, Explained

Fri, 07 Aug 2026 00:00:00 GMT

Every demo of computer use agents for QA tells the same story. An AI takes over a browser, glides through a signup flow, and the caption implies the testing profession is over. I built TestAutomate, whose computer use agents run intent-based QA checks in a real browser, and I still don’t buy that story. The clicking was never the hard part. What decides whether you can trust a green result is everything wrapped around the clicking.

A computer use agent is an AI model that operates software through the same interface a person uses. It looks at the current state of the screen or page, decides on one action, performs it, and looks again. Applied to QA, that loop is the engine of agentic testing: it lets you state a test’s intent and have an agent carry it out in a real browser, with a separate judge grading what actually happened.

That definition hides two design questions that matter more than any demo. First, how much autonomy should the agent get, because the research record on open-ended computer use is humbling. Second, who checks the agent’s work, because an agent grading its own run is a conflict of interest wearing a lab coat. I want to work through both, along with the comparison I find most clarifying, which is how each kind of check sees your app in the first place.

What is a computer use agent?

It’s an AI model handed the controls of ordinary software, pursuing a goal through the same buttons, forms, and pages you’d use yourself. A conventional script talks to an application through code. An agent operates the application. Ask one to submit an expense report and it opens the app, finds the form a person would find, fills it the way a person would fill it, and copes with the confirmation dialog nobody warned it about.

How do computer use agents work, then? The general recipe is a loop. The model is shown the current state of the interface, chooses a single action toward its goal, takes it, and is shown the new state. Repeat until done or stuck. Everything else, and there is a lot of else, is scaffolding around that loop, deciding what the model gets shown, which actions it may take, and when to stop it.

The research record says the unscaffolded version is genuinely hard. When the WebArena benchmark put language-model agents to work on realistic, open-ended web tasks, the best agent its authors evaluated finished 14.41% of tasks end to end, against 78.24% for humans. A year later, OSWorld measured the best model it tested at a 12.24% success rate on open-ended desktop tasks that humans completed at over 72.36%. Agents have improved a lot since both papers were published, but the gap they measured is the honest starting point for anyone proposing to let one roam free across your product.

Those numbers are also why I think QA is one of the few places where computer use agents already earn their keep. A regression test is the opposite of an open-ended task. The starting point is known. The flow is described. The finish line is a stated expected outcome. Scoping the agent that hard is what converts a research demo into something you can run every night, and it’s the first of two design decisions the rest of this piece hangs on. The second is that the agent never gets to mark its own homework.

How do computer use agents for QA run a test?

In TestAutomate, a run moves through four stations you can watch from the outside: describe, act, observe, verdict.

Describe is your half of the bargain. A test is three required fields, an id, a prompt saying what to do, and an expected outcome saying what must be true afterward, and the runner refuses to load a test that’s missing its prompt or its expected outcome. Optional setup and cleanup prompts, written the same plain way, establish preconditions before the flow and remove whatever the run created after it. Even the tests the product drafts from your product docs obey the format’s ban on embedded code and selectors, and their expected outcomes arrive as itemized end-state assertions, negatives included, like no other records being modified. A bullet prefixed [incidental] is declared a nice-to-have that can never fail the run, which keeps a missing success toast from outranking the behavior the test exists to verify.

It helps to see how little that format asks of you. A prompt for a billing flow might read, in full: sign in as the demo user, upgrade the workspace to the cheapest paid plan, and confirm the change from the billing page. The expected outcome is a short list of end-state assertions. The billing page shows the paid plan as active. An invoice for that plan appears in billing history. No other workspace settings were modified. And one bullet reading, the upgrade confirmation banner appears, prefixed [incidental] because a missing banner shouldn’t sink a run that provably upgraded the plan. Notice what’s absent. Nothing names an element, a URL pattern, or a wait. What to click is the agent’s problem on the day the test runs. What must be true afterward is permanently yours.

Act is the agent’s half. It drives a real browser through the flow you described, reading the live page and finding elements like a person would, so there are no selectors to write and none to maintain. I’ve covered what browser agent testing changes and why removing stored selectors matters elsewhere, so I’ll keep the short version. The agent finds what it needs fresh from the live page on every run, which means a renamed button or a reshuffled layout gets absorbed rather than repaired. The run is bounded, too. A confused agent can’t loop forever, and in my view a test that keeps pressing against that bound is really several smaller tests wearing a trench coat.

Observe means the agent works from what the app actually did, never from what the test hoped it would do. Screenshots hold exactly one job in this loop, judging visual state. Did the layout hold, did the header freeze, did the chart render. For everything else, the agent reads the page.

Verdict is where I spent the most design effort, because an agent that grades its own run will grade on a curve. A separate verifier model reads the recorded run against the expected outcome, under standing instructions to stay skeptical, judge only from evidence, and treat an unevidenced success claim as not met. It returns every expectation itemized as met or missing, and the run lands in exactly one of four outcomes: passed, failed, skipped, or blocked. Blocked is the outcome most tools get wrong. It means the test’s preconditions couldn’t be established, so the app was never exercised, and the record says so plainly instead of dressing an environment problem up as a failure. When a first attempt genuinely fails, the flow is re-run once on a stronger model before the failure is treated as real, because a first attempt can fail for agent reasons rather than app reasons. And when the app’s behavior has changed but still satisfies the intent, the verdict carries an assumption flag recording what was observed, what was expected, and why the change looks deliberate, so drift gets disclosed instead of silently absorbed.

How does each kind of check see your app?

Differently enough that “automated testing” is almost a useless phrase, and the seeing is the real dividing line between approaches. A check can only catch what it can perceive, so before comparing features, compare eyes.

How the check sees the app	What it looks at	What a green result proves	Where it’s blind
Scripted assertions on markup	Whether nodes matching stored selectors exist and hold the expected values	The markup contained what the selectors pointed at	Everything a user experiences that no assertion encoded, plus every run after a selector goes stale
Pixel-diff visual testing	One rendered frame against a stored reference image	This frame is visually identical to the reference	Everything behavioral, wrong data saved behind unchanged pixels sails through
Human eyes	The real product, with full context and taste	Someone competent watched it work	Nothing in principle, only whatever limited hours and attention never reached
An agent reading the page	The live page, finding elements like a person would, screenshots for visual judgment	A separate verifier found evidence for every core expectation	Expectations nobody wrote down, and visual polish no test asked it to judge

Honest caveats belong next to that table. Scripted assertions are cheap to run at enormous scale and belong on every commit in a way agent runs don’t, because an agent run spends model inference and wall-clock time. Pixel-diff tools catch a class of visual regression, the one-pixel layout break, that an agent won’t notice unless a test asks a visual question. Human eyes remain the gold standard for meaning and the most expensive thing on this list to schedule. And the agent’s blindness to unwritten expectations is a real cost you pay in the currency of carefully written expected outcomes. The table isn’t a ranking. It’s an argument that these checks see different layers of the same product, and that the agent’s layer, the one your user actually inhabits, is the one that historically went unchecked between releases.

The split I’d actually run follows from those blind spots. Keep fast scripted assertions on every commit, since nothing else gives you feedback in seconds. Put pixel diffs only on the few surfaces where visual precision is the product, a pricing page, a chart library, an email template. Then point agent checks at the user journeys that matter commercially, checkout, onboarding, permissions, on a nightly or pre-release cadence, where their tolerance of UI churn pays for their slower runs. If a journey’s scripted tests have been rewritten twice this year for reasons that weren’t bugs, that journey is the first candidate to move up a layer.

Can AI replace manual testing?

No, and the useful version of that answer names what it does replace, which is the re-verification. Walking checkout again before every release. Checking login again after a dependency bump. The regression pass nobody has attention left for by Friday. That work is repetitive, fully described, and judged against known expectations, which is exactly the shape of task the scoped loop above is good at. Handing it to an agent isn’t replacing manual testing so much as admitting this slice of it stopped being testing years ago and became typing.

What can’t be handed over is the judgment. Exploratory testing, in the sense practitioners mean it, is simultaneous learning, test design, and test execution, a person building a model of the product and probing where that model is most likely wrong. The industry now pitches autonomous exploratory testing, agents roaming an app unsupervised and reporting whatever they find. The benchmark numbers earlier in this piece are one reason I’m skeptical of the roaming half. The other is that exploration without judgment mostly produces noise, and noise is what kills a QA program’s credibility long before any missed bug does.

So we drew the line for TestAutomate’s AI QA agent deliberately. Suite runs execute described flows with stated outcomes rather than wandering open-endedly, and the consequential clicks stay human. When a failed run produces a bug candidate, that candidate waits, and when a person clicks to file it, it’s first checked against your existing issues for likely duplicates. Nothing reaches your tracker without a person reviewing the evidence and clicking to file it. I want the agent supplying evidence and a human supplying judgment, in that order, every time.

Where does that leave testers? Doing the parts that were always the actual job. Deciding what matters enough to test, writing expected outcomes sharp enough to fail, reading verdicts and flags with product context no agent has, and exploring the new feature before anyone writes a test for it. If you want the system view of the whole loop, generation through graded verdict, it’s mapped in the agentic testing guide.

I’ll end with the design intent, because it’s the honest summary. I didn’t build computer use agents for QA out of a belief that agents test like humans. I built them because most of what we call manual testing had already stopped being human, and because with hard scoping, a skeptical second model on every verdict, and a person on every consequential click, the boring majority of QA can finally run every night without anyone having to pretend the demo was the product.

Writing Expected Results for Automated Tests

Thu, 06 Aug 2026 00:00:00 GMT

Expected results for automated tests should describe the end state a run must leave behind, written as a short list of separately checkable assertions. Name the records that should exist and the values they should hold, state what must not have changed, and split what must hold from what’s merely nice to see. Skip that discipline and the strongest runner in the world ends up grading your product against a shrug.

That definition matters more in agentic testing than it ever did in scripted testing, because the expected result is no longer an assertion in code. It’s a piece of prose that a model reads, interprets, and judges a real browser run against, which is where AI test automation relocates the trust question. In TestAutomate every test is three required fields, an id, a prompt, and an expected outcome, and the runner refuses to load a test that’s missing any of them. I built it that way because the third field is where suite quality actually lives, and the craft of writing that field well is what the rest of this piece works through.

What do strong expected results for automated tests look like?

They read like a checklist a skeptical stranger could grade without asking you anything. Here’s the kind of weak outcome our generation rules exist to stamp out:

expected_outcome: |
  - The new cycle is created successfully
  - Everything works as expected

And here’s the same test with an outcome that can actually lose:

expected_outcome: |
  - A cycle named 'qa-regression-cycle-01' appears in the cycles list
  - Its status reads Active and its start date is the date entered
  - No other cycles were modified or deleted
  - [incidental] A confirmation toast appears after saving

The difference isn’t length. It’s that every line in the second version names an observable artifact with a checkable property. A verifier reading the recorded run can find the cycles list, look for the named row, read its status, and confirm nothing else moved. The first version gives it nothing to look for, which means the verdict floats free of the product. Testing theory has a name for the underlying difficulty, the oracle problem. Running the test was never the hard part. Deciding what “correct” looks like is, and a vague expected result is a decision you declined to make.

Our generation rules hold drafted tests to the same bar. An outcome that only says the page loaded gets rejected, while one that commits to a concrete rendered end state, say a named table showing at least one row, survives, because that’s something the run’s evidence can confirm or deny. When I review hand-written outcomes I apply the exact same test to each bullet, could a stranger check this against a recording, and rewrite any line that fails it.

How do you split core expectations from incidental ones?

By deciding, per bullet, whether its absence should fail the run. In TestAutomate every bullet in an expected outcome is a core assertion unless you prefix it with [incidental], and the semantics are strict in both directions. A missing core expectation always fails the test. An incidental miss can never fail it, no matter what else happened. The verifier applies exactly that split when it grades, and the verdict comes back itemized, every expectation met and every one missing, so you can see which bullet did the work.

We made unprefixed mean core on purpose. The dangerous default is the opposite one, where assertions are lenient unless you remember to harden them. Here, forgetting the prefix makes a test stricter, not looser, and the failure mode of that mistake is a red run you loosen after reading, not a green run that lied to you.

What earns the prefix is anything cosmetic riding along with the real outcome. A success toast appearing. The exact wording of a confirmation message. A count badge updating in the corner. Those are worth recording because their absence is sometimes an early smell, but none of them is the reason the test exists, and a suite that fails on toast wording trains people to stop reading red. The load-bearing lines, the record exists, the values are right, nothing else changed, stay unprefixed. Use the prefix sparingly. If most of your bullets are incidental, the test doesn’t know what it’s for.

This split is also what keeps a judge-graded test honest about severity. Without it, every bullet carries equal weight and the verifier has to guess which misses matter. With it, you’ve told the judge in advance where the line is, and the judge’s discretion is spent on evidence instead of on importance.

Why do negative assertions belong in the checklist?

Because an agent that can do anything on a real page can also do things you didn’t ask for, and a checklist that only describes the happy path will never notice. Our generation rules require expected outcomes to be verifiable end-state assertions including negatives, lines like “no other records were modified”. I’d give the same advice to anyone writing outcomes by hand, whatever their tooling.

The reasoning is mechanical. A verifier grades what you wrote down. If your bullets only assert that the new record exists, a run that also deleted three unrelated records satisfies every bullet and passes. The deletion isn’t hidden, it’s right there in the recorded run, but nothing in the checklist makes it load-bearing. One negative line converts that silent damage into a missing core expectation.

TestAutomate’s verdicts do carry a separate list of extras, actions the agent took that weren’t requested, so surprising behavior surfaces even when your bullets missed it. But extras are disclosure, not enforcement. The pass still hangs on your expectations alone, which is exactly why the expectations should state what must not have changed. My working rule is one negative assertion per mutation the flow performs. A test that creates something asserts no duplicates appeared. A test that edits one record asserts the neighbors kept their values. A test that deletes asserts only the target vanished. Cheap to write, and it’s the line that catches the expensive class of bug.

How do different assertion styles compare?

Each style is really a decision about where correctness lives, in pixels, in strings, in a person’s head, or in an itemized prose contract. Most best practices for writing test assertions were written for the middle two and carry over further than people expect. The table compares how each style responds to the two events that matter most, a real regression where wrong data got saved, and a cosmetic change where a toast got reworded.

Assertion style	Where you’ll meet it	Wrong data saved	Reworded toast	The recurring upkeep
Pixel-perfect snapshot comparison	Record/replay tools and visual-diff layers	Caught only if the wrong value is visible inside the captured region	Fails the run, a false alarm to triage	Re-approving baselines after every visual change
Exact-string and value matching	Scripted, selector-based frameworks	Caught where an assert targets that field, missed wherever nobody wrote one	Fails if the string was asserted, silent otherwise	Hand-editing asserts as copy and markup drift
Human judgment	Managed QA services	Caught when the tester knows the intended values	Correctly waved through	Per-cycle cost and turnaround of human passes
Itemized end-state expectations, core plus incidental	TestAutomate	Fails the core bullet that named the value	An incidental miss or an assumption flag, never a failure	Keeping the prose checklist true as features evolve

The honest caveats. Snapshot comparison is genuinely the right tool for pure rendering regressions, and nothing prose-based competes with it there. Exact-value asserts are fast, deterministic, and precise, and a well-maintained scripted suite full of them is a strong position, with the cost showing up as maintenance rather than as weak verdicts. Human judgment remains the best oracle for intent that nobody wrote down anywhere. And the itemized style’s strength is entirely downstream of the writing. A lazy checklist graded by a careful judge is still a lazy checklist, which is why this whole piece is about the writing.

How do acceptance criteria become checkable expectations?

By translation, and the translation is where most of the value gets added. Acceptance criteria for automated tests usually arrive written for humans, “the user can filter the report by date”, which is a statement about capability, not about any particular run. An expected outcome is a statement about one run. The given-when-then frame does the heavy lifting. The given belongs in a setup phase that guarantees preconditions, the when belongs in the prompt, and the then becomes your bullets, pinned to the concrete values this run should produce. “Can filter by date” becomes “after filtering, the table shows only rows dated within the range entered, and the row count shown matches the rows listed”.

Two consistency rules from our generation design are worth stealing for hand-written tests. First, the expected outcome must be satisfiable by exactly what the setup and prompt do, no more and no less. Second, never require a specific named resource unless something in the test guarantees that exact resource exists. The named-resource trap is the subtle one. An outcome demanding ‘regression-cycle-001’ when the prompt only conditionally creates it produces a test that fails for environment reasons, and environment problems should never wear a failure’s clothes. When a test does need pre-existing data, the setup phase guarantees it, and the verifier is explicitly told what state setup established so it never penalizes the run for a resource it didn’t create.

Generated suites get these rules enforced rather than suggested. When TestAutomate drafts a regression suite from your docs, a lint pass drops flows that assume data from a prior step, hedge with a fallback no setup backs up, or demand a named resource nothing guarantees, on the principle that a bad test is worse than a missing one. Hidden data dependencies between tests are the same disease in chained form. Whether a human or a model wrote the outcome, the acceptance question is identical. Does anything in this test guarantee the state this sentence assumes?

What happens to your expected results at verdict time?

They become the entire contract. Our runner hands a separate verifier model two things, your expected outcome and the trajectory of everything the agent actually did, and instructs it to judge only on evidence. The instruction is deliberately skeptical. If the trajectory doesn’t contain clear evidence for an expectation, that expectation is not met, and the agent’s own claim of success counts for nothing on its own. When the judge is in doubt about a core expectation, it’s told to mark it missing, because a false pass on a broken behavior is the worst outcome the system can produce.

A real graded run from our recorded QA environment shows a specific expectation doing its job. The test required work on a cycle named ‘360-QA-Regression-2026’, the agent operated on ‘360-Review-Fiscal-Year-2026’ instead, and the run failed precisely because the expected outcome had named its target. A vaguer outcome, “the participants are added to the cycle”, would have passed that run. The named expectation turned a wrong-target run into a red result with a reason attached, and that verdict gets a full walkthrough in the judge piece.

Two mechanisms keep this strictness from curdling into noise. When the app behaves differently from what your outcome implied but the difference still satisfies the intent, a renamed button, an extra confirmation step, the verifier doesn’t fail the run. It records an assumption flag with what was observed, what was expected, and why the change looks deliberate. A flagged pass is still a pass, and a flag is never allowed to excuse a genuinely unmet core expectation. And when a run does fail on the first attempt, the act phase is re-run once on a stronger model before the failure is treated as real, so agent fumbles don’t get billed to your product. Every run still lands in exactly one of four outcomes, passed, failed, skipped, or blocked, and the full loop from generation to verdict inherits its signal quality from the one prose field this piece is about.

So that’s where I’d spend your review time. When a generated suite lands, the prompt gets a skim. The expected outcome gets the scrutiny a code reviewer gives an assertion block, because that’s exactly what it is, an assertion block that happens to be written in prose and enforced by a judge that believes evidence over enthusiasm.

False Positives in Automated Testing

Wed, 05 Aug 2026 00:00:00 GMT

False positives in automated testing are results that report the wrong thing in either direction: a false alarm marks the app broken when only the test or the agent driving it misfired, and a false pass marks the app healthy when a real defect slipped through. Fixing them takes two separate mechanisms, one per direction. TestAutomate attacks the pair with a skeptical verifier against false passes and one disclosed retry on a stronger model against false alarms.

Most AI test automation tooling only worries about one direction, and usually treats it with the bluntest instrument on the shelf, retrying until the red goes away. I think the blunt instrument does more damage than the disease. So what follows is both failure directions, the mechanism we designed for each, and the decision rules I’d apply to any suite, ours or not.

What’s the difference between a false positive and a false negative in software testing?

A false positive is a run that reports a defect that doesn’t exist, and a false negative is a run that stays green past a defect that does. The vocabulary trips people constantly, because the statistical convention testing borrowed defines positive as the condition being detected. The condition a test screens for is a defect, so a positive is a red result, and a false positive is red with nothing real behind it. Plenty of engineers use the words the other way around, since a positive result sounds like it ought to mean a pass. Both camps are internally consistent, which is exactly why the false positive vs false negative in software testing argument never converges.

My decision rule is to drop the statistics vocabulary at the door and name errors by the harm they do. A false alarm wastes an investigation and erodes attention. A false pass ships a bug. Neither phrase gets misunderstood in a standup.

Naming them separately matters because their costs behave differently. A false pass is worse per incident, since the defect it hides travels to production wearing a green stamp. False alarms are worse in aggregate, because they compound. Each red that turns out to be nothing lowers the energy anyone brings to the next red, and a suite whose reds are mostly noise ends up unread, which converts every future false pass from unlucky to inevitable. That’s why I treat the two directions as one problem with two fronts. A fix that buys fewer false alarms by quietly accepting more false passes, which is what blind retries do, isn’t a fix. It’s a transfer.

Why do agent-driven tests fail when the app is fine?

Because two different systems can fail during a run, the application or the agent operating it, and both failures surface identically as a red verdict. In agentic testing, an intent-based test is executed by an agent that reads the live page and acts the way a person would. That removes the classic false-alarm factory of scripted suites, the stale stored selector, but it introduces a new source: judgment. The agent can pick the wrong record from a list, misread an ambiguous menu, or act on a page that merely resembles the right one. The verifier judging the run then reports, correctly, that the expected outcome wasn’t met. Correct judging, wrong culprit.

Here’s a real one from our QA environment, recorded against a test tenancy full of demo data:

The test needed the agent to work inside a review cycle named 360-QA-Regression-2026. It did everything inside 360-Review-Fiscal-Year-2026 instead. The application behaved fine the whole time, and the verifier’s reasoning names the wrong-cycle mistake outright. The red is real in the narrow sense, because the expectations tied to the required cycle genuinely weren’t met, and a false alarm in the sense that matters, because no defect exists. That’s the exact species of failure that needs disambiguating before a human gets paged.

Scripted automation has its own well-measured version of the problem. Google reported that almost 16% of its tests showed some level of flakiness, and the flaky test literature catalogs how timing, environment, and test interdependence produce reds with no code change behind them. Agentic suites don’t escape that gravity. They swap some failure sources for others. So the honest question isn’t whether first attempts can be wrong. They can. It’s what the runner does next.

How do we cut false positives in automated testing?

By making a failure earn its verdict twice, under strict conditions, in the open. When the verifier judges a run failed, TestAutomate re-runs the test’s own steps exactly once on a stronger model than the first attempt used, then judges the new attempt from scratch. Setup isn’t repeated. The precondition the setup phase already established is carried into the retry, so the second attempt starts from the same guaranteed state as the first and the comparison stays clean.

The guards around that retry matter as much as the retry itself, because escalation is only meaningful while it’s scarce. It fires solely for a failure the verifier genuinely judged. A blocked run, where setup couldn’t establish the precondition, never escalates, because re-running a missing precondition on a smarter model fixes nothing. A skipped run never escalates, since nothing was judged. A run aborted mid-flight never escalates. And when the verifier itself was unreachable and the run fell back to a synthetic failure, that never escalates either, because a verdict nobody actually judged hasn’t earned a second attempt.

The two outcomes read very differently, by design. If the stronger attempt passes, the first red was an agent-execution issue rather than an app defect, and the verdict says so. It’s annotated as escalated and passed-on-retry, never laundered into ordinary green, so a test that habitually needs the stronger model stays visible in your history instead of hidden by it. If the stronger attempt fails too, you now hold two independent attempts, judged separately, agreeing the expectations weren’t met. That’s a high-confidence red, and it’s the only kind we ask a human to treat as a probable defect.

One edge case shows how conservative the bookkeeping is. If the verifier goes down during the retry, the runner doesn’t fabricate a confident failed-both-attempts story out of a judgment that never happened. It keeps the first, genuinely judged failure as the authoritative verdict and records that the retry couldn’t be judged.

Why one retry and not three? Because every extra blind attempt is another roll of the dice, and dice rolls launder flakiness. A retry on the same model mostly tests your luck. A single retry on a stronger model changes the capability behind the attempt, which turns the re-run into an experiment: if capability was the problem, the retry passes, and if the app is the problem, no amount of capability will make broken behavior pass. One disclosed, escalated attempt extracts nearly all the information a re-run can offer, and it’s bounded like everything else in the run, so a failing test can’t spiral into an open-ended retry loop.

How do other approaches handle a first-attempt failure?

With one of three defaults: go red immediately, retry blind until green, or queue the red for a human to triage. Each default answers the agent-error-or-app-defect question differently, and the differences show up most clearly in what happens to each kind of red.

Policy after a first red	Where it’s the default	A real defect ends up	A flaky or agent-error red ends up	What the record discloses
Red immediately, no second look	Scripted, selector-based suites in stock CI	Reported at once	Reported identically as a failure	Nothing, every red reads the same
Blind auto-retry until green	Retry flags bolted onto scripted and record/replay suites	Occasionally lucked past on a retry, a false pass	Silently converted to green	Usually nothing, the pass looks first-try
Human triage queue	Managed QA services	Confirmed by a person, on the service’s timeline	Correctly dismissed, at per-red human cost	An accurate write-up, after the wait
One disclosed re-run on a stronger model, judged failures only	TestAutomate	Fails both attempts, marked high confidence	Passes on retry, annotated as escalated	The verdict carries the escalation and both outcomes

Honest caveats, because every row is defensible in its own house. Failing fast is the right default when reds are cheap to triage and the suite is small. Blind auto-retry exists because transient infrastructure hiccups are real, and a bounded retry genuinely absorbs them. Its damage isn’t the retry, it’s the silence, a pass that hides its own history, and teams that log retried outcomes and review the log have rebuilt much of the missing disclosure by hand. A good human triage queue is the most accurate classifier on the table, since people are excellent at telling agent error from defect. You’re paying latency and per-red cost for a distinction structure could have made upfront. And our policy has a cost too. The failing subset of a suite runs twice, with the second attempt on a stronger model that costs more per run. I think that’s the right place to concentrate spend, on exactly the runs where a second data point changes the conclusion, but it is spend.

The disclosure is also what makes flaky test detection and quarantine workable without a separate tool. A test that passes on retry once probably had an ordinary agent stumble. A test whose history shows escalation after escalation is telling you something is unstable, in the test, the feature, or the environment, and because every escalation is annotated on the run’s record, that history is evidence rather than anecdote. Quarantine should then be a decision, not a side effect. Pull the test out deliberately, with a ticket and an owner, the way Martin Fowler’s quarantine advice has long recommended, rather than letting silent retries quarantine it for you inside a green checkmark. There’s a longer treatment of what AI changes about flaky tests on this blog if flakiness is your suite’s dominant noise source.

What stops the false pass in the other direction?

A verifier that’s structurally separate from the agent and openly skeptical of it. The agent that drives the browser never grades its own run. A separate judge model reads the recorded trajectory against the test’s expected outcome and returns an itemized verdict, every expectation met and every expectation missing. Its instructions encode the asymmetry from the top of this piece: claiming a pass when a core behavior broke is far worse than a false alarm, an agent’s unevidenced claim of success counts as not met, and when in doubt on a core expectation, the verifier marks it missing and fails the run.

That skepticism is deliberately biased toward false alarms, which is precisely why the escalation retry exists. A judge that errs toward red needs a disclosed mechanism on the other side to keep its strict failures honest, and a retry that errs toward rescue needs a strict judge to keep its greens meaningful. Neither mechanism works alone, and I wouldn’t ship either one by itself.

Two walls keep the remaining noise out of the failure channel entirely. Expectations prefixed [incidental] can never fail a run, so a missing success toast doesn’t produce a red that outranks the point of the test, and the cosmetic assertions that generate so many trivial false alarms in scripted suites get a place to live where they can’t hurt anyone. And runs that never exercised the app, blocked on a precondition or skipped by configuration, end in their own outcomes rather than as failures. Four outcomes, passed, failed, skipped, blocked, each meaning one thing, so the failure column only ever contains runs where the application itself did something wrong.

The far end of the pipeline holds the same line. A judged failure can become a bug candidate, deduplicated against existing issues by a strict check, but nothing is ever filed anywhere without a human clicking to confirm it. Blocked and skipped runs can’t produce candidates at all. By the time a defect report reaches your tracker, it has survived a skeptical verdict, a stronger-model re-run, a dedup check, and a person.

Your half of the bargain is the expected outcome itself. A vague assertion is a false-pass generator no verifier can rescue, because a test that can’t lose isn’t testing anything, and I keep a separate checklist for writing expected results that’s mostly about closing that hole.

What all of this buys is test suite trust, which is the only asset a suite really has. Green has to mean examined and found whole. Red has to mean twice-earned and worth your minutes. Every mechanism above exists to keep one of those sentences true, and the day either stops being true, the suite stops being a safety net and starts being a dashboard. I built the verdict pipeline so we’d never have to choose which direction of wrong to tolerate. The answer is neither, enforced separately.

How Does AI Test Case Generation Work?

Tue, 04 Aug 2026 00:00:00 GMT

Every pitch for AI test case generation shows you the moment a test appears out of a docs page. I want to show you the opposite moment, the one that actually decides whether the generated suite is any good: the moment a page gets thrown away. I built TestAutomate’s ingestion pipeline to be a rejection machine first and a generator second, and what follows is an inventory of everything it refuses to turn into a test.

AI test case generation, done honestly, is a filtering problem before it’s a writing problem. TestAutomate crawls your product docs, discards every page that doesn’t describe a real user task, extracts the survivors into skills, and only then drafts runnable plain-language tests from them. Most of the input never becomes a test, on purpose, because a generated test with nothing real behind it costs more than no test at all.

When I pointed the pipeline at a real docs site and read the diagnostics afterward, the reject pile was the more interesting half of the story. So rather than walk the pipeline forward the way that post does, I’ll walk it as a funnel: four cuts between a docs URL and a kept test, each with a reason you can inspect.

How does AI test case generation work in TestAutomate?

Four stages run in order, crawl, extract, classify, emit, and then a separate drafting step turns what survived into tests. The crawler walks your documentation starting from the URL you paste. Extraction reads each fetched page and pulls out the user-facing actions it describes, keeping the doc’s own wording for buttons and fields instead of paraphrasing it. A classification pass sorts each extracted intent by what kind of handling it needs at runtime. The emit stage stores the survivors in your workspace as skills, each one a distilled procedure a user could actually perform.

Nothing gets generated at ingestion time. Test drafting waits until the first time someone clicks Run suite on an app that has no test plan yet, and I made it lazy deliberately: the skills sit in your workspace where you can look them over before a single test exists, and an ingestion run you abandon never litters your suite with drafts. When drafting does run, it works in two steps. First the app’s skills get clustered into topics, without you defining categories anywhere. Then a small number of test flows get written per topic.

Every drafted test carries the same three required fields as a hand-written one, an id, a prompt, and an expected outcome, with optional setup and cleanup prompts around them. The expected outcome comes out as itemized end-state assertions, including negative ones like “no other records were modified”, and each assertion is load-bearing unless it’s deliberately marked incidental, in which case its absence can never fail the test. The format rules are blunt: plain natural language only, with code, JSON, YAML, and selectors explicitly forbidden inside the prompt and the expected outcome. The agent re-derives how to execute each step against the live page at run time, the same contract behind everything else in our agentic testing setup, so nothing brittle gets frozen in at generation time.

That’s the machinery, and it’s also the least interesting part. None of it explains why the output is worth running. The filters do.

Which pages never even get fetched?

Whole sections of your docs site are excluded by URL before a single request goes out. The crawl derives its scope from the URL you paste: it stays on that site’s own origin, inside the docs subtree you pointed it at, and it carries an exclusion list for address patterns that never describe testable behavior. Changelog paths, blog posts, about pages, pricing, customer showcases, careers pages, and login and signup routes are all rejected on the URL string alone. A web crawler that fetched everything reachable would just manufacture work for the later filters, so the cheapest cut happens first.

There’s a practical decision rule hiding in that scope derivation. The crawl treats the docs section containing the page you paste as its territory, so pasting any page inside your user docs gets you the whole section, while pasting a developer portal gets you a pile of API reference that the task filter will throw away later anyway. Point it at the help content your users actually read, and the funnel starts full of the right raw material.

Discovery is deliberately boring. The crawler tries the site’s sitemap first, since a published sitemap is the most reliable inventory of a docs site’s pages, and falls back to following in-scope links when no sitemap exists. The whole walk is bounded and politely paced, and both bounds are qualitative on purpose: enough pages to cover a real docs section, slow enough not to hammer anyone’s server.

What does get fetched is then reduced to its actual content. Navigation menus, headers, footers, and scripts are stripped before anything downstream sees the page, so a page gets judged on what it says rather than the chrome around it. Pages with almost nothing left after that, nav-only stubs and error shells, are dropped at the door.

One limitation gets surfaced loudly instead of papered over. The default crawl reads static HTML, so a docs site that renders its content entirely through JavaScript can come back nearly empty. When that happens, the pipeline says so in plain language, names the likely reason, and can crawl by rendering pages through your own browser instead for sites that need it. A pipeline that silently produced zero tests from a JavaScript-rendered site would look identical to one that judged your docs untestable, and removing that ambiguity is exactly what the streamed diagnostics are for.

What does extraction refuse to turn into a test?

Anything that isn’t a discrete action a user takes in the UI. Extraction is the stage where a model reads each surviving page and pulls out intents, numbered steps, and the domain entities the page mentions, and its rules are ruthless about the boundary. Marketing copy is not a task. A conceptual overview without steps is not a task. API endpoint references, pricing and billing pages, changelog entries, and what’s-new announcements are all named exclusions. A page describing none of the above simply emits nothing, and the pipeline treats an empty result as a legitimate answer rather than a failure to try harder.

The contrast is easy to picture. A page called “About projects” that spends four paragraphs on philosophy emits nothing. A page called “Create a project” with numbered steps and the button names spelled out emits an intent, a procedure with those steps intact, and the entities the page mentioned. Both pages look like documentation. Only one of them describes something a browser agent could execute and a verifier could check.

The empty-result rule is the design decision I’d defend hardest. The extraction contract says that when information isn’t clearly in the docs, the model does not make it up, and an empty result beats a half-answer. Generation systems fail in two directions, refusing good input or inventing structure that was never there, and the second failure is far more expensive in testing. An invented procedure becomes an invented test case that fails against the real product for reasons nobody can diagnose.

Two size gates run alongside the task filter, both kept qualitative. Pages too thin to describe a real procedure are skipped outright, because nav indexes and redirect stubs never carry steps worth extracting. Very long pages are truncated with an explicit note rather than silently cut off, so the missing half is never mistaken for absent behavior.

For a page that survives, fidelity rules take over. Steps keep the doc’s exact wording for buttons, fields, and menu items. Steps the doc doesn’t describe don’t get invented. A procedure that comes back without any steps is dropped in validation, as is one that doesn’t attach to a real extracted intent. And a single page failing mid-extraction, a timeout or a malformed response, gets logged and skipped without killing the run, because one bad page shouldn’t cost you the rest of the site.

Which filters run at suite time instead of ingestion?

Three more: an auth strip, a merge, and a lint, all applied when tests get drafted rather than when docs get ingested.

The auth strip is the filter people notice first, because the progress stream calls it out, right down to telling you how many skills it removed. Skills that are purely about logging in, signing up, or resetting a password are stripped before drafting. The runner assumes you’re already signed in to your own browser profile, so a generated login test would be broken by design and would demand credentials the system never asks you for. The nuance is what survives: admin-side configuration flows like SSO setup stay in, because an admin configuring authentication is a real product task, while a user authenticating is a precondition.

The merge is quieter. Skills don’t map one-to-one onto tests. The drafting rules push related actions into end-to-end flows and skip trivial sub-actions already covered by a larger flow, so a topic with a dozen skills might emit a few tests rather than twelve. For a topic mixing reads and writes, the rules ask for a happy path, a variant exercising optional fields, and an edge case where an obvious one exists. That’s the honest scope of AI-generated edge case test coverage here: an edge case earns a test when the docs imply one, not because a quota demanded one.

The lint is the final cut and the one I consider least optional. The drafting rules already ban cross-test dependencies, wording that assumes a resource was created in a prior step, and hedged fallbacks about whether a resource exists without a setup prompt to guarantee it. A model can still slip. So a deterministic lint re-reads every drafted flow and drops the ones that assume data another test created, or that grade against a named resource nothing guarantees exists, recording the test and the reason. A flow like that isn’t a weak test, it’s a false alarm on a timer, destined to fail for environment reasons and to erode trust in the verdicts that matter. A bad test is worse than a missing one, and the lint is that sentence enforced in code.

What do different generation approaches accept, and what do they emit?

Manual authoring accepts whatever the author knows and emits documents, record/replay accepts a click-through and emits a recording, test-case-text generators accept a requirements doc and emit more text, and TestAutomate accepts a docs URL and emits runnable tests. The input contract is the clearest axis I know for comparing these, because the input decides how much filtering you get for free and how much junk you inherit downstream.

Approach	Accepts as input	Emits as output
Manual authoring	Anything the author knows: docs, tickets, exploratory sessions, tribal knowledge	Test cases in whatever format the team maintains, automated later if ever
Record/replay capture	A live click-through of one flow in the running app	A recording of that session, tied to the pages it saw that day
Test-case-text generators	A requirements doc, user story, or PRD pasted in by hand	Test-case text, steps and expected results, that still needs automating
TestAutomate docs-to-executable pipeline	A documentation URL, filtered down to pages describing real tasks	Runnable plain-language tests: id, prompt, expected outcome, optional setup and cleanup

Every row has an honest advantage the table undersells. Manual authoring is the only approach where judgment about what matters commercially arrives for free, and no filter substitutes for a person who knows which flow bleeds revenue when it breaks. Record/replay needs no documentation at all, which makes it the fastest route to covering one flow that exists today, precisely where a docs-first pipeline has nothing to read. Text generators do fine when a current requirements document exists, and if you’re evaluating a PRD to test cases workflow while your PRDs are genuinely maintained, that route is defensible. A companion post examined why that document so often doesn’t exist for the features that need regression coverage most. And our own row inherits the quality of your docs wholesale. A page describing a flow wrong yields a test that faithfully checks the wrong thing, and a thin page describing a genuinely important task gets skipped, because no filter can know it mattered.

What’s left after the funnel is an editor’s job, not an author’s. On the one ingestion run we’ve recorded, a real docs site came out the far end as roughly 40 skills and a clustered draft suite, and the draft still included one test whose expected outcome was too loose until I tightened it by hand. I’d rather tell you that than pretend the funnel ends at perfect. Filters decide what’s worth drafting. Generation drafts it. A skeptical verifier grades every run afterward. Your judgment stays where it belongs, on the one question no filter can answer: whether the test that survived is checking something worth knowing.

Autonomous Bug Reporting to Jira, Explained

Mon, 03 Aug 2026 00:00:00 GMT

Autonomous bug reporting in TestAutomate means the agent notices a defect mid-run and writes it down, not that it decides to open a ticket. When it hits a broken control or an error page, it records a pending bug candidate with a summary, a severity, and a short account of what it did and what happened, then keeps working. That candidate waits in your dashboard until a person reviews it, a dedup check runs, and someone clicks File. Nothing reaches Jira by itself.

I’m spelling that out because the phrase usually gets read the opposite way. Most of what ranks for it treats autonomy as removing the human entirely. An AI agent files Jira tickets the instant a check goes red, and the pitch is speed. We made the opposite trade on purpose. An agent mid-task is good at noticing and bad at knowing what your backlog needs, so TestAutomate automates everything up to the filing decision and leaves the decision itself alone.

The write to Jira is one API call, so it’s also the least interesting part. The design work lives in what a candidate has to survive before a human earns a one-click File button. That’s the pipeline worth walking through.

How does a bug candidate get captured mid-run?

Capture starts the moment the agent decides that what it’s looking at is the app’s fault rather than its own. The standing instruction it carries is specific about that boundary. An error page or an error toast, a broken or non-functional control, a console or network error, a crash, or behavior that clearly contradicts what the app should do all qualify. Its own navigation mistakes don’t, and neither does anything that looks like a missing precondition or a feature the user simply hasn’t configured. When something qualifies, the agent records the defect with a one-line summary, a short account of what it did and what happened, and a rough severity on a four-step scale from low to critical, and then carries on with its task. The severity is a hint for the reviewer, not a gate anything enforces. Capture never interrupts the run. The candidate appears in the dashboard’s review list while the agent is still working.

Candidates arrive from three directions. A live session, where you’re driving the agent interactively and it trips over something real. A suite run, where a regression test failed and the failure survived triage. And a fix re-check, where an issue that was supposedly resolved turns out to still reproduce, and the fresh evidence comes back as a new candidate instead of a comment nobody reads.

The suite path has the most machinery behind it, because a failed test is not automatically a bug. Before a suite failure is even considered, it has been through a skeptical verifier that itemizes which expectations were met and which weren’t, then a full retry on a stronger model, since some first-attempt failures are agent error rather than app defects. Only if the failure stands does a triage judge read the verifier’s reasoning and decide whether it describes a real application defect or a test problem, things like a missing precondition, a timing wobble, or a test written for a feature this build doesn’t ship. Only real defects become candidates, and the judge is instructed to be conservative about the call. That step is AI bug triage in the literal sense, and it runs before anything asks for a human’s attention rather than after.

One class of run is shut out entirely. A blocked run, where the test’s preconditions couldn’t be established, never becomes a candidate, because the application was never exercised and there’s no behavior to report. That exclusion comes from the same design position as why blocked is not a failure, and it keeps the queue’s meaning sharp. Every card in the candidates list traces back to a run where the app actually did something wrong.

Why autonomous bug reporting stops at a candidate

Because filing is a judgment about your backlog, not just about the defect, and the person who owns the backlog is a better judge of it than an agent halfway through a task. A wrongly filed ticket bills its cost to whoever triages it later. So we drew the line where the cost changes hands. TestAutomate automates the noticing, the evidence, the severity guess, and the duplicate check, and it wires exactly one path that creates issues, the File button in the dashboard. A candidate can be filed, dismissed, or left to wait. Two of those three paths never touch Jira at all.

Placing the human there is the real difference between the ways a red check becomes a ticket today. Plenty of pipelines auto create Jira tickets from failed tests through a CI webhook, and most guides you’ll find for that phrase are happy to show you how. Here’s how the paths compare on the trip from a failed check to a filed ticket.

Path from red check to ticket	Who decides it’s ticket-worthy	When Jira gets written	Duplicate handling	Evidence on the ticket
CI webhook auto-filing	Nobody, every failure qualifies	Immediately, on each red run	None by default, so retries and flakes arrive as clones	Whatever the job exports, often a log link
Manual QA write-up	The tester who saw it	Whenever the write-up gets done	The tester’s memory of the backlog	What the tester pastes in by hand
Managed service report	The vendor’s tester	On the service’s reporting cycle	The vendor’s internal process	The vendor’s report format
TestAutomate candidates	A person reviewing pre-built evidence	Only after a File click clears the dedup check	Live Jira search plus cached inventory, strictly judged, human-confirmed	Summary, severity, source context, verifier reasoning and unmet expectations

That table needs its caveats said in prose. CI auto-filing is a reasonable trade for a small suite whose failures are rare and nearly always real, and Jira’s own automation rules can bolt duplicate guards onto it if someone maintains the conditions. My objection is that the guard is opt-in engineering rather than the default. A careful QA engineer’s manual write-up is still often the single best bug report in any tracker. The cost is that it competes with everything else in that person’s day. And managed services do produce accurate reports. What you’re paying is the cycle, since the ticket lands on their timeline rather than the minute the defect was seen.

The review itself has to stay quick, or the gate stops earning its cost. My working rule has three branches. File when the evidence shows the app misbehaving and the backlog owner would want to know this week. Dismiss when the candidate describes behavior that’s real but already understood, an accepted quirk, a known limitation, or something the run mislabeled. Leave it pending when you want a second opinion, since a waiting candidate costs nothing and loses nothing. The habit to resist is dismissing just to keep the list tidy. Every candidate carries evidence a run already paid for, and dismissing unread is the one way this pipeline can still lose a real defect.

How do you prevent duplicate Jira tickets from test failures?

You check before you create, and you make a judge defend every match. When someone clicks File, TestAutomate searches your live Jira for issues whose summaries overlap the candidate’s, folds in anything similar from a locally cached inventory of your board, and hands that shortlist to a judge that is deliberately strict about what counts as the same bug, so two candidates don’t get merged just for living in the same corner of the app. I made it fail in that direction because same-area matching is exactly how real bugs get swallowed by overeager dedup.

A match doesn’t make anything disappear. The dashboard shows the existing issue’s key, status, and summary alongside the judge’s one-line reason, and asks whether you want to file anyway. Forcing a genuinely new ticket is one more click. A clean miss goes straight to creation. The failure mode is chosen deliberately too. If the dedup judge can’t run at all, the pipeline treats the result as no match and offers the create path, because an occasional duplicate is recoverable with a link and a close, while a silently swallowed defect is gone.

None of this pretends duplicate detection is a solved problem. It’s an old and genuinely hard research area, and a recent benchmark study of duplicate bug report detection found that even sophisticated techniques struggle to beat simpler retrieval on real issue-tracker data. What makes the job tractable here is scope. The judge isn’t matching against the world’s bugs. It’s matching one fresh, well-evidenced candidate against one team’s board, with a human confirming the verdict either way. And under our bring-your-own-key model, that judge is one more Claude call billed to your own Anthropic key with no markup, so running dedup on every File click costs what one model call costs and nothing more.

What lands in Jira when someone clicks File?

A boring, well-formed bug, which is exactly the goal. The issue is created as a Bug type in the Jira project you’ve configured. Its summary is the candidate’s one-liner, and its description carries the candidate’s details, the source it came from, its severity, and the URL where the defect was observed. For a suite-run candidate, the details name the test and quote the verifier’s reasoning along with the itemized expectations that went unmet, which makes a serviceable starting point for reproduction without anyone writing it up by hand.

What the ticket carries is written evidence, not pictures. A candidate records the agent’s own account of what it did and what the app did back, and for a suite-run candidate the verifier’s reasoning and the unmet expectations travel with it. Visual evidence stays where it was captured, in the run’s recorded trajectory on the dashboard, which is where you go when the written account isn’t enough.

There’s no browser automation anywhere in this write path either. Filing talks to Atlassian’s Jira Cloud REST API directly, and once the issue exists, a fresh copy of it is pulled into that cached inventory so the next dedup pass and the next status question both see it. The inventory earns its keep beyond dedup. When Jira is connected, the agent answers issue questions through structured lookups and is barred outright from driving the Jira website in a browser, which would be slower and flakier than asking the API.

The candidate itself doesn’t vanish after filing either. It flips to a filed state with the new issue key stamped on it, so the dashboard keeps the paper trail from first observation to ticket. If the same defect ever comes up in conversation again, the trail answers where it was observed, which ticket it became, and when it was filed, without anyone spelunking through Jira history.

One small design choice I’ll defend from the same corner of the product. The setup mistake we anticipated most was pointing the integration at your-org.atlassian.com instead of .net, so the connection error hint says so in as many words and tells you to check the site URL. An integration’s error message should name the likely mistake, not recite the HTTP status.

Can the loop close the ticket after a fix?

It can, with the same shape of gate on the way out. TestAutomate can re-check filed issues against the live app. A re-check agent attempts the reported defect and lands on a verdict, it still reproduces, it’s fixed, it was blocked by missing access, or it genuinely couldn’t tell. When the verdict is reproduces, the agent records it while the failing state is still on screen, so the run’s own record captures what it saw, and the issue comes back into the pipeline as a fresh bug candidate whose details carry the evidence trail. A bad fix resurfaces through the same review queue a new defect would, instead of dying in a comment thread.

Closing is just as explicit as filing. Moving a ticket to done is a separate, deliberate action, never a side effect of a verdict, and the close path posts the fix evidence as a comment before attempting the transition, so an audit trail exists even if the transition step fails. Nothing in the product transitions tickets on a schedule, and an inconclusive or blocked re-check is reported for a human to read, not acted on.

Put the gates in a row and you get what autonomous bug reporting means as we built it. Autonomy in the noticing, the evidence gathering, the triage, and the duplicate check. A human at both writes, the create and the close. The observations come from an agent that reads the page and finds elements the way a person would, the grading comes from a separate judge that doesn’t take the agent’s word for anything, and the full loop those pieces belong to is mapped in the agentic testing guide. The ticket your teammates finally see is the least autonomous artifact in the entire chain. That’s by design. It’s the one thing a person chose to send.

Bring Your Own Key AI Testing, Explained

Sun, 02 Aug 2026 00:00:00 GMT

Bring your own key AI testing means the testing tool doesn’t resell model access. You paste your own provider API key into its settings, every model call the tool makes runs against your account at provider rates, and the spend shows up on your own billing dashboard instead of inside a vendor invoice. In TestAutomate this is the only billing model there is. Claude calls bill the workspace’s own Anthropic key, with no markup and no per-seat fee.

I designed the billing path that way because I distrust blended pricing in agentic tools, where the vendor’s margin hides inside a number you can’t decompose. A key you own is a number you can audit. But BYOK drags a set of fair questions behind it, about whether it’s safe to hand a tool your key, whether it really comes out cheaper than a subscription, and who holds the data once every call runs on your account. I’ll take them in order, and I’ll be honest about the places where BYOK asks more of you rather than less.

What does bring your own key AI testing actually change?

It changes who your billing relationship is with, and that decides what you can verify for yourself. Under a subscription, the model spend is the vendor’s cost and your invoice is an abstraction over it. Under BYOK the meter is yours. Every Claude call TestAutomate makes on your behalf lands on your own Anthropic key at Anthropic’s rates, and you can open your provider console and watch it happen, per model and per day, without taking anyone’s word for anything.

It pays to itemize which calls those are, because a “no markup” claim deserves an inventory behind it. The main kinds of work that bill your key: the agent driving a real browser through a test’s steps, the separate verifier that grades each run against its expected outcome, suite generation, which drafts intent-based tests from your product documentation, and the one-time ingestion pass that reads and classifies your docs pages when you first point the product at an app. The retrieval machinery underneath the generation side, the embedding and lookup work behind generating a regression suite from your docs, runs locally on the server and costs nothing per call.

I won’t claim every feature bills Anthropic, because one doesn’t. Optional voice input relays audio through OpenAI and needs its own OpenAI key configured server-side before it works at all. Leave voice off, which is where it starts, and your Anthropic key is the only per-call API bill in the product.

There are also two ways to pay that Anthropic bill. The first is a plain first-party key from the Anthropic Console. The second is Claude Platform on AWS, where you authenticate with a workspace API key, a workspace ID, and an AWS region, and the money flows through AWS Marketplace billing. The naming invites confusion, so to be exact, that is Anthropic’s own API reached through AWS, not Amazon Bedrock, and TestAutomate treats the two providers identically once connected.

BYOK vs subscription: where does the money actually go?

Four billing models cover most of the AI testing market, and the real difference between them is less the total than what the money buys and how much of the truth you can see. This is the comparison I’d run before taking any quote at face value.

Billing model	What the money buys	How the cost scales	Where you see real usage	An idle month
Per-seat subscription	Access per person, with model spend bundled inside	With headcount, whether or not tests run	The vendor’s invoice, usage abstracted away	Bills the same as a busy one
Usage-based with markup	Model calls resold with the vendor’s margin inside	With volume, plus the margin on every call	The vendor’s credit meter, one step removed	Low, subject to plan minimums
Managed QA retainer	Human testers’ time on a recurring contract	With scope and cycles, renegotiated rather than metered	Reports and invoices, no usage meter at all	Bills the retainer regardless
TestAutomate, BYOK pay-as-you-go	Model calls at provider rates on your own key, no markup, no per-seat fee	With how much testing you actually run	Your own provider console, per model and per workspace	Costs nothing beyond what you ran

Honest caveats, because a table flattens things. A per-seat subscription is a bundle, not a trick. It buys support, hosting, and a predictable invoice a finance team can approve once, and for some organizations that predictability is worth real money. Vendors who mark up usage are funding their operations out of the margin, which is a legitimate way to run a company, just not a transparent one. A managed QA retainer buys human judgment, and no metering model replaces that. BYOK is the leanest of the four precisely because it hands you jobs the other models do for you, key custody, spend monitoring, and provider account hygiene. Most of the rest of this piece is about doing those jobs well.

Is it safe to use your own API key in a testing tool?

It’s exactly as safe as the tool’s key handling, so judge the handling rather than the slogan. When people ask whether it’s safe to use your own API key, what they usually mean is what happens between pasting the key and the tool being trusted with it, and that part I can answer mechanically, because we built each step to be inspectable.

TestAutomate verifies before it stores. Pasting a key fires a deliberately tiny live call against the Anthropic API, so an invalid or expired key surfaces as an immediate, plainly worded authentication error instead of failing silently on your first real suite run. Only after that probe succeeds does the key get saved at all.

Storage is write-only by design. The key lives server-side, scoped to your own workspace, and it’s encrypted at rest when the deployment has an encryption secret configured. When one isn’t configured, the server says so plainly instead of pretending otherwise. After that, no endpoint ever returns the key. Not the settings page, not the status API, nothing. What comes back is a masked last-four hint, enough to tell which key is connected without the secret ever being echoed.

One more property matters on shared deployments. Your stored key always outranks any operator-level default, so on a hosted instance where an operator configured a fallback key for the deployment, your tests still bill your account, not theirs. That precedence is deliberate. It keeps every workspace’s costs on its own meter no matter who runs the server.

Whatever tool you’re evaluating, mine included, the checklist is the same. Ask whether the key is verified before it’s accepted. Ask whether it’s encrypted at rest and what happens when the encryption secret is absent. Ask whether any endpoint can return the raw key, because a key that can be read back is a key that can leak. Then do your part on the provider side. Issue the tool its own key from a dedicated workspace rather than reusing anything shared with production, give the key an expiration, and rotate it on a schedule. Anthropic’s key management documentation covers auditing keys, expiration timestamps, and deactivating a key without touching the rest of your org.

Is BYOK cheaper than a subscription?

Usually, for a team that runs tests regularly, and I can argue that without quoting anyone’s price list, because the structure does the arguing. With no markup on calls and no per-seat fee, the only thing you pay for is work that actually happened, at the same rates you’d pay your provider for anything else. So the question of whether BYOK is cheaper than a subscription reduces to a comparison you can run yourself. Connect a key, run a representative couple of weeks of suites, and read the real number off your own console next to whatever quote is on the table.

It helps to know what that number is made of, because pay-as-you-go AI testing has a cost anatomy that flat pricing hides.

Run length dominates. A long, driving-heavy journey test spends more tokens than a short smoke check, roughly in proportion to how much reading and acting it takes, which is exactly the kind of variance a blended per-test price averages into invisibility. Verdicts add one grading call per test. Failures cost more than passes, deliberately, because a genuinely failed first attempt is re-run once on a stronger model before the red is allowed to stand. That design exists to kill false alarms, and it concentrates the expensive model on the small subset of runs where a second opinion could change the conclusion. Blocked runs sit at the cheap end, since a test whose precondition can’t be established stops before its own steps ever execute, one more reason an environment problem is reported as blocked rather than failed.

Two parts of the anatomy cost nothing at all. Retrieval, the machinery that ingests your docs and looks up suite context, runs locally at zero API cost. And an idle month is genuinely idle. No seats to true up, no minimum to hit, nothing metered while nothing runs.

When does a subscription still win? If your testing is tiny and sporadic, a bundled plan’s convenience may beat metering’s precision, and there’s no shame in that. The stronger subscription argument is organizational rather than financial. A fixed invoice is easier to budget than a variable one. That’s a real preference, and my answer to it is the next section, because a variable cost with a hard cap behaves like a fixed cost minus the paying-for-idle part.

How do you set spending limits on AI API keys?

On the provider’s side, where a limit is actually enforced rather than politely requested. A testing tool can promise restraint, and TestAutomate does bound its runs so nothing hangs forever, but the enforcement that survives bugs, misconfiguration, and vendor optimism is the one attached to the key itself.

Anthropic gives you the pieces directly in the Console, documented on its workspaces page. Create a dedicated workspace for automated testing and issue TestAutomate’s key inside it, so the key can only touch that workspace’s resources. Then cap the workspace’s monthly spend on its spend-limits settings and add alert thresholds below the cap. Rate limits are configurable per model tier on the same workspace, requests and tokens per minute, and workspace limits can only be set lower than your organization’s, never higher. Usage and cost reporting breaks out per workspace too, which keeps the testing meter clean of everything else your org runs.

Three rules I’d apply to those knobs. Put the alert threshold meaningfully below the cap, because an alert that arrives after spending has already stopped is just bad news with a timestamp. Treat the cap as a circuit breaker rather than a budget, sized to catch a runaway rather than to ration ordinary testing, since a suite that halts mid-regression at month’s end costs you coverage exactly when release pressure peaks. And route the alerts somewhere a human actually looks, not into a dashboard nobody opens.

Done this way, the fixed-versus-variable objection mostly dissolves. A capped, metered cost gives you the invoice ceiling a subscription promises while keeping the idle-month price near zero. What you give up is only the certainty of paying the maximum.

What does BYOK mean for data privacy?

It moves the data relationship from the vendor’s provider account to yours, and it moves the responsibility along with it. When TestAutomate drives a test, the content the agent reads and the prompts and results around it flow to the model provider under your account, your agreement, and your retention settings, not through a vendor’s pooled key where your product’s screens would sit under someone else’s terms. For teams whose test environments hold data that behaves like production data, that’s the substantive BYOK data privacy argument. The calls run under the data-processing terms you already negotiated with your provider.

I want to keep that framing honest, though, because BYOK is a transfer of duty, not a guarantee. Under a vendor’s pooled key, the vendor owns the retention arrangement with the provider. Under your key, you do. What your provider retains, for how long, and under which controls is governed by your own account configuration and agreements, and a vendor saying it never touches your model traffic is not the same sentence as nobody retaining anything. If your compliance posture depends on specific retention behavior, verify it against your own provider agreement, because under bring your own key that agreement is now the one that matters.

That trade, more visibility in exchange for more ownership, is the one I set out to build. A testing tool sees your product at its most unguarded, pre-release and bugs included, and I’d rather the costs, the calls, and the data of that arrangement sit where you can read them than behind an abstraction I’d be asking you to trust. If you want to see how the metered pieces fit together in a full run, generation through verdict, the agentic testing guide maps what bills where end to end.

How Do AI Agents Test Software? A Deep Dive

Sat, 01 Aug 2026 00:00:00 GMT

Agentic testing means an LLM-driven agent operates your application the way a person would, reading the page, deciding what to click, typing into forms, while a separate model judges whether the run actually satisfied a stated outcome. Nothing about the test steps is scripted in advance. The agent decides the path, and a verifier decides the verdict.

I build TestAutomate, so I hear the skeptical version of the question a lot: what makes this “agentic” rather than a fancy wrapper around the same old scripts? Fair challenge. The term gets used loosely enough that the only useful answer is a concrete one, so this guide walks through what actually happens between you typing a sentence and a verdict landing on your dashboard, with real product behavior at every step and real screenshots of it.

How does an agent actually perceive a web page?

Not by holding a script: the agent reads the live page each time it acts and finds the control it needs the way a person would, by what the element is and says rather than by a stored locator string, so a test never contains a CSS class, an XPath, or a screen coordinate. It decides every next step from what the page shows right now, not from a picture of how the page looked when someone wrote the test. That’s the entire reason there’s nothing for you to author when you create a test and nothing for you to repair when a redesign renames half your markup.

It also changes what a test is. In TestAutomate a test is three things you write in natural language: an id, a prompt (the intent, what to do), and an expected_outcome (what must be true afterward). You can add an optional setup step to establish a precondition and a cleanup step to undo whatever the test created, both written the same way. There’s no step recorder to babysit and no exported script that starts rotting the day you record it. When the UI changes, the sentence “create a project called Demo and verify it appears in the list” doesn’t change with it.

Is agentic testing deterministic or probabilistic?

Both, in different places: the test’s declaration is deterministic, the same expected outcome graded against the same contract every single run, while the agent’s execution is probabilistic and free to vary its route through the page between runs, and that split is the real fault line between agentic and scripted automation. Which order the agent reads fields in, how it goes hunting for a control, whether it takes an extra look because a menu was still animating in: none of that is fixed. Two runs of the same test can take visibly different routes through the UI and both correctly pass, because the pass condition is the stated outcome, not a fixed sequence of clicks.

Runs aren’t purely binary either. A run comes back blocked when a setup step couldn’t establish what the test needed, say the app errored or a required control was missing. Blocked is scored as neither a pass nor a failure, because the test never actually exercised the feature, and a blocked run never becomes a bug candidate. That tri-state (pass, fail, blocked, plus a separate skipped outcome for a genuinely inapplicable test) exists specifically so an environment problem doesn’t get miscounted as your app being broken. There’s a full write-up of that distinction in why automated tests get blocked instead of failed.

Here’s what that looks like on a real run rather than in theory. This is our dashboard showing a recorded suite run where the test “Add Participants to Existing Cycle” came back blocked, with the verifier’s reasoning spelled out on the row: the setup phase couldn’t establish the precondition, the app was never exercised, and the run is explicitly called out as a test-environment problem, not an app defect.

How does the verifier grade an agentic test run?

Every run gets graded by a separate verifier model, not by the agent that did the driving: the verifier reads the full record of the run against your expected_outcome and returns an itemized verdict naming which expectations were met and which are missing, so what you get back is a checklist with reasoning, not a single red or green light. It’s built to be skeptical by design. It looks for evidence in what actually happened on the page rather than taking the agent’s own account of success at face value, and when a core expectation is genuinely in doubt it errs toward failing the run, because a false pass that lets a broken feature ship is worse than a false alarm that costs you a second look.

The verifier can also attach an assumption flag to a passing run when something looked off without breaking a core expectation, but a flag is never a way to pass a broken test. If a core expectation is unmet, the run fails, full stop. The mechanics of the judging setup have their own write-up in how the LLM-as-a-judge verifier grades runs.

A probabilistic actor doing the driving is only trustworthy if something separate and skeptical checks its work. Here’s a real failed verdict from a recorded run, where the agent operated on the wrong item, a review cycle named “360-Review-Fiscal-Year-2026” instead of the required “360-QA-Regression-2026”, and the verifier caught it and failed the run with that exact reason on the row:

That screenshot is worth pausing on, because it shows the failure mode people worry most about with agentic testing, the agent doing something plausible but wrong, being caught by the layer that exists to catch it. And a genuine judged failure isn’t final on the first attempt: the test gets exactly one automatic re-run on a stronger model before the failure is called real. A pass on that retry is recorded as agent error rather than an app defect, and failing both attempts is recorded as high confidence the issue is real. False alarms get filtered without a human triaging every red; the full logic is in why we re-run failures before calling them real.

How is agentic testing different from traditional test automation?

A scripted test is a bet on a locator staying put, a specific class, id, or DOM position, and when a redesign or even a class-rename voids that bet, the test fails on a string mismatch that has nothing to do with whether the feature works. An agentic test doesn’t hold that locator at all. It reads the live page each run and finds what it needs by description, so a rename or a reflow that would break a selector-based script is invisible to it. What it can still fail on is different, not absent. A hidden element, a slow-loading menu, or a description that’s genuinely ambiguous will still leave the agent with nothing to click. Removing the selector removes one whole failure mode; it doesn’t remove all of them.

Laid against the ways teams actually automate today, the differences stack up like this:

	Scripted (code + selectors)	Record & replay	Managed QA service	Agentic (TestAutomate)
Who writes the steps	An engineer, in code	You, by recording once	The vendor’s staff	Nobody: you state the goal, the agent picks the steps each run
What a UI change does	Breaks selectors; you repair them	Breaks the recording; you re-record	You wait on the vendor’s queue	Usually nothing: the agent re-reads the changed page
What the assertion is	”Element X has value Y”	The recorded state	Whatever was contracted	An intent-stated outcome, graded item by item
Who judges a failure	You, from a stack trace	You, from a diff	The vendor	A separate verifier model, with reasoning you can read, then one retry on a stronger model
Cost model	Engineer time	Tool seats + re-record time	Per-seat or per-test contract	Your own Anthropic usage, no markup

The other real difference is in what the test declares. A scripted test’s assertion is usually “element X has value Y,” a check against the DOM. An agentic test’s expected_outcome is closer to how a person would describe success: “the task is assigned to the right person, no other tasks were created.” Writing that well is genuinely a skill. Each line of an expected outcome is a core assertion that must hold for the test to pass, unless it’s prefixed [incidental], which marks a nice-to-have, like a success toast, whose absence shouldn’t fail the run. A bad expected outcome produces a confidently wrong verdict in either direction.

What does it cost to run an agentic test?

Whatever your Anthropic usage costs, and nothing on top: none of the calls in a test run are metered or marked up by TestAutomate itself, every one bills straight to whichever Anthropic credentials you’ve configured, and the real cost sits on your own provider dashboard rather than hidden inside a per-seat price. That’s a deliberate contrast with how this market usually charges, where the model spend is bundled into a seat or a quote and you can’t see what any given test actually cost you.

Honesty requires saying that per-test cost isn’t a single number. A long, driving-heavy test costs more than a short one almost by definition, which is exactly the kind of thing a bring-your-own-key setup lets you see for yourself instead of taking a vendor’s blended average on faith. The BYOK post goes deeper on what bills where.

What does a full agentic test run actually look like?

Four phases run in order: an optional setup step establishes a precondition idempotently (“check whether X exists; if not, create it”), the prompt phase drives toward the goal, the verifier grades the outcome, and an optional cleanup step runs afterward to undo what the test created, without ever affecting the verdict. Every phase runs on a bounded time budget and the run itself is bounded too, one of the guardrails that decide what an agent may conclude on its own, so a confused run ends with a real, explained verdict instead of hanging your suite. If setup can’t establish its precondition, the run stops there as blocked, before the feature under test is ever touched, which is exactly what the blocked screenshot above shows.

The shape becomes concrete with a worked example: a test that creates three dependent tasks in a project tracker (“task1,” then “task2” blocked by task1, then “task3” blocked by task2) and switches the project to Timeline view. Its expected_outcome isn’t one assertion. It’s five, spelled out as separate lines: each task exists with the right dates, each dependency is actually set, the view is Timeline and not List or Board, and all three bars render across the correct date range. If the agent gets four of those five right and mislabels one due date, the verdict isn’t a blanket fail. It’s a structured list naming exactly which expectation is missing, which is a materially different debugging experience than a scripted test’s single red X.

If you want the two most concrete pieces of this in TestAutomate specifically: selectorless testing covers the no-selectors idea in more depth, and generating a suite from your own docs walks through what the test-generation side actually produces, including a case where it got an expected outcome wrong. Generated tests follow the same intent-only rule as hand-written ones, no code and no selectors allowed in them, and drafts that would depend on leftover state from another test get dropped before they ever run, because a bad test is worse than a missing one. The product walkthrough shows the same pieces assembled end to end.

For background on why fixed selectors break so often in the first place, the W3C WAI-ARIA authoring practices document the accessible names and roles that well-built pages declare on their controls, the stable, human-meaningful layer that outlives any particular class name or DOM position.

Test Automation Without Selectors, Explained

Fri, 31 Jul 2026 00:00:00 GMT

Selectorless testing is test automation in which the test artifact stores no element locators at all. No CSS classes, no id attributes, no XPath strings. The test states its intent in natural language, and at run time an agent reads the live page and finds each element fresh, the way a person would. Because nothing about how elements get found is written down in advance, there is no stored locator to go stale when the UI changes.

That’s the definition I’ll defend here, and I want to pin it down carefully because “no selectors” is the kind of claim that’s easy to say and easy to fudge. A tool can hide selectors inside a recording, generate them behind a friendly interface, or keep them on life support with a repair layer and still market itself as selector-free. So I’ll use TestAutomate as the working example of selectorless testing, since I built its runner and can point at exactly where the claim holds, but the definition itself doesn’t depend on any one product.

Four questions cover the whole idea: why selectors break tests at all, what replaces them inside the test, how selectorless differs from self-healing, and what it honestly does not fix.

Why do selectors break tests in the first place?

Because every stored selector is a bet that markup you don’t control will hold still. A CSS selector or an XPath expression pins the test to implementation details no user ever sees. Ship a redesign, roll out an A/B variant, or let a styling refactor rename .btn-primary to .btn-primary-v2, and the bet is void. The test fails because a string stopped matching, not because the feature broke. Nobody who used the product that day noticed anything change, and the suite went red anyway.

The standard mitigations are worth naming because each one quietly concedes the point. Teams pin dedicated test-id attributes to controls so the selector has something stable to grip, which works exactly as long as every component, every contributor, and every future refactor honors the convention, and which leaves the frontend carrying markup that exists only for the tests. Teams centralize locators in an abstraction layer so one rename costs one edit instead of forty, which shrinks the repair without questioning why the repair recurs. The selector is still there. It’s still a bet, and it’s still yours to keep alive by hand.

One distinction keeps this whole conversation honest, because two different failure classes get blamed on “unstable tests.” Brittle tests vs flaky tests is a real split, not two words for one problem. A brittle test fails deterministically when something it depends on changes, and the stale selector is the textbook case. The red arrives right after a frontend deploy and repeats on every run until somebody edits the test. A flaky test fails nondeterministically with nothing changed at all, from timing, data, or environment, and that failure class needs entirely different medicine. My diagnostic rule is to look at what shipped just before the suite went red. Red that tracks markup changes is brittleness. Red that comes and goes on an unchanged build is flakiness. Selectorless testing targets the first class, the deterministic breakage caused by stored strings going stale, and it makes no promise about the second class on its own.

What does a selectorless test contain instead?

Three required fields, all natural language: an id, a prompt, and an expected outcome. That is the entire load-bearing artifact in TestAutomate. The id names the test. The prompt tells the agent what to do end to end, the way you’d brief a careful colleague. The expected outcome states what must be true afterward for the run to count as a pass. A test can also carry optional setup and cleanup prompts, written the same way, for establishing preconditions before the run and removing what the run created after it.

Concretely, a whole test can read like this:

id: billing.invoice.download
prompt: sign in as the demo billing admin, open the most recent invoice, and download it as a PDF
expected outcome: the invoice list shows at least one invoice, the downloaded file matches the invoice number shown on screen, and [incidental] a confirmation message appears

That’s everything the artifact holds, and the interesting part is what’s absent. Nothing says how to find the invoice list, which control triggers the download, or where any of it sits in the markup. Finding is the agent’s job at run time, not the author’s job at authoring time.

I made the runner strict about this shape on purpose. It refuses to load a test that’s missing its prompt or its expected outcome, full stop. And there is no other field to fill. No locator slot, no code block, no place where a CSS class or an XPath could live even if you wanted to write one. When people ask me what separates genuinely selectorless tools from selector-hiding ones, that’s my test. Look at the schema of a stored test and check whether a selector could exist in it.

The same bar applies to tests the product writes for you. When TestAutomate drafts a regression suite from your docs, the generation rules explicitly bar code, JSON, YAML, and selectors from the prompt and the expected outcome, and setup and cleanup prompts are held to the identical plain-language restriction. Machine-authored tests don’t get to smuggle a locator in through the back door.

The expected outcome deserves a closer look, because it’s where the rigor that used to live in assertion code moves. It isn’t one vague sentence. It’s an itemized list of end-state assertions, negatives included, in the style of “the confirmation page shows the ordered item” and “no other orders were created.” A bullet can be prefixed [incidental] when it’s a nice-to-have, and an incidental bullet can never fail a run, which stops a cosmetic miss like a toast’s exact wording from outvoting the behavior the test exists to verify.

At run time the artifact meets the page. The agent reads the live page as it exists on this run and finds each element the way a person would, at the moment it’s needed. Nothing the agent holds between runs identifies an element permanently, so the rename that would void a stored selector is simply absorbed on the next read. There’s nothing to update because there was never anything recorded.

Here’s how the stored artifact itself compares across the approaches teams usually weigh. This is a narrower question than how each approach behaves when the UI changes, and it’s the question I find most predictive:

Approach	What the test artifact contains	What can go stale inside it
Scripted, selector-based	Step code plus author-written CSS or XPath locators	Every locator the markup outgrows, plus any step logic written against the old structure
Record/replay	A captured click-by-click recording of one walkthrough of an older UI	The whole recording, the moment the app stops matching it
Scripted plus self-healing	The same code and selectors, plus fallback attribute data the tool maintains for repairs	The selectors still go stale, and each automated patch joins a queue for human review
TestAutomate, selectorless	An id, an intent-stating prompt, an expected outcome, optional setup and cleanup prompts	Only the intent itself, when the product’s actual purpose changes

Two honest caveats before that table looks too tidy. An intent-based artifact is not automatically a better test. A sloppy prompt with a lazy expected outcome is exactly as much a liability as a sloppy script, and it can be harder to catch in review because it reads like reasonable prose. And scripted artifacts hold real virtues the table doesn’t show. A script is deterministic in a way an interpreted intent is not, clicking the same element in the same order every run, while an agent may take a slightly different path to the same outcome. A team that needs step-for-step reproducibility more than it needs resilience is giving something up here, and pretending otherwise would be marketing.

Is selectorless testing the same as self-healing?

No, and the shortest honest version is that self-healing fixes selectors; selectorless removes them. A self-healing tool keeps the author-written CSS or XPath and, when the string stops resolving, uses heuristics like fallback attributes and nearby text to relocate the element and patch the stored locator. The artifact still contains selectors, now joined by the fallback data that repairs them, so the test still depends on a selector, just one that gets patched instead of rewritten by hand. A selectorless artifact contains nothing to patch. There is no repair step because there is no stored thing to repair.

That single paragraph is deliberately the whole treatment here. The practitioner’s comparison, where the maintenance actually goes, when a healing layer earns its keep, what a wrong heal costs, and how to interrogate both pitches, gets its own full post in self-healing vs selectorless testing, written by someone who carried the locator repair queue for years.

What are the limits of no-selector testing?

Selectorless removes one failure class, stale locator strings, and leaves every other one standing. Anyone selling it as unbreakable is overreaching, so let me draw the boundary from the inside.

Start with what the term does not mean. It doesn’t mean vision magic. The claim worth trusting is narrower and more checkable. No human writes or stores selectors anywhere in the test, and the agent finds elements by reading the live page the way a person would. Test automation without selectors also doesn’t mean the agent can find what isn’t there. A control hidden behind an unopened menu, a component that never rendered, or a description so ambiguous that two elements plausibly match can still leave the agent with nothing safe to act on, and the run fails. That last one is a feature, not a gap. A control that vanished is exactly what you want a red result for.

The limit that’s genuinely yours to own is the expected outcome. Intent-based testing moves your rigor from locators to specifications, and a vague specification becomes the new weak link. If the expected outcome would be satisfied by nearly any run, the test can’t lose, and a test that can’t lose verifies nothing. I keep a separate checklist for writing expected results that hold up under that pressure, and the regression-suite writeup linked above records a real case where a generated test’s weak point was its own too-loose expected outcome rather than anything about finding elements. The failure surface doesn’t disappear in a selectorless world. It moves up a level, from string matching to intent matching, and your review effort should follow it there.

And because the finder adapts, the grading can’t be left to the finder. We designed the run so the agent that drives the browser never grades its own work. A separate verifier reads the recorded run against the expected outcome under a standing instruction to stay skeptical, treats any success claim without evidence as an expectation not met, and returns every expectation itemized as met or missing rather than a bare pass or fail. Each run lands in exactly one of four outcomes, passed, failed, skipped, or blocked, where blocked means the preconditions never held, an environment problem rather than evidence against your app. A failed first attempt is retried once on a stronger model before the failure is treated as real. Strictly speaking, none of that machinery is part of the definition of selectorless testing. All of it is what makes the definition safe to run, because an adaptive finder without a skeptical judge is just a very confident guesser.

The decision rule I’d hand an evaluating team is short. Pull up your last month of red runs and check what preceded them. If the reds cluster after frontend deploys and the fixes were locator edits, your failures are selector brittleness, and removing the selector removes the class. If your UI barely moves and a scripted suite is humming along, the rewrite may not pay for itself yet. Either way, hold any selectorless pitch to the artifact test from earlier. Ask to see a stored test and check whether a selector could live in it. For how these three fields become a graded verdict end to end, the agentic testing guide maps the full loop, and the how-it-works walkthrough shows the same flow on the product page itself.

Generate Test Cases From Documentation

Thu, 30 Jul 2026 00:00:00 GMT

I wanted to see how much of a regression suite I could get without writing a single test by hand, so I pointed TestAutomate at linear.app/docs and clicked ingest. That recorded run is the spine of this post, because the honest way to find out whether you can generate test cases from documentation is to watch one real attempt end to end, including the part where a drafted test had to be fixed.

You can generate test cases from documentation when two conditions hold. The docs describe discrete tasks a user performs in the UI, and something downstream turns each task into an intent-based test with an expected outcome a verifier can grade. TestAutomate does its AI test case generation in two stages, an ingestion pipeline that crawls a docs site and distills task-shaped skills, and a planner that clusters those skills into topics and drafts runnable test flows from them.

What follows is the mechanism behind each stage, the Linear run as the worked example, and the two places where a human reviewer still earns their keep.

Can you really generate test cases from documentation?

Yes, if the documentation actually describes tasks, and only up to a draft that deserves review. Testers have a name for whatever artifact tests are derived from, the test basis, and most teams reach for a requirements doc or a stack of user stories by reflex. I built TestAutomate’s ingestion around a different bet. Published product documentation is the one artifact that describes what actually shipped, in the vocabulary users see on screen, and it stays maintained because users depend on it.

The pipeline behind that bet runs in four stages: crawl, extract, classify, emit.

The crawl is deterministic, with no model involved. It fetches the site’s pages at a polite pace, stays inside the documentation section you point it at, and caches what it fetches, so re-running ingestion over unchanged pages downloads nothing twice. The whole walk is bounded, a run can’t wander off and crawl forever. Pages too thin to carry a real procedure, nav indexes and redirect stubs, get skipped before any model sees them, and very long pages are truncated rather than swallowed whole.

Extraction is where the reading happens. A model works through one page at a time and pulls out three things. Intents, the user-facing actions the page describes. Procedures, the numbered steps behind each intent, with the doc’s button and field names preserved rather than paraphrased. And entities, the app’s domain objects and how they relate to each other. The extraction step is deliberately conservative, built to come back empty-handed rather than guess, and strict about what counts as a task. A discrete action a user takes in the UI qualifies. Concept overviews, marketing material, API reference, and release notes don’t.

A classification pass then sorts the extracted material by the kind of handling it needs at runtime, and the emit stage writes each procedure out as a skill, one distilled task per skill, with its source page recorded. Every stage is independently re-runnable and cached, and one page failing mid-run gets logged and skipped instead of killing the ingestion. The design goal I held onto while building it was that a successful crawl should always produce something useful.

If you’re evaluating tools that generate test cases from documentation, AI extraction quality is where I’d aim your skepticism first. Ask what the tool refuses to extract. A pipeline that turns a pricing page into a test case isn’t being thorough, it’s being credulous.

What did the crawl keep from Linear’s docs?

Task pages, and very little else. I picked Linear because its docs are clean, an actual <article> tag around every page, none of the marketing-chrome-around-content mess you get on some doc sites. I figured if the ingestion pipeline was going to fall over anywhere, it wouldn’t be here. It didn’t. Within a few minutes I had a set of extracted skills, each one a distilled procedure: “create an issue,” “assign a project lead,” “set a cycle deadline.”

What surprised me was watching what it skipped. There’s a whole page in Linear’s docs about their API reference, and none of it showed up as a skill. Part of that is the crawl itself, whose source configuration keeps it on the end-user docs and excludes changelog and marketing paths outright. The rest is the extraction step, which throws out anything that isn’t a discrete UI action. Marketing copy, changelog entries, conceptual overviews with no steps, all filtered before a model ever gets asked to turn them into a procedure. That’s the right call. I didn’t want a test called “learn about Linear’s philosophy.”

Each surviving page went through the extraction pass with the actual button and field text preserved, plus the domain entities the page mentions, Project, Issue, Cycle, and how they relate, followed by the classification pass that sorts each intent by what kind of handling it needs at runtime.

The detail worth pausing on is that nothing in this stage invents anything. Extraction is a compression step, not a creative one. If a page describes no steps, no steps come out of it, and a page can legitimately contribute zero skills. That restraint matters two stages later, because a test drafted from invented steps doesn’t fail loudly. It fails by quietly testing something your product never claimed to do.

How do extracted skills become an executable regression suite?

In two stages you can watch from the dashboard, clustering and drafting, and none of it runs until you ask. Generation is triggered lazily, the first time you request a suite for an app that doesn’t have one, rather than as a side effect of ingestion. Progress streams to the dashboard as it works, so you see topics forming and tests landing while it happens.

Then came the part I actually cared about, turning ~40 skills into a runnable suite. I clicked “Generate suite” and watched it cluster the skills into topics. It grouped things like “Cycles & Planning” and “Issue Triage” on its own, without me defining categories anywhere. For each topic it drafted a handful of test flows, each with a prompt carrying the natural-language instruction and an expected_outcome that a separate verifier checks the run against afterward.

A drafted test is structurally identical to one you’d write by hand. It carries the same three required fields, an id, a prompt, and an expected outcome, plus optional setup and cleanup prompts, and the runner refuses to load any test missing its prompt or expected outcome. The drafts are plain natural language by hard rule. The generation contract bars code, JSON, YAML fragments, and selectors from every field, so what you review reads like instructions to a careful colleague, not a script. Expected outcomes come out as itemized end-state assertions, including negative ones such as confirming nothing else was modified, and any bullet prefixed [incidental] is barred from failing the run, so a missing success toast can’t outrank the point of the test.

One thing it got right without me asking was silently dropping every skill that was purely about logging in or signing up. Makes sense, since the runner assumes you’re already authenticated in your bound browser profile, so a generated test that tries to log in would just be broken by design. It didn’t drop SSO setup, though, because that’s an admin configuring something rather than a user authenticating. It’s a distinction I wouldn’t have thought to encode into a filter, but it’s exactly right.

The strictest rule in generation is self-containment. No draft may assume a resource was created in a prior step, because tests run independently and in any order. Anything that must already exist goes into an idempotent setup prompt that checks for the resource and creates a minimal one if it’s missing. And because a model can slip past its own instructions, a deterministic lint pass runs over every draft and drops flows that assume prior-step data or hedge about resources nothing guarantees. I wrote that lint after watching what hidden data dependencies do to chained tests. A bad test is worse than a missing one.

Executability is the other half of the phrase “executable regression suite,” and it’s the half most docs-to-test-cases tooling quietly drops. What lands here isn’t a spreadsheet of test titles. Each draft joins the same list as your hand-written tests and runs through the same machinery. An agent drives a real browser, a separate skeptical verifier grades the run against the expected outcome item by item, every run ends passed, failed, skipped, or blocked, and a failed first attempt is retried once on a stronger model before the red is treated as real. That’s the full agentic testing loop, and generated tests get no special leniency inside it. AI regression testing earns trust at the verdict, not at the drafting step.

Which source of truth should you generate tests from?

The one that’s still true on the day the suite runs. A regression suite exists to re-verify shipped behavior after every change, so the artifact you generate it from should describe shipped behavior, not intended behavior. That’s the lens I’d use on the four sources teams actually reach for.

Source of truth	How stale it gets	Coverage of shipped features	Executability of what you get
Requirements doc	Frozen at sign-off, drifts the moment scope changes mid-build	Describes what was promised, including features that never shipped	Test ideas and titles a person still has to turn into runnable steps
User stories	Each is accurate when closed, then nobody updates it	Fragmented, a full user journey rarely lives in one story	Acceptance criteria per story, seldom an end-to-end flow
Screen recordings	Accurate on the day of capture, decays with every UI change	Only the paths someone remembered to record	Replayable, but bound to the exact clicks and layout recorded
Published product docs	Maintained while users depend on them, updated as features change	Describes what shipped, task by task, in the UI’s own vocabulary	With TestAutomate, intent-based tests the runner executes directly

Honest caveats, because that table flatters my choice. Requirements docs and user stories are better than product docs at capturing intent, what a feature is for and what must never happen, and the trade-offs between those inputs get a full treatment in generating test cases from requirements with AI. Docs lag releases at plenty of companies, and a new feature can be live for a sprint before its page exists. Internal tools often have no public docs at all, in which case this whole approach has nothing to eat. Recordings deserve their due too. For a stable flow that never changes, a recording is an honest, if brittle, source. My claim isn’t that docs are perfect. It’s that among the four, they’re the only source that is simultaneously maintained, written in the user’s vocabulary, and specific enough to execute from.

Where does generating tests from docs fall short?

On assertion strictness, and on anything your documentation doesn’t say. The recorded run produced a clean example of the first. One of the drafted tests for “Cycles” had an expected_outcome that was too loose, something like “the cycle is created successfully,” and it passed on a run where the cycle got created with the wrong date range. That’s a generated-test problem, not a bug in Linear, and it’s exactly the kind of thing the schema expects a human to tighten before trusting it in a real suite. I rewrote the expected outcome to name the actual date range and re-ran it, and it caught the same issue immediately.

That incident is why I treat generation as a drafting step, never a publishing step. The review I’d actually do on a generated suite is narrow. Read every expected outcome and ask whether a wrong-but-plausible result could still satisfy it, then tighten the ones where it could. Check that the setup prompts guarantee whatever the prompts assume. Cut the flows you know don’t matter, applying the same editorial instinct behind deciding which generated test ideas deserve to survive. Writing assertions that can actually lose is its own craft, and there’s a separate checklist for expected results that hold up on this blog.

The second limit has no workaround inside the product. Coverage is bounded by what’s documented. An undocumented admin corner produces no skills, so it produces no tests, and no generation pass can know that your revenue depends on a flow your docs never mention. The suite you get is a floor, the documented core of the product, executable on day one. Someone who knows the product still decides what matters above that floor.

It’s not a hands-off “regression suite in one click,” but it is a genuinely useful first draft. It turned documentation I would’ve had to read anyway into 15 minutes of review-and-edit instead of a day of writing test prompts from scratch. Docs to test cases is a real pipeline now, not a metaphor. Just remember that the draft’s confidence and the draft’s correctness are two different things, and a skeptical review pass is what turns the first into the second.

TestAutomate Blog

Automated UAT Testing: What It Really Takes

What is automated UAT testing?

Why does UAT resist automation?

What does automation-ready UAT look like?

How does sprint UAT run while you sleep?

How do you adopt UAT automation without a rewrite?

Real Browser Testing vs Cloud Browsers

What does real browser testing actually mean?

Where does automated browser testing actually run?

Why does the execution environment change test results?

How does TestAutomate run tests in a real browser?

When is cloud browser testing the right tool?

Best Agentic QA Tools in 2026: The New Category, Mapped

What separates an agentic tool from an AI-assisted one?

Who’s writing this list, and how the grading works

The seven agentic testing tools at a glance

1. TestAutomate (ours, pre-launch)

2. Momentic

3. TestSprite

4. Autonoma

5. Checksum

6. TestMu AI (KaneAI)

7. QA Wolf

How to choose between agentic QA tools

The Best AI Testing Tools in 2026, Compared Honestly

How we graded the best AI testing tools

AI testing tools compared at a glance

The 10 best AI testing tools in 2026

1. TestAutomate (that’s us)

2. testRigor

3. mabl

4. TestMu AI (KaneAI)

5. Testim

6. Katalon

7. Testsigma

8. Functionize

9. Autify

10. Rainforest QA

How do you choose between AI testing tools?

Best Test Case Generation Tools in 2026

How we graded every tool here, ourselves included

Eight test case generation tools at a glance

Where do test case generation tools get their test cases from?

The best test case generation tools in 2026

TestAutomate: generation from docs, repo, and the running app

testRigor: plain English across the widest platform surface

Functionize: ML element models at enterprise scale

Katalon StudioAssist: AI assist inside the broadest IDE

Testsigma: templated English with an open-source core

Autonoma: tests generated from your codebase

Checksum: tests generated from code and recorded user traffic

TestMu AI: KaneAI on the biggest device cloud

How do you choose between AI test case generation tools?

AI Testing Cost: Engineering the Bill Down

What drives AI testing cost in an agent run?

Which levers actually cut LLM testing costs?

Match the model tier to the step’s difficulty

Cache the prompt prefix, and then defend it

Spend premium certainty only where it changes a verdict

Make the second run cheaper than the first

Why does credit-based pricing hide the real cost?

What cost visibility should a testing tool give you?

What should you ask any AI testing vendor about cost?

Automated Testing Behind Login: SSO and MFA

Why automated testing behind login is where suites die

What does each kind of login do to a scripted test?

How do teams work around login walls today?

Why can’t cloud grids test staging behind a VPN?

How does TestAutomate handle authenticated session testing?

What still needs a human at the login page?

How a Test Run Gets Graded, Step by Step

What has to be true before the browser opens?

What does the act phase actually record?

How does a recorded trajectory become a verdict?

What happens when the verdict comes back failed?

Where does a graded run end up?

Testing in Plain English: How It Works

What does testing in plain English actually mean?

Is plain-English testing the same as Gherkin?