Testing in Plain English: How It Works

When we built TestAutomate’s authoring layer, I held one line I wouldn’t trade away. A stored test would have nowhere for code to live. No selector slot, no script block, no step-definition hook, nothing. Every test would be a few English sentences, and those sentences would be the thing that runs. The product page calls that plain-English authoring. Searchers call it testing in plain English. What follows is my attempt to pin the phrase down properly, because people use it to mean at least three different things, and only one of them survives contact with a changing UI.

Testing in plain English means the English is the executable test itself, not a readable label on top of code. You write what a user does and what must be true afterward, the way you’d brief a careful teammate. At run time an AI agent performs that intent in a real browser, and a separate model grades what happened against your written expectations. No selectors, scripts, or step definitions exist underneath the words.

That definition draws some hard boundaries, so let me walk them one at a time.

What does testing in plain English actually mean?

Three different pitches wear the phrase, and they behave nothing alike, so the first job is telling them apart. The weakest sense is cosmetic. A framework gives tests readable names and prints friendly reports, and every assertion underneath is still code. The middle sense is translation. You type an English sentence and the tool generates a script from it once, selectors included, and from that moment on you maintain the script, not the sentence. The strongest sense, the one worth searching for, is execution. The English is the stored artifact, and something interprets it fresh against the live app every time the test runs.

TestAutomate sits in the third camp, and the honest disclosure is that I built it, so hold my definition to the same scrutiny I’m applying to everyone else’s.

The probe I’d use on any plain-English testing tool is simple. Take a stored test, change one English word, and ask what has to happen before the next run reflects the change. If a human or a generator must touch code first, the English is documentation with good posture. If the answer is nothing, because the runner reads the English itself, you’re looking at natural language test automation in the strict sense. A second probe works from the other side. Ask to see the stored artifact and check whether a selector could live anywhere in it. I proposed that check in the piece on test automation without selectors, and it transfers to this vocabulary unchanged, because intent-based testing and selectorless testing are two names for the same structural bet.

Is plain-English testing the same as Gherkin?

No, and the difference sits exactly one layer down from the surface. Gherkin, the language behind Cucumber, has made tests read as Given, When, Then sentences for nearly two decades, and those sentences are genuinely plain English. But each line works by matching a step definition, a function somebody on the team wrote in a programming language, and the step definition is what actually drives the browser. Cucumber’s own reference is straightforward about the mechanism. The English selects code. It doesn’t replace it.

That design decides who can really write tests. The vocabulary is closed. If no step definition exists for “When I archive the oldest project”, the line can’t execute until an engineer implements one, so the set of sentences a non-programmer can compose is bounded by the glue code the team has accumulated. And the glue code inherits every maintenance burden the plain English was supposed to remove. The selectors live inside the step definitions, and they go stale there just as fast as they would anywhere else.

Plain-English execution removes the matching layer entirely. There is no fixed vocabulary, because there are no step definitions to match against. Any instruction you could hand a competent teammate is a legal sentence, and the agent works out the mechanics against the page it actually sees on that run.

I want to be fair to Gherkin on the way past, because its specification discipline holds up. Martin Fowler’s description of Given-When-Then as a way to separate precondition, action, and check is exactly how a good prompt and expected outcome divide their labor in an agentic test. The grammar survives. What goes away is the requirement that every sentence be pre-implemented before it means anything.

What does a plain-English test actually store?

Three required fields, all natural language. An id names the test. A prompt describes what to do, end to end. An expected outcome itemizes what must be true afterward for the run to count. A test can also carry optional setup and cleanup prompts written the same way, for establishing preconditions before the run and removing what the run created. The runner refuses to load a test that’s missing any of the three required fields. The optional fields around them are housekeeping, a name, tags, a timeout, and nothing in the stored test is ever executed as code or resolved as a selector.

Here’s a complete test in that shape:

• id: reviews.cycle.create
• prompt: sign in as the demo HR admin, create a review cycle named 360-QA-Regression-2026, and add one participant from the employee list
• expected outcome: the cycles list shows 360-QA-Regression-2026 with exactly one participant, no other cycles were modified, and [incidental] a confirmation message appears

The bracketed prefix does real work. Every unprefixed bullet is a core assertion that must hold for the run to pass. A bullet marked [incidental] is a nice-to-have, and a missing incidental can never fail the test, which stops a cosmetic toast from outvoting the behavior the test exists to check.

The plain-language rule binds machines as well as people. When TestAutomate drafts tests itself, the generation rules state it verbatim: “Do NOT include code, JSON, YAML, or selectors.” A generated test meets the same bar as a hand-written one, sentences only.

Notice what the artifact never says. Nothing about where the create button sits, what the form’s fields are called in markup, or how the participant picker is implemented. Finding all of that is the agent’s job at run time, on every run. The product page on plain-English authoring makes the pitch for why this is a pleasant way to write tests, so I’ll leave that side to it. What matters for the definition is narrower. The English above is not a description of the test. It is the test.

What happens when the UI changes?

Nothing in the test goes stale, because nothing in the test referred to the old UI in the first place. This is the property most people are actually shopping for when they search for plain-English testing tools, so it deserves precision rather than a slogan. There are three cases, and they end differently.

First, the change is cosmetic. A button gets renamed, a form moves into a modal, a redesign reshuffles the layout. This is why automated UI tests keep breaking after frontend deploys that changed no behavior at all, since a scripted suite fails the moment its stored strings stop matching the markup. A plain-English test carries no stored strings. The agent reads the live page on this run and finds the control as it exists now, so the rename is absorbed without anyone editing anything.

Second, the change is real but compatible. The flow still satisfies the written intent through different behavior, say an extra confirmation step before the record is created. When a run succeeds through behavior that differs from what the test described, the verifier can attach an assumption flag to the pass, recording what it observed, what was expected, and the assumption it made, for a human to review. A flagged run is still a pass, and a flag never excuses an unmet core expectation.

Third, the change breaks the behavior. The record never appears in the list, and the test fails. That’s the outcome you wanted, because the red is now about your app instead of your locators.

One boundary is worth drawing plainly here. This is not self-healing. A self-healing tool keeps stored selectors and patches them when they stop resolving, so the artifact still contains the thing that breaks. A plain-English test has nothing to patch. The practitioner’s comparison between the two approaches gets its own post, written by someone who carried a locator repair queue for years.

How does an English sentence become a pass or a fail?

Through two deliberately separated jobs, acting and judging. On the acting side, the agent drives a real browser, and specifically your own Chrome, through an extension that binds to a tab. It signs in, clicks, and types the way a person at the keyboard would, reading the live page to find each control, against your real app rather than a cloud replica of it. Everything the agent does, and everything the page did in response, is recorded as the run’s trajectory.

On the judging side, the agent that drove the browser never grades its own work. A separate verifier model reads the recorded trajectory against the expected outcome and judges each written expectation strictly on evidence. It’s skeptical by instruction. A success the trajectory can’t substantiate counts as an expectation not met, however confidently the agent declared victory. The verdict comes back itemized, each expectation met or missing, and every run lands in exactly one of four outcomes, passed, failed, skipped, or blocked. Blocked means the test’s preconditions never held, which is an environment problem rather than evidence against your app, and it’s counted as neither pass nor failure.

Strictness needs a check on itself, so a judged failure isn’t final on the first attempt. The runner re-runs the failed attempt once on a stronger model, and the second verdict stands. A pass on retry is recorded as evidence the first failure was agent error. A second failure is recorded as high confidence the issue is real.

Here’s a real graded run from our QA environment, on an HCM test app:

That test required adding participants to one specific cycle, 360-QA-Regression-2026. The agent operated on a similarly named cycle instead, 360-Review-Fiscal-Year-2026, and the verifier failed the run with reasoning that names exactly that mismatch. That failure is the discipline plain English needs before you can trust it. The sentence allowed a wrong-but-plausible reading, and the grader caught it because the expected outcome wasn’t loose.

A failure that survives all of that can become a bug candidate, carrying the verifier’s reasoning as context. Nothing files itself. A candidate reaches Jira only after it’s checked against existing issues for duplicates and a person reviews it and clicks to file. The verdicts page walks this grading pipeline end to end.

Can you dictate a test by voice?

Yes, and it turned out less gimmicky than I expected when we wired it up. The mic opens a live voice session, you talk, and the prompt box fills in as you speak. Editing works by voice too, in the register you’d actually use, telling it to replace one phrase with another or to scratch the last sentence, and the draft updates live. The fastest way to spec a flow is often to walk through the app while narrating what you’re doing, and dictation matches how that knowledge actually arrives.

Two honest notes on the plumbing, because billing claims deserve exactness. Voice capture is the one part of the product that doesn’t run on Anthropic models. The live session runs on OpenAI’s Realtime API, billed to your own OpenAI key, and the handshake is proxied through your local server so the key never reaches the browser extension. The spoken edits themselves are applied to your draft by Claude, on the same bring-your-own-key terms as everything else. The Claude calls that drive and grade tests bill your own Anthropic account directly, with no markup and no per-seat fee, and the server itself is self-hosted, running on your machine against your own database. The economics page lays out that cost structure in full.

What doesn’t plain English fix?

Ambiguity, lazy specifications, and nondeterminism, for a start, and pretending otherwise is how the phrase gets a bad name.

Ambiguity first. A prompt that would confuse a new teammate will confuse an agent, and the wrong-cycle failure above is what that looks like in practice. Plain English moves your rigor, it doesn’t retire it. The effort that used to go into locator hygiene now goes into prompts that name their targets precisely and expected outcomes that could actually fail. A test whose expected outcome would be satisfied by nearly any run can’t lose, and a test that can’t lose verifies nothing. I keep a separate checklist for writing expected results that hold up, and it matters more in this world, not less.

Determinism is a real tradeoff too. A script clicks the same element in the same order on every run. An agent interpreting intent may take a slightly different path to the same outcome, which is precisely what makes it resilient to UI change and precisely what a team needing step-for-step reproducibility should weigh honestly before switching.

The layer matters as well. Plain-English tests don’t replace unit tests, which are fast, cheap, and belong in CI on every commit. This approach earns its cost at the end-to-end regression layer, the multi-step flows through a real UI that are the most expensive tests to write by hand and the first ones abandoned when a suite starts rotting.

So, my decision rule, compressed. If your reds cluster after frontend deploys and the fixes are locator edits, the English-as-artifact bet removes that entire failure class. If you need bit-identical replay of exact steps, keep scripts on those paths. Either way, hold every tool claiming testing in plain English to the two probes from the top. Change one word and see what must be regenerated. Ask where a selector could hide. The phrase means something only when the answers are nothing and nowhere.