Writing Expected Results for Automated Tests

Expected results for automated tests should describe the end state a run must leave behind, written as a short list of separately checkable assertions. Name the records that should exist and the values they should hold, state what must not have changed, and split what must hold from what’s merely nice to see. Skip that discipline and the strongest runner in the world ends up grading your product against a shrug.

That definition matters more in agentic testing than it ever did in scripted testing, because the expected result is no longer an assertion in code. It’s a piece of prose that a model reads, interprets, and judges a real browser run against, which is where AI test automation relocates the trust question. In TestAutomate every test is three required fields, an id, a prompt, and an expected outcome, and the runner refuses to load a test that’s missing any of them. I built it that way because the third field is where suite quality actually lives, and the craft of writing that field well is what the rest of this piece works through.

What do strong expected results for automated tests look like?

They read like a checklist a skeptical stranger could grade without asking you anything. Here’s the kind of weak outcome our generation rules exist to stamp out:

expected_outcome: |
  - The new cycle is created successfully
  - Everything works as expected

And here’s the same test with an outcome that can actually lose:

expected_outcome: |
  - A cycle named 'qa-regression-cycle-01' appears in the cycles list
  - Its status reads Active and its start date is the date entered
  - No other cycles were modified or deleted
  - [incidental] A confirmation toast appears after saving

The difference isn’t length. It’s that every line in the second version names an observable artifact with a checkable property. A verifier reading the recorded run can find the cycles list, look for the named row, read its status, and confirm nothing else moved. The first version gives it nothing to look for, which means the verdict floats free of the product. Testing theory has a name for the underlying difficulty, the oracle problem. Running the test was never the hard part. Deciding what “correct” looks like is, and a vague expected result is a decision you declined to make.

Our generation rules hold drafted tests to the same bar. An outcome that only says the page loaded gets rejected, while one that commits to a concrete rendered end state, say a named table showing at least one row, survives, because that’s something the run’s evidence can confirm or deny. When I review hand-written outcomes I apply the exact same test to each bullet, could a stranger check this against a recording, and rewrite any line that fails it.

How do you split core expectations from incidental ones?

By deciding, per bullet, whether its absence should fail the run. In TestAutomate every bullet in an expected outcome is a core assertion unless you prefix it with [incidental], and the semantics are strict in both directions. A missing core expectation always fails the test. An incidental miss can never fail it, no matter what else happened. The verifier applies exactly that split when it grades, and the verdict comes back itemized, every expectation met and every one missing, so you can see which bullet did the work.

We made unprefixed mean core on purpose. The dangerous default is the opposite one, where assertions are lenient unless you remember to harden them. Here, forgetting the prefix makes a test stricter, not looser, and the failure mode of that mistake is a red run you loosen after reading, not a green run that lied to you.

What earns the prefix is anything cosmetic riding along with the real outcome. A success toast appearing. The exact wording of a confirmation message. A count badge updating in the corner. Those are worth recording because their absence is sometimes an early smell, but none of them is the reason the test exists, and a suite that fails on toast wording trains people to stop reading red. The load-bearing lines, the record exists, the values are right, nothing else changed, stay unprefixed. Use the prefix sparingly. If most of your bullets are incidental, the test doesn’t know what it’s for.

This split is also what keeps a judge-graded test honest about severity. Without it, every bullet carries equal weight and the verifier has to guess which misses matter. With it, you’ve told the judge in advance where the line is, and the judge’s discretion is spent on evidence instead of on importance.

Why do negative assertions belong in the checklist?

Because an agent that can do anything on a real page can also do things you didn’t ask for, and a checklist that only describes the happy path will never notice. Our generation rules require expected outcomes to be verifiable end-state assertions including negatives, lines like “no other records were modified”. I’d give the same advice to anyone writing outcomes by hand, whatever their tooling.

The reasoning is mechanical. A verifier grades what you wrote down. If your bullets only assert that the new record exists, a run that also deleted three unrelated records satisfies every bullet and passes. The deletion isn’t hidden, it’s right there in the recorded run, but nothing in the checklist makes it load-bearing. One negative line converts that silent damage into a missing core expectation.

TestAutomate’s verdicts do carry a separate list of extras, actions the agent took that weren’t requested, so surprising behavior surfaces even when your bullets missed it. But extras are disclosure, not enforcement. The pass still hangs on your expectations alone, which is exactly why the expectations should state what must not have changed. My working rule is one negative assertion per mutation the flow performs. A test that creates something asserts no duplicates appeared. A test that edits one record asserts the neighbors kept their values. A test that deletes asserts only the target vanished. Cheap to write, and it’s the line that catches the expensive class of bug.

How do different assertion styles compare?

Each style is really a decision about where correctness lives, in pixels, in strings, in a person’s head, or in an itemized prose contract. Most best practices for writing test assertions were written for the middle two and carry over further than people expect. The table compares how each style responds to the two events that matter most, a real regression where wrong data got saved, and a cosmetic change where a toast got reworded.

The honest caveats. Snapshot comparison is genuinely the right tool for pure rendering regressions, and nothing prose-based competes with it there. Exact-value asserts are fast, deterministic, and precise, and a well-maintained scripted suite full of them is a strong position, with the cost showing up as maintenance rather than as weak verdicts. Human judgment remains the best oracle for intent that nobody wrote down anywhere. And the itemized style’s strength is entirely downstream of the writing. A lazy checklist graded by a careful judge is still a lazy checklist, which is why this whole piece is about the writing.

How do acceptance criteria become checkable expectations?

By translation, and the translation is where most of the value gets added. Acceptance criteria for automated tests usually arrive written for humans, “the user can filter the report by date”, which is a statement about capability, not about any particular run. An expected outcome is a statement about one run. The given-when-then frame does the heavy lifting. The given belongs in a setup phase that guarantees preconditions, the when belongs in the prompt, and the then becomes your bullets, pinned to the concrete values this run should produce. “Can filter by date” becomes “after filtering, the table shows only rows dated within the range entered, and the row count shown matches the rows listed”.

Two consistency rules from our generation design are worth stealing for hand-written tests. First, the expected outcome must be satisfiable by exactly what the setup and prompt do, no more and no less. Second, never require a specific named resource unless something in the test guarantees that exact resource exists. The named-resource trap is the subtle one. An outcome demanding ‘regression-cycle-001’ when the prompt only conditionally creates it produces a test that fails for environment reasons, and environment problems should never wear a failure’s clothes. When a test does need pre-existing data, the setup phase guarantees it, and the verifier is explicitly told what state setup established so it never penalizes the run for a resource it didn’t create.

Generated suites get these rules enforced rather than suggested. When TestAutomate drafts a regression suite from your docs, a lint pass drops flows that assume data from a prior step, hedge with a fallback no setup backs up, or demand a named resource nothing guarantees, on the principle that a bad test is worse than a missing one. Hidden data dependencies between tests are the same disease in chained form. Whether a human or a model wrote the outcome, the acceptance question is identical. Does anything in this test guarantee the state this sentence assumes?

What happens to your expected results at verdict time?

They become the entire contract. Our runner hands a separate verifier model two things, your expected outcome and the trajectory of everything the agent actually did, and instructs it to judge only on evidence. The instruction is deliberately skeptical. If the trajectory doesn’t contain clear evidence for an expectation, that expectation is not met, and the agent’s own claim of success counts for nothing on its own. When the judge is in doubt about a core expectation, it’s told to mark it missing, because a false pass on a broken behavior is the worst outcome the system can produce.

A real graded run from our recorded QA environment shows a specific expectation doing its job. The test required work on a cycle named ‘360-QA-Regression-2026’, the agent operated on ‘360-Review-Fiscal-Year-2026’ instead, and the run failed precisely because the expected outcome had named its target. A vaguer outcome, “the participants are added to the cycle”, would have passed that run. The named expectation turned a wrong-target run into a red result with a reason attached, and that verdict gets a full walkthrough in the judge piece.

Two mechanisms keep this strictness from curdling into noise. When the app behaves differently from what your outcome implied but the difference still satisfies the intent, a renamed button, an extra confirmation step, the verifier doesn’t fail the run. It records an assumption flag with what was observed, what was expected, and why the change looks deliberate. A flagged pass is still a pass, and a flag is never allowed to excuse a genuinely unmet core expectation. And when a run does fail on the first attempt, the act phase is re-run once on a stronger model before the failure is treated as real, so agent fumbles don’t get billed to your product. Every run still lands in exactly one of four outcomes, passed, failed, skipped, or blocked, and the full loop from generation to verdict inherits its signal quality from the one prose field this piece is about.

So that’s where I’d spend your review time. When a generated suite lands, the prompt gets a skim. The expected outcome gets the scrutiny a code reviewer gives an assertion block, because that’s exactly what it is, an assertion block that happens to be written in prose and enforced by a judge that believes evidence over enthusiasm.