How a Test Run Gets Graded, Step by Step

The run I keep going back to in our QA environment is a failure. Run 33 on our recorded timeline drove two tests against an HCM test app, took 12 minutes and 10 seconds, failed both, and the verdict on one of them names the exact wrong turn. The agent operated on a review cycle called 360-Review-Fiscal-Year-2026 when the test required 360-QA-Regression-2026. Nobody re-watched the session to learn that. The grading pipeline produced it, and this post walks through that pipeline one stage at a time, because how TestAutomate grades every run is easier to trust once you’ve followed a single run all the way through it.

The short version fits in a paragraph. Every run moves through the same fixed sequence. A setup phase must declare the test’s preconditions ready or the run records as blocked. An agent then drives the flow in a real browser while every action and result is recorded as a trajectory. A separate verifier model reads that trajectory against the expected outcome and issues one of four verdicts, passed, failed, skipped, or blocked, and a judged failure earns one retry on a stronger model before the grade stands.

I’ve written separately about why the judge must be a separate model, and our verdicts page makes the case for the philosophy. This is the operational companion to both. Not why the pipeline is shaped this way, but what actually happens to your run inside it.

What has to be true before the browser opens?

The runner won’t start a run it can’t grade. A test here is three required plain-language fields, an id, a prompt saying what to do, and an expected outcome saying what must be true afterward, and a test missing its prompt or expected outcome doesn’t load at all. That gate exists for the verifier’s sake. The expected outcome is the checklist the whole pipeline will eventually be graded against, so a run without one would be motion with no way to score it. Most of the craft in writing expected results for automated tests is really about making this stage strong enough to carry the rest.

Two exits can end a run before the browser does anything interesting. The first is a skip. If you run a suite in already-logged-in mode and a test’s whole purpose is to exercise the login flow, the two are mutually exclusive, so the runner records the test as skipped instantly, with a plain-words reason and neither a pass nor a failure. On our recorded timeline that costs literally nothing. Run 31 shows one skipped test in a run rendered at zero seconds.

The second early exit is the setup gate. A test can carry an optional setup prompt whose only job is to establish preconditions, idempotently, inside its own fixed time budget. Setup must end by declaring exactly one of two outcomes in a form the runner reads mechanically, either ready, with a one-line description of the precondition state that now exists, or blocked, with the reason it couldn’t be established. Anything short of ready and the run is recorded as blocked on the spot. The test’s own steps never execute, no verdict about the application is issued, and the record says so explicitly.

That’s run 36 from the same dataset, blocked in 1 minute and 3 seconds because the precondition proved unreachable. Under the hood a blocked run is recorded as neither a pass nor a failure and tagged as a precondition problem rather than a test result, which is what keeps it out of failure counts everywhere downstream. I’ve argued the full case for why tests get blocked instead of failed elsewhere. Operationally, the thing to know is that a blocked run stops early, runs cleanup to revert any partial state, and never reaches the verifier at all.

What does the act phase actually record?

Once setup declares ready, the act phase begins, and this is where the evidence gets made. A browser-driving model receives the test’s prompt plus the one-line description of what setup established, and it operates your real application in your own Chrome. That’s not a simulation layer or a stripped-down clone of a browser. It’s the same Chrome you use, carrying your real logged-in sessions, which is why the run sees your app the way your users’ browsers do. The execution page covers that architecture in more depth.

What matters for grading is that every step becomes a record. Each turn, the agent picks a browser action, the action executes, and the result comes back and is appended to the trajectory. After every navigation or click, the agent also reads the page that resulted, the way a person would glance up to check their work, and that observation lands in the record too, so the trajectory fills up with written observations instead of image spam. Screenshots still happen when visual state is genuinely the question, and they live in the trajectory too.

The act phase is bounded, not open-ended. It runs inside its own fixed time budget, and the agent loop carries a hard cap on turns so a confused agent can’t wander forever. Inside those bounds, speed is never a grading input. A flow that takes ninety seconds on a warm environment and four minutes on a cold one produces the same verdict if the same things ended up true, which is a large part of how this design sidesteps the timing brittleness I covered in the flaky tests post.

When the agent finishes, it writes a closing summary of what it believes it accomplished. That summary goes into the trajectory clearly labeled as the agent’s own account. It is a claim, and the next stage exists because claims aren’t evidence.

How does a recorded trajectory become a verdict?

First the trajectory gets condensed into something a judge can actually read. A long run can produce hundreds of events, and a verifier drowning in context produces garbage verdicts, so on a very long run the runner keeps the opening stretch and a larger closing stretch, inserts a note that the middle was elided, and bounds the total size. The head preserves how the run began, the tail preserves where the outcome was decided, and the bound guarantees the judge reads all of what it’s given.

Then a separate verifier model, one that took no part in driving the browser, receives three things. The test’s expected outcome. The established precondition from setup, presented as guaranteed so the judge credits it rather than demanding the run re-prove it. And the condensed trajectory. Its standing instruction is to judge only on evidence in that record, and it’s warned specifically about the agent’s closing summary, that the agent may have claimed success without verifying, so the agent’s word alone never counts as evidence. The same instructions set a deliberate asymmetry, that a false pass is worse than a false alarm, so a doubtful core expectation gets marked missing rather than waved through.

The verifier can’t answer with a vibe. Every verdict comes back in the same fixed structure, carrying six things:

• the call itself, whether every core expectation held
• the written reasoning behind that call
• each expectation the trajectory clearly satisfies, itemized
• each core expectation it doesn’t, also itemized
• any flags, changed-but-equivalent observations passed on a stated assumption
• any extras, actions the agent took that nobody asked for

Two escape valves keep that strictness from turning into noise. An expectation the author prefixed with [incidental] can never fail the run, and behavior that changed but still satisfies the intent becomes an assumption flag on a passing verdict instead of a failure. A flagged run is still a pass, and a flag may never excuse an unmet core expectation.

That’s run 33 again, the failure I opened with, exactly as the dashboard renders it. The reasoning names the wrong cycle and the right one, and the itemized checklist behind it says which expectations died with that wrong turn. This stage is the one that turns a browser session into something you can act on without replaying it, and it’s also the clearest place to see how differently a red gets made here than in the usual alternatives.

What happens when the verdict comes back failed?

One thing, once. The runner re-runs the act phase on a stronger model than the first attempt used, and that retry produces its own fresh trajectory and its own fresh verdict from the same skeptical verifier. Nothing is averaged between attempts. The second verdict simply becomes the authoritative one.

Both outcomes are recorded in words a human can use. When the retry passes, the verdict’s reasoning states that the first failure was an agent-execution issue, not an app defect, and the record is annotated as escalated and passed-on-retry so the intermittent behavior stays visible instead of being laundered into a plain green. When the retry fails too, the reasoning states high confidence that this is a real issue, which is the sentence that earns a failure your attention. I walk the whole path in the post on false positives in automated testing, including the honest caveat that a genuinely intermittent app bug can pass the retry and wear the agent-error label.

Just as important is when this stage refuses to run. Blocked and skipped runs never escalate, because re-running a missing precondition on a smarter model fixes nothing. And if the verifier itself couldn’t be reached, the run records as failed with a verdict saying the trajectory needs manual review, and no retry fires, because escalating a run that no judge actually graded would manufacture confidence out of an outage. The retry budget exists for exactly one situation, a judged failure where a second attempt could genuinely change the conclusion.

Where does a graded run end up?

Cleanup runs first, whatever the verdict was. It reverts what the run created, follows the same idempotency rule as setup, gets its own time budget, and has no way to touch the grade. A cleanup hiccup can never turn a pass into a failure, and a doomed run still tidies up after itself so the next run doesn’t trip over its debris.

Then the record persists, and it’s the whole case file, not a letter grade. Verdict, reasoning, the itemized met and missing lists, any flags and extras, the full trajectory including any screenshots the agent took, the escalation annotation if one fired, and the duration. The suite’s progress streams to the dashboard while it runs, and the finished timeline renders each run’s verdict mix as separate counts rather than collapsing everything into red and green. On our recorded timeline that’s the difference between run 33, two failures in over twelve minutes that deserve an investigation, and runs 34 through 36, three blocked runs finished in about a minute or less apiece that deserve an environment fix. Same color on most dashboards. Different work, and the report says which.

One more door opens off this stage, deliberately gated. A judged failure’s evidence can become a bug candidate, and the verifier’s reasoning plus its unmet expectations ride along as the reproduction context. Candidates are never auto-filed. A human reviews each one in the dashboard and clicks to file it to Jira, after a strict duplicate check, and blocked runs never produce a candidate at all because there’s no application behavior to report. The trajectory and its screenshots stay attached to the run record itself, which is where anyone triaging the filed issue goes to see what the agent saw.

That’s the full journey. A three-field test either skips, blocks at the setup gate, or gets acted out in a real browser while every step is recorded, and the record gets read by a skeptical judge whose failure calls survive one stronger-model retry before reaching you. Each stage exists to make the next one’s output trustworthy, and the sum is the property I care most about in AI test automation, that by the time a red result asks for a human’s attention, the pipeline has already done the triage a human used to do by hand.