What a graded test run looks like
One real run from our own dashboard, walked end to end: the plain-English test, the grade it earned, the reasoning behind the grade, and what happens to a failure afterwards. Every screenshot on this page is unedited dashboard output from a recorded run.
A real run, not a mock-up
When I show TestAutomate to a QA lead, the first thing they ask for isn't the pitch. It's a report. They want to see what an AI test verdict actually looks like before they'll trust one, and I think that's the right instinct. So this page is a single recorded suite run against a demo HR app, with the screenshots exactly as the dashboard rendered them. The app is a QA test environment for an HCM product, populated with fictional demo personas. Nothing is composited, and no number appears in the text that you can't read off the image yourself.
TestAutomate is pre-launch (early access), so these are runs we drove ourselves while building the grader for our agentic testing tool. That's worth saying up front. What this page can show honestly is how the product records, grades and explains a run. What it can't show is a customer's production suite, because there isn't one yet.
traineryhcm app: 25 suite runs, newest first. Each collapsed row carries its verdict mix (passed, failed, blocked, skipped), the date and the duration. Run 36 graded one test blocked in 1m 3s. Run 33 graded two tests failed in 12m 10s.A few things stand out in that timeline before we open a row. The verdict mix has four symbols, not two. Blocked and skipped sit alongside passed and failed, and they're counted separately, so a run that never reached the app can't inflate a failure total. Durations range from under a minute to just over twelve minutes, which tells you how far into the flow the agent got each time, and the newest three runs all graded blocked rather than failed. We'll come back to why that distinction matters.
The test as it was written
The test inside every expanded row below is called "Add Participants to Existing Cycle". It's a single plain-English test: an intent written as a prompt, plus an expected outcome, stored together in versionable YAML. There's no selector layer under it. Nothing in the file says which button to click or what the page structure looks like; the agent works that out in a real Chrome on each run. The same file carries the test's setup and cleanup phases, each with its own timeout, so the run can establish its own precondition and tidy up afterwards without depending on whichever test ran before it.
You can read part of the expected outcome straight off the failed verdict further down. The verifier notes that the expected outcome "explicitly requires changes to the '360-QA-Regression-2026' cycle detail view". That is the standard the run is graded against. Not "did the agent get somewhere", but "did the specific thing the author asked for happen".
The verdict, and what each grade means
TestAutomate grades every run one of three ways. Pass means the evidence met the expected outcome. Fail means the agent exercised the app and the expected outcome wasn't met. Blocked means the agent couldn't establish the precondition or reach the thing under test, so the run makes no claim about the product either way. A broken environment can never turn into a false fail, because the grader has a separate word for it.
The expected outcome was met
The agent exercised the app and the evidence satisfies the expectations the author wrote. If the pass only came on a re-drive, the report says so.
The app was exercised and the outcome wasn't met
Something the author asked for didn't happen. This is the grade that earns a second opinion from a stronger model before you ever see it.
The thing under test was never reached
Missing precondition, down service, expired account. Reported separately, with the evidence, so you fix the environment instead of the product.
sonnet.Here's Run 36 opened up. The test is rendered in amber as BLOCKED, and the verifier explains why in plain language: the setup phase couldn't establish the precondition, so "the application was not exercised, so this is neither a pass nor a failure; it indicates a missing test precondition (test-environment), not an app defect." I like this one because it's the boring case binary tools get wrong. A red/green tool would have painted it red, and someone would have spent twenty minutes hunting a bug that isn't there.
Run 33 is a genuine fail, and the reasoning shows why the two grades are different animals. The agent got into the app and did work, but on the wrong 360-review cycle. The expected outcome named a specific cycle, so the expectations tied to it are not met. That's a result the author needs to look at, whether it turns out to be an app defect or a test that needs a tighter precondition, and the verdict hands them the exact sentence they need to decide.
The evidence behind the grade
What the dashboard renders inline is the verifier's reasoning summary, the model that graded (the sonnet tag on both rows) and the run's duration. The itemized checklist, each expectation from the test listed as met or missing, lives in the run report behind the "Save as PDF" button you can see in the expanded row. That report also carries the step trail: the path the agent actually walked through the app, with the screenshots it captured along the way, so you can see what the agent saw at the moments that mattered.
A run can also be shared by link and replayed, scrubbing through the evidence, and that same evidence is what gets attached to a bug if one is filed. None of that is pictured here. I'd rather leave the gap visible than fake a frame, so I've only used the screenshots our own capture rules let me publish.
Flags, second opinions, and the tracker
Two more things a report can carry, neither of which appears in these particular runs. The first is an advisory flag, sometimes called an assumption flag. When the UI has been renamed but is functionally equivalent, say a button now reads "Save changes" instead of "Save", the run passes and the verdict carries a flag pointing at the difference, with the screenshot, so you hear about drift without your dashboard turning red. Flags landed after the runs above were recorded, which is why you don't see one, and I'm not going to imply one.
The second is how a failure is re-driven before it's reported. When the everyday driving model fails a test on a genuinely judged failure, the run is handed to a stronger model that re-drives the whole test from the start. Only if the stronger model also fails does the verdict come back as fail. Blocked and skipped runs are never escalated, since there's nothing to re-drive when the app wasn't reached. If a test fails once and passes on the re-drive, it's reported as "passed on retry", never quietly as a clean pass. The dataset behind this page contains no escalated or passed-on-retry runs, so there's no re-drive to show you here.
When a fail survives the re-drive, it becomes a bug candidate. TestAutomate checks it against a live search of your tracker and a cached inventory of known issues, so a failure matching an open bug is linked to it rather than cloned. It then drafts the Jira issue, steps to reproduce, expected versus actual, the screenshots from the run, and waits. Filing is always confirm-gated; nothing lands in Jira on the agent's say-so. Once the fix ships, the close-the-loop path re-verifies the issue, comments on it and transitions it. The bug loop page walks that path in detail. I haven't pictured the filing step here, because the only tracker connected to this demo instance is a production board, and filing to it would have created a real ticket.
- Grading rules in full. Blocked versus failed, escalation and retry labels, explained on the graded verdicts page.
- Side by side. How this report compares with binary red/green tools is on the comparison pages.
Straight answers
What does a blocked test result mean?
Blocked means the agent couldn't reach the thing it was asked to test, so the run makes no claim about the product either way. In the example above, the setup phase couldn't establish the test's precondition, and the verifier says so in plain language. Blocked runs are counted and reported separately from failures, with the evidence of what went wrong, so you fix the environment or the test data instead of chasing a defect that doesn't exist.
How does an AI test verdict avoid false failures?
Three mechanics do most of the work. Broken environments grade blocked rather than failed. A failure is re-driven by a stronger model before it's reported, and only comes back as a fail if that model fails too. And a test that passes on a retry is labeled "passed on retry" rather than reported as a clean pass. On top of that, every verdict ships with its reasoning and screenshots, so overruling a wrong one costs a minute of reading, not a re-run.
What evidence does an automated test report include?
A TestAutomate run report carries the verifier's written reasoning, each expectation from the test marked met or missing, screenshots from the moments that mattered, which model drove and which model graded, and the run's duration and step trail. The dashboard shows the reasoning summary and model tag inline; the itemized checklist is in the PDF export, and the same evidence is attached to any bug filed from the run.
Run one test against your own app.
The fastest way to judge a report is to read one about software you already know.