Graded test verdicts: pass, fail, blocked
A red result means a real bug. An AI verifier grades every run pass, fail, or blocked — with screenshots and written reasoning attached. If you can't trust the grade, nothing else matters. This page is how we earn it.
Four rules that keep the grade honest
Most AI test tools have two answers: green and red. Real software has more states than that, and pretending otherwise is how teams learn to ignore their test results. TestAutomate grades with four mechanics built in.
Broken staging is never a product bug
Your staging environment returns a 502 halfway through checkout. A selector tool marks the test red and someone burns twenty minutes triaging a bug that doesn't exist. TestAutomate grades that run blocked: the agent couldn't reach the thing it was asked to test, so no claim is made about the product either way. Blocked runs are reported separately, with screenshots of what actually went wrong — an expired test account, a down service, missing seed data — so you fix the environment instead of chasing a ghost.
Failures are re-driven by a stronger model first
Suppose the everyday driving model misreads a redesigned dropdown and can't complete the flow. Before that failure ever reaches you, the run is handed to a stronger model that re-drives the whole test from the start. Only if the stronger model also hits the wall does the verdict come back fail. You pay the expensive model only on the runs that look broken — which is exactly when a second opinion is worth it.
Flakes are labeled, not laundered
A test fails once, then passes on the retry. Plenty of tools quietly report green and move on — and six weeks later you have a suite nobody believes. TestAutomate reports the pass but labels it passed on retry, so intermittent behavior stays visible. A flake label on the same test three sprints running is a signal: something in that flow is genuinely unstable, even if it usually squeaks through.
Changed-but-equivalent UI is a note, not a failure
Design renames the button from "Save" to "Save changes". The flow still works. A brittle assertion would fail; a careless AI would say nothing. TestAutomate passes the run and attaches an advisory flag: the UI differs from what the test described, here's the screenshot, update the test if the change was intentional. You hear about drift without your dashboard turning red over a copy tweak.
What a verdict contains
A verdict is not a checkmark. It's a small case file you can audit in under a minute — and overrule if you disagree.
- Written reasoning. The verifier explains, in plain English, why it graded the run the way it did.
- Expectations, met and missing. Each expectation from your test is listed individually — satisfied, unsatisfied, or unreachable.
- Screenshots. Captured at the moments that matter, so you can see exactly what the agent saw.
- Model used. Which model drove and which model graded — including whether escalation kicked in.
- Duration and steps. How long the run took and the path the agent actually walked through your app.
Straight answers
What if the verifier is wrong?
Sometimes it will be — any grading system, human or AI, has an error rate. The design assumes that. Every verdict ships with its screenshots and reasoning, so checking one takes seconds, not a re-run. Failures get a second opinion from a stronger model before you ever see them, which removes most false alarms at the source. And nothing is filed to Jira without your explicit confirm — a wrong verdict costs you one click of review, never a polluted bug tracker. When you overrule a verdict, that's your call to make; the evidence is right there.
Is this deterministic? Will the same test run the same way twice?
Honestly: no. An agent driving a real browser from a plain-English test may take a slightly different path on different runs — that's also why it survives UI changes that shatter selector-based suites. What is held stable is the grading standard: the same expectations, checked against evidence, every time. Flake labeling and advisory flags exist precisely because we don't pretend runs are identical. Deterministic trace replay — re-executing a recorded run step for step — is on the roadmap. coming
Run one test. Read one verdict.
The fastest way to judge the grading is to see it grade your own app.