LLM as a Judge Software Testing in Practice

The first reaction most QA people have to this corner of AI test automation is suspicion. A model drove the browser, and now another model decides whether the run passed? I designed TestAutomate’s verification layer around that suspicion rather than against it, because the suspicion is correct. An unexamined judge is a rubber stamp with extra steps.

LLM as a judge software testing means a language model, separate from whatever executed the test, grades the run against the expected outcome and returns a verdict. It works when three properties hold. The judge never drives the run it grades, it treats unevidenced success as failure, and it must name every expectation met and missing.

Most writing about LLM judges comes from the AI-evaluation world, where models grade chatbot answers for helpfulness or tone. Grading a functional test run is a different job with sharper edges, and it deserves its own treatment. So here it is. What the judge reads, why it never grades its own driving, when a deterministic check beats it, and where reliability and bias actually get decided.

What does LLM as a judge software testing actually mean?

It means a language model plays the test oracle, the mechanism that decides whether observed behavior counts as correct. The term migrated over from AI evaluation, where the study that popularized it found a strong model judge agreeing with human preferences more than 80 percent of the time, roughly the rate at which humans agree with each other. That literature grades essays and chat answers against rubrics. In software testing the judge grades something more concrete, a recorded run of a real application, against something more falsifiable, a written expected outcome.

The test oracle problem is much older than any of this. Every automated test needs a way to separate right behavior from wrong, and for end-to-end flows through a browser, that mechanism has historically been either a brittle coded proxy or a human. A skeptical model judge is a third option, and in TestAutomate it’s the only thing standing between an agent’s browser session and a verdict you’ll be asked to act on.

The shape of the test artifact decides whether the judge has anything real to check. A TestAutomate test is three required plain-language fields, an id, a prompt saying what to do, and an expected outcome saying what must be true afterward, and the runner refuses to load a test that’s missing its prompt or its expected outcome. The expected outcome isn’t a vibe. It’s itemized assertions about the end state, including negative ones like no other records being modified, and writing those assertions so they hold up is most of the craft. The judge’s entire job is checking that checklist against what actually happened.

Framed that way, the laziest version of the idea is already ruled out, the one where a model glances at a final screenshot and emits a thumbs-up. A checklist oracle has to show its work. Which items were satisfied, which weren’t, on what evidence. Everything else in the design follows from demanding exactly that.

Why shouldn’t the agent that ran the test grade its own run?

Because the agent’s account of its own run is a claim, not evidence. The same evaluation literature that validated model judges also catalogued their biases, and the one with real teeth here is self-enhancement bias, a model rating its own output more generously than a neutral party would. An agent that has just fought its way through a long flow is the least neutral party available. Its final summary reflects what it set out to accomplish, and intent is precisely what a verdict must not be built from.

So the split in TestAutomate is architectural rather than stylistic. One model drives the browser. A separate verifier model, which never participated in the run, makes one judgment per attempt. It receives the test’s expected outcome and the recorded trajectory of the run, meaning every action the agent took, every result that came back, and the agent’s closing summary presented as the agent’s own account. Its standing instruction is to judge only on evidence in that trajectory. If nothing in the record clearly shows an expectation was satisfied, the expectation is not met, however confidently the summary claimed success. AI test verification that takes the executor’s word for it isn’t verification. It’s transcription.

Two quieter details keep the separation honest. The judge grades one attempt’s trajectory, never a blend. When a failed first attempt gets retried once on a stronger model, the retry receives its own fresh judgment of its own trajectory, and the stronger attempt’s verdict becomes the authoritative one. Nothing is averaged across attempts. And the judge is told what the setup phase established before the test began, so it credits preconditions instead of demanding the run re-prove them. It grades the test’s own actions on top of that guaranteed state, which keeps verdicts about the flow under test instead of the scaffolding around it.

The blunt version is that we assumed the agent would sometimes be wrong about itself, then built the grading so that being wrong about itself can’t survive contact with the record.

When do deterministic checks beat an LLM judge?

Whenever the property you’re checking is machine-decidable and the interface it lives behind holds still. That isn’t a grudging concession. A coded assertion on a stable API returns the same verdict for the same input every time, costs nearly nothing to run, and fails in a way a stack trace can explain. Determinism is a real virtue in a test suite, and the classic case for eradicating non-determinism in tests applies with full force to any probabilistic judge, ours included. Run a model twice on a genuinely borderline trajectory and you can get two different verdicts. Anyone presenting LLM as a judge vs deterministic checks as a rout in either direction is skipping the interesting part.

The interesting part is what the check is actually checking. Unit and API layers state their expectations in machine terms, and there a deterministic assertion is strictly better. End-to-end flows state their expectations in human terms. The new cycle appears with the specified date range. No other participants were modified. The true oracle for sentences like those is judgment, and a deterministic proxy for judgment, a selector plus an equality check, encodes it badly and then silently drifts away from it as the UI changes. You don’t escape judgment by coding it. You freeze one day’s version of it and hope.

Here’s who grades a run under each approach, and on what evidence.

Honest caveats belong under that table. Keep deterministic layers deterministic, because nothing about a model judge belongs anywhere a coded assertion already decides the question cheaply. Screenshot diffing genuinely beats a trajectory reader at catching a subtle layout regression, which a judge reading actions and results will sail straight past. The human reviewer remains the standard every other row approximates. The verifier’s row earns its place in one specific territory, open-ended flows whose outcomes are written in natural language and whose interfaces refuse to hold still, and that territory happens to be where end-to-end suites spend most of their lives.

How do you handle LLM as a judge reliability and bias?

By engineering against named failure modes instead of trusting a capable model to be fair. The bias catalogue from the evaluation literature maps onto specific countermeasures in how our verifier is framed, and the mapping is more instructive than any reassurance.

Self-enhancement bias is handled structurally. The judge never grades its own work because it never does the work, which is the previous section in one sentence. Position bias, the tendency to favor whichever answer comes first, mostly afflicts pairwise comparison, and grading a single run against a checklist avoids that framing by construction. Verbosity bias, the tendency to reward long confident prose, would be the deadliest one here, because an agent’s closing summary is exactly that. The countermeasure is the evidence rule. Claims earn nothing. Only recorded actions and their results count toward an expectation, so a fluent summary with no supporting trajectory reads, to this judge, as a failure.

Leniency is the bias with the biggest cost in QA, the drift toward giving a plausible run the benefit of the doubt. Our verifier carries an explicit asymmetry instead. Claiming a pass when a core behavior broke is treated as much worse than a false alarm, and when in doubt about a core expectation, the judge marks it missing. On its own that rule would shred every run where the UI drifted harmlessly, so two pressure valves keep strictness from curdling into noise. An expectation the author prefixes with [incidental] is declared leniency, a nice-to-have whose absence can never fail the run. And changed-but-equivalent behavior, a renamed button, an extra confirmation dialog, becomes an assumption flag on a passing verdict rather than a failure, recording what was observed, what was expected, and why the change looks intended. The boundary holds in both directions. A flagged pass is still a pass, and a flag may never excuse a genuinely unmet core expectation.

Reliability also means behaving well at the edges. A very long run is condensed to what the judge can reliably read before grading, because a judge drowning in context produces garbage verdicts. And when the verifier itself can’t be reached, the system refuses to improvise. The run is recorded as failed with a verdict stating that the judge was unavailable and the trajectory needs human review, and that fallback never triggers the stronger-model retry, because escalating a run no judge actually graded would manufacture confidence out of an outage.

What should a verdict give you beyond pass or fail?

Named evidence you can act on without re-watching the run. Every verdict our verifier produces states whether the run passed, a reasoning summary behind the call, every expectation met, every expectation missing, any assumption flags, and any extras, meaning actions the agent took that nobody asked for, which is its own quiet category of signal. The dashboard surfaces the reasoning summary right on the test row, and the full run report itemizes the met and missing checklists.

That’s a real failed verdict from our QA environment. The test required the agent to work in a review cycle named 360-QA-Regression-2026. The verifier’s reasoning states that the agent operated on the wrong cycle, 360-Review-Fiscal-Year-2026, so the expectations tied to the required cycle were not met. Notice what that sentence does for whoever reads it next. A bare red X says investigate everything. A named unmet expectation says look at cycle selection first, and it turns the follow-up question, agent error or genuine app problem, into a reading exercise instead of a re-run.

The four-outcome vocabulary keeps those verdicts unpolluted. A run ends passed, failed, skipped, or blocked, and a blocked run never reaches the judge at all, because a precondition that couldn’t be established says nothing about the application. Every verdict the verifier issues therefore describes a run where the app was genuinely exercised, which is what makes the failure list worth triaging at all.

Downstream, the same structure feeds the bug workflow. Evidence from a failed run can become a bug candidate, checked against existing issues by a deliberately strict duplicate judge, and nothing is ever filed until a human clicks to confirm, with the verifier’s reasoning and unmet expectations carried into what does get filed. The judge grades runs. People decide what enters the tracker.

Where this leaves the human is reading verdicts rather than re-watching runs, which is the productivity claim of the whole agentic testing approach, and it only holds if the verdicts deserve the trust. None of what earns that trust is exotic. Separation, an evidence rule, an asymmetry, a checklist the judge must fill in item by item. We wrote the skepticism down where the model can’t ignore it, and that, more than any cleverness, is what makes an LLM judge safe to build a testing practice on.