Assumption Flags: UI Changed, Nothing Broke

Every UI suite eventually meets the same awkward run: the app changed, the test’s wording didn’t, and the flow still did exactly what it was supposed to do. Assumption flags are how TestAutomate’s verifier reports that run honestly. Instead of failing on a harmless difference or waving it through in silence, the verdict passes and carries a note saying what changed, what the test expected, and why the judge believed the change was deliberate.

An assumption flag is an annotation attached to a passing test run when the application did something different from what the expected outcome implied, but the difference still satisfied the test’s core intent. The flag records three things: what the app actually did, what the test’s wording expected, and the assumption the verifier made in letting it pass. A human reviews the flag later. The verdict itself stays green.

One clarification before anything else, because search results muddy it badly: assumption flags have nothing to do with feature flags. A feature flag is a switch that turns product behavior on or off. An assumption flag is a verdict annotation, and as far as I can tell almost nobody ships one, which is why I wrote the rule into our verifier instead of leaving the gap to a reviewer’s memory.

What do assumption flags actually record?

Three fields, and the third is the one I care about most. Every flag holds what the app actually did, what the test’s expected outcome implied, and why the verifier believes the difference was probably intended. On the record those fields are literally named observed, expected, and assumption. The first two establish the gap. The third is the judgment call, written down where a reviewer can accept it or reject it.

A worked example makes the shape obvious. Your test’s expected outcome says the user clicks Save and the record shows as saved. A release renames the button to Save as and adds a small confirmation dialog. The agent runs the flow, handles the dialog, and the record ends up saved. Nothing in the core intent failed, but two details of the app’s surface no longer match the test’s wording. So the verifier writes a flag instead of a failure. Observed, the button now reads Save as and a confirmation dialog appeared. Expected, a Save button and no dialog. Assumption, the rename and the dialog look like deliberate design changes because the save still completed. The run is green, and the note rides on it.

Notice how often a flag turns out to be a claim about what a control is now called. That’s no accident. The agent finds elements the way a person would, by reading what the page says they are, so the accessible name a page declares on its controls is exactly the layer where renames surface. A well-named control that changes its label produces a crisp, reviewable flag. An anonymous div that changes produces nothing readable for anyone.

On the dashboard, a flagged pass stays visually a pass. The test’s row keeps its green verdict and gains a small flag marker, the expanded view lists each assumption in full, and suite results count how many assumptions a passing run carried, tagged as worth a look. I sized that treatment deliberately. A flag shouldn’t shout like a failure, because it isn’t one, but it should be impossible to open a flagged run without seeing what the judge assumed on your behalf.

Why does a changed UI still produce a pass?

Because the run met every core expectation the test actually asked for, and a verdict should report on outcomes, not vocabulary. The sharper question is why the difference gets recorded at all rather than ignored, and the answer is that both easy defaults are corrosive. Fail the run and you’ve created a red result that means nothing, the classic false positive that trains a team to stop trusting red. Pass it silently and you’ve discarded the only interesting fact the run produced, which is that the app’s surface has drifted from what the test’s author believed. Somebody should confirm that drift was on purpose. The flag keeps the pass and keeps the fact.

The rule that makes this safe is the one we enforce hardest on the verifier. A flag applies only to a difference that still satisfies the expectation. If a core expectation genuinely went unmet, the action failed, the wrong thing happened, the data is wrong or absent, then the run is a failure with a missing expectation, and passing a broken run on the strength of a flag is the failure mode we designed the verifier hardest against. Flags annotate success. They can’t manufacture it. That boundary is what separates an assumption flag from the quiet leniency that ruins suites, where each accommodation makes a test a little harder to fail until it can’t fail at all.

It also helps to place flags next to the one other softening mechanism in the system, because the two point in opposite directions. An expectation the test’s author prefixes with [incidental] is declared leniency. You decide in advance that a nice-to-have can never fail the run. A flag is discovered leniency. The judge finds an unexpected difference at verdict time and documents the assumption it made. One is your call, made early. The other is the model’s call, made late, and surfaced precisely because it was the model’s call. The broader case for letting a model judge test outcomes rests on the same skepticism. Our verifier treats unevidenced success as failure, so the flag is the one narrow lane where it may accept a difference at all, and the lane comes with a paper trail.

One consequence is worth knowing before you see it in a run list. A flagged pass never triggers the retry machinery. When a run genuinely fails, TestAutomate re-runs the flow once on a stronger model before letting the failure stand. A flagged pass isn’t a failure, so it burns no retry and delays nothing. The extra scrutiny a flag requests is human and asynchronous, not computational and blocking.

How does each approach report an intentional UI change?

Mostly as breakage, which is odd once you notice what actually happened: the product changed on purpose and kept working. A rename that ships in a planned redesign isn’t a defect, yet most reporting pipelines render it as one. The comparison that matters is what lands in your report when the UI changes intentionally and the flow still works.

Honest caveats, because that table flatters us if read uncritically. A scripted suite written against roles and visible labels rather than brittle CSS will survive many renames outright, and disciplined teams build exactly that. The catch is that surviving and reporting are different things, since the run that survives says nothing about the change at all. Managed QA humans are genuinely the best judges of intent on this list, and their eventual note is often better written than any machine’s. What you trade there is turnaround time and per-cycle cost. And our own row leans on a model’s judgment that the intent was still satisfied, which is exactly why the assumption is written down for review instead of trusted silently. A judge that can be wrong and shows its reasoning beats a judge that can be wrong and doesn’t.

What should you do when a run comes back flagged?

Answer one question: was the change intended? Everything else follows from the answer, and the flag exists to make the question cheap to ask.

If the change was intended, update the test’s wording so the next run doesn’t need the assumption. The flag’s expected field tells you which phrase went stale, so this is editing a sentence, not repairing a script. Change the expected outcome from a Save button to Save as, mention the confirmation dialog, done. It’s the same tightening discipline that writing good expected results demands in general. Expectations should describe the app you have, precisely, so that deviations mean something. Leave the wording stale and the verifier will keep making the same assumption on every run, which works, but every repeat is the judge guessing where your test should be stating.

This is also where most test maintenance after a redesign actually lands. When tests are plain-language descriptions of intent rather than selector scripts, a redesign doesn’t hand you a pile of broken locators to fix. It hands you a set of flags telling you which descriptions drifted, each with the new reality already transcribed in the observed field. The maintenance is reading and rewording, and I designed the flag record to double as that worklist.

If the change was not intended, the flag is one of the earliest honest signals you’ll get, precisely because nothing failed. An unplanned rename, a dialog somebody’s change introduced by accident, a control that moved when a layout refactor went further than intended. None of these break a flow, so a regression process keyed entirely to failures stays silent about all of them. The flag describes the drift in words a developer can act on directly. What happens next stays with you. If it deserves a ticket, bug candidates in TestAutomate are only ever filed by a human clicking to confirm, deduplicated against existing issues first, and nothing about a flag changes that rule.

Recurrence tells you which of the two readings you’re in. The same flag appearing on one test across several runs is almost always stale wording, and the fix is the one-sentence edit above. Similar flags appearing across many unrelated tests at once suggest a release shipped surface changes nobody catalogued, and that’s a conversation with whoever shipped it, with the observed fields as your inventory. Either way, don’t let flags pile up unread. Each unreviewed flag is a question the verifier asked and nobody answered, and the whole value of the mechanism is in the answers.

How are assumption flags different from self-healing tests?

They move the decision in opposite directions. A self-healing tool detects that a step no longer matches the app, patches the step, and keeps the run green. The patch is the tool’s decision, reported after the fact if at all, and the failure mode is well known. The UI changed but tests still pass, and the record of what changed lives in a healing log nobody reads, if it lives anywhere. When the change was an accident, the suite has just absorbed a regression on your behalf. There’s a fuller comparison in self-healing versus selectorless testing, but the short version is that healing edits the test and tells you second, while a flag edits nothing and tells you first.

Flags don’t need healing underneath them, because there’s nothing to heal. The agent reads the live page and finds elements like a person would, so a renamed button doesn’t break a locator. There isn’t one. The run proceeds against the app as it now is, and the flag captures the difference at verdict time rather than papering over it at runtime.

The flaky comparison fails for a different reason. A flaky test returns different verdicts against an unchanged app, which is why Martin Fowler treats non-determinism in tests as something to eradicate rather than manage. A flagged pass has a determinate cause. The app changed and the wording didn’t, so re-running it yields the same pass with the same flag until someone updates the test or the app. Nothing about it is noise. It’s the opposite, a signal precise enough to name the sentence you should edit.

Two boundaries complete the picture, so nobody buys more than we built. There is no watcher diffing your UI between runs. A flag is a judgment the verifier makes per run, at verdict time, from the trajectory and the expected outcome, so a test that never runs will never flag anything. And nothing rewrites tests in response to a flag. The review, the decision, and the edit are deliberately human work, because an assumption a model makes and a model then confirms is just the silent healing we set out to avoid.

The design bet underneath all of this is small and specific. When an automated judge makes an assumption, that assumption should appear on the record next to the verdict it shaped, in terms a person can accept or reject. That’s the entire feature. A flagged pass is still a pass, a flag can never excuse a genuine failure, and every flag ends its life the same way, with a human deciding whether the app or the test is the thing that should change. Suites earn trust by being caught reasoning honestly. Writing the assumptions down was the cheapest honesty we could ship.