What is regression testing? A practical guide

Regression testing is the practice of re-running an application's existing tests after a code change to confirm that what worked before still works now. The new feature isn't the subject. The subject is everything the change wasn't supposed to touch, because software changes have a long history of breaking things far from where anyone was looking. If a release process runs one kind of automated testing, it runs this kind.

I build TestAutomate, an agentic tool whose day job is exactly this, so I spend most of my time inside regression suites: what belongs in them, why they rot, and what a result has to look like before anyone trusts it. This page is the practical version of that. It defines the term properly, separates it from the things it gets confused with, and shows a real regression test you can read in full, because the top pages for this query define the concept for a thousand words without ever showing one.

What is regression testing, precisely?

The name comes from "regression" in its plain sense: a return to a worse state. A regression is a feature that used to work and now doesn't, and regression testing is the discipline of catching that class of bug before users do. The ISTQB glossary defines it as testing a previously tested component after modification, to detect defects introduced or uncovered by the change, and Wikipedia's entry adds the part practitioners feel daily: it means re-running functional and non-functional tests, and it consumes a large share of most teams' testing budget.

Two properties make a check a regression test rather than just a test. First, it asserts behavior that already shipped and already passed, so a failure means something got worse. Second, it runs repeatedly, on a cadence tied to change: every deploy, every nightly build, every sprint. A test you run once is an acceptance check. The same test, kept alive and re-run against every change afterward, is regression coverage.

Regression testing vs retesting vs smoke testing

Three terms get tangled here, and the difference is about what you're trying to learn. Retesting confirms a fix: a test failed, someone repaired the code, and you run that specific test again expecting it to pass. Regression testing confirms the opposite set, the tests that were already passing, still pass after the change. Smoke testing is the thin, fast slice of regression you run first: can users log in, does the core flow complete, is the build even worth testing further. A reasonable pipeline runs smoke on every commit, full regression on every release candidate, and retesting whenever a fix lands.

The distinction matters operationally because the three have different failure meanings. A retest failing means the fix didn't work. A smoke test failing means stop the line. A regression test failing means a change had a side effect nobody predicted, and that class of failure is why the suite exists at all.

What belongs in a regression test suite?

Not everything, and the teams that try to regression-test everything end up maintaining a suite so slow and brittle that it stops running. My working decision rule has three questions. Does this flow carry revenue or data, meaning login, checkout, billing, anything that writes records users care about? Has this flow broken before, because past regressions are the best predictor of future ones? Would a user notice within a day if this silently broke? Two yeses puts the flow in the suite. Zero yeses means an exploratory check covers it fine.

The other half of the decision is what each test asserts. A regression test should declare the outcome that must stay true, not the route the app takes to get there. Suites built from routes, recorded clicks and stored selectors, fail whenever the route changes cosmetically, and my colleague wrote up the mechanics of that decay in why regression suites break. Suites built from outcomes only fail when behavior changes, which is the only failure you wanted to hear about.

What does a regression test look like in practice?

Here is a complete regression test in TestAutomate, every field of it. There are three required fields, an id, a plain-language prompt, and an expected outcome, and nothing in it is a selector:

id: saved-filter-persists
prompt: >
  Create a saved filter named "My open tickets" that shows only open
  tickets assigned to the current user, then reload the app and open
  the filter again.
expected_outcome: |
  - A filter named "My open tickets" appears in the saved-filters list.
  - Opening it shows only tickets that are both open and assigned to
    the current user.
  - After a full page reload, the filter still exists and returns the
    same view.
  - [incidental] A confirmation toast appears when the filter is first
    saved.

Read what the expected outcome asserts: existence, correctness of the filtered view, and persistence across a reload. It never mentions a button, a CSS class, or a URL. When the front end gets redesigned, this test doesn't need repair, because an agent re-reads the new page and works out the route fresh on every run. Tests carry optional tags like smoke and regression, so the fast slice and the full suite are filters over one set of artifacts rather than two codebases. And because tests are stored as plain YAML, the suite lives in version control and diffs like any other code. The deeper walkthrough of the format lives in plain-English test authoring.

How does automated regression testing work?

The honest answer depends on which generation of tooling you're holding, and the differences compound at suite scale, which is where regression lives:

In our loop specifically, the suite can be drafted rather than hand-built: documentation, the repository, and the running app are read together, and draft regression tests come out of the reconciliation for a human to review and approve. That matters for coverage because the suite starts from what the product is supposed to do, not from whichever flows an engineer had time to script. Runs go on a schedule, results arrive in Slack or email, and confirmed failures become deduplicated, evidence-backed bug drafts a human approves before anything reaches the tracker.

How do you read regression test results?

Here's a real regression timeline from our own dashboard, twenty-five recorded suite runs against a test application, each row showing its verdict mix and duration:

The detail worth stealing regardless of your tooling is that the verdicts aren't binary. A run in that timeline can be passed, failed, blocked, or skipped, and the difference between failed and blocked is the difference between a real regression and a broken test environment. Run 36 in that dataset is the example I show people: one test, blocked, one minute three seconds, because setup couldn't establish the precondition, so the app was never exercised and the verifier explicitly recorded it as a test-environment problem rather than an app defect. In a binary red-green world, that run is a false alarm that wastes a triage. Graded, it's a maintenance signal. No failure gets reported to us until a stronger model has re-driven the flow and agreed, and a pass that needed a retry is labeled as exactly that. The full grading mechanics live in how verdicts work, and there's a complete graded run you can read end to end.

How often should you run regression tests?

Tie the cadence to change, not the calendar. The rule I give teams: smoke slice on every deploy, the suite for a touched area on every merge to main, the full regression suite nightly and before every release. If any of those feels too expensive to run that often, that's not a cadence problem, it's a suite problem, and the fix is cutting cost per run rather than running less. Cost per run is a first-class number for us for exactly this reason: on your own key, model spend passes through at cost with a ceiling that pauses the run before overspend, so the economics of running nightly are visible before you commit to them. The wider arithmetic is in what AI testing costs.

If you take one thing from this page: regression testing is a promise about the past, kept on every change. Keep the suite small enough to run always, assert outcomes rather than routes, and demand results that distinguish a broken product from a broken test. The rest is tooling, and the AI test automation guide covers how that landscape is shifting.