How Does AI Test Case Generation Work?

Every pitch for AI test case generation shows you the moment a test appears out of a docs page. I want to show you the opposite moment, the one that actually decides whether the generated suite is any good: the moment a page gets thrown away. I built TestAutomate’s ingestion pipeline to be a rejection machine first and a generator second, and what follows is an inventory of everything it refuses to turn into a test.

AI test case generation, done honestly, is a filtering problem before it’s a writing problem. TestAutomate crawls your product docs, discards every page that doesn’t describe a real user task, extracts the survivors into skills, and only then drafts runnable plain-language tests from them. Most of the input never becomes a test, on purpose, because a generated test with nothing real behind it costs more than no test at all.

When I pointed the pipeline at a real docs site and read the diagnostics afterward, the reject pile was the more interesting half of the story. So rather than walk the pipeline forward the way that post does, I’ll walk it as a funnel: four cuts between a docs URL and a kept test, each with a reason you can inspect.

How does AI test case generation work in TestAutomate?

Four stages run in order, crawl, extract, classify, emit, and then a separate drafting step turns what survived into tests. The crawler walks your documentation starting from the URL you paste. Extraction reads each fetched page and pulls out the user-facing actions it describes, keeping the doc’s own wording for buttons and fields instead of paraphrasing it. A classification pass sorts each extracted intent by what kind of handling it needs at runtime. The emit stage stores the survivors in your workspace as skills, each one a distilled procedure a user could actually perform.

Nothing gets generated at ingestion time. Test drafting waits until the first time someone clicks Run suite on an app that has no test plan yet, and I made it lazy deliberately: the skills sit in your workspace where you can look them over before a single test exists, and an ingestion run you abandon never litters your suite with drafts. When drafting does run, it works in two steps. First the app’s skills get clustered into topics, without you defining categories anywhere. Then a small number of test flows get written per topic.

Every drafted test carries the same three required fields as a hand-written one, an id, a prompt, and an expected outcome, with optional setup and cleanup prompts around them. The expected outcome comes out as itemized end-state assertions, including negative ones like “no other records were modified”, and each assertion is load-bearing unless it’s deliberately marked incidental, in which case its absence can never fail the test. The format rules are blunt: plain natural language only, with code, JSON, YAML, and selectors explicitly forbidden inside the prompt and the expected outcome. The agent re-derives how to execute each step against the live page at run time, the same contract behind everything else in our agentic testing setup, so nothing brittle gets frozen in at generation time.

That’s the machinery, and it’s also the least interesting part. None of it explains why the output is worth running. The filters do.

Which pages never even get fetched?

Whole sections of your docs site are excluded by URL before a single request goes out. The crawl derives its scope from the URL you paste: it stays on that site’s own origin, inside the docs subtree you pointed it at, and it carries an exclusion list for address patterns that never describe testable behavior. Changelog paths, blog posts, about pages, pricing, customer showcases, careers pages, and login and signup routes are all rejected on the URL string alone. A web crawler that fetched everything reachable would just manufacture work for the later filters, so the cheapest cut happens first.

There’s a practical decision rule hiding in that scope derivation. The crawl treats the docs section containing the page you paste as its territory, so pasting any page inside your user docs gets you the whole section, while pasting a developer portal gets you a pile of API reference that the task filter will throw away later anyway. Point it at the help content your users actually read, and the funnel starts full of the right raw material.

Discovery is deliberately boring. The crawler tries the site’s sitemap first, since a published sitemap is the most reliable inventory of a docs site’s pages, and falls back to following in-scope links when no sitemap exists. The whole walk is bounded and politely paced, and both bounds are qualitative on purpose: enough pages to cover a real docs section, slow enough not to hammer anyone’s server.

What does get fetched is then reduced to its actual content. Navigation menus, headers, footers, and scripts are stripped before anything downstream sees the page, so a page gets judged on what it says rather than the chrome around it. Pages with almost nothing left after that, nav-only stubs and error shells, are dropped at the door.

One limitation gets surfaced loudly instead of papered over. The default crawl reads static HTML, so a docs site that renders its content entirely through JavaScript can come back nearly empty. When that happens, the pipeline says so in plain language, names the likely reason, and can crawl by rendering pages through your own browser instead for sites that need it. A pipeline that silently produced zero tests from a JavaScript-rendered site would look identical to one that judged your docs untestable, and removing that ambiguity is exactly what the streamed diagnostics are for.

What does extraction refuse to turn into a test?

Anything that isn’t a discrete action a user takes in the UI. Extraction is the stage where a model reads each surviving page and pulls out intents, numbered steps, and the domain entities the page mentions, and its rules are ruthless about the boundary. Marketing copy is not a task. A conceptual overview without steps is not a task. API endpoint references, pricing and billing pages, changelog entries, and what’s-new announcements are all named exclusions. A page describing none of the above simply emits nothing, and the pipeline treats an empty result as a legitimate answer rather than a failure to try harder.

The contrast is easy to picture. A page called “About projects” that spends four paragraphs on philosophy emits nothing. A page called “Create a project” with numbered steps and the button names spelled out emits an intent, a procedure with those steps intact, and the entities the page mentioned. Both pages look like documentation. Only one of them describes something a browser agent could execute and a verifier could check.

The empty-result rule is the design decision I’d defend hardest. The extraction contract says that when information isn’t clearly in the docs, the model does not make it up, and an empty result beats a half-answer. Generation systems fail in two directions, refusing good input or inventing structure that was never there, and the second failure is far more expensive in testing. An invented procedure becomes an invented test case that fails against the real product for reasons nobody can diagnose.

Two size gates run alongside the task filter, both kept qualitative. Pages too thin to describe a real procedure are skipped outright, because nav indexes and redirect stubs never carry steps worth extracting. Very long pages are truncated with an explicit note rather than silently cut off, so the missing half is never mistaken for absent behavior.

For a page that survives, fidelity rules take over. Steps keep the doc’s exact wording for buttons, fields, and menu items. Steps the doc doesn’t describe don’t get invented. A procedure that comes back without any steps is dropped in validation, as is one that doesn’t attach to a real extracted intent. And a single page failing mid-extraction, a timeout or a malformed response, gets logged and skipped without killing the run, because one bad page shouldn’t cost you the rest of the site.

Which filters run at suite time instead of ingestion?

Three more: an auth strip, a merge, and a lint, all applied when tests get drafted rather than when docs get ingested.

The auth strip is the filter people notice first, because the progress stream calls it out, right down to telling you how many skills it removed. Skills that are purely about logging in, signing up, or resetting a password are stripped before drafting. The runner assumes you’re already signed in to your own browser profile, so a generated login test would be broken by design and would demand credentials the system never asks you for. The nuance is what survives: admin-side configuration flows like SSO setup stay in, because an admin configuring authentication is a real product task, while a user authenticating is a precondition.

The merge is quieter. Skills don’t map one-to-one onto tests. The drafting rules push related actions into end-to-end flows and skip trivial sub-actions already covered by a larger flow, so a topic with a dozen skills might emit a few tests rather than twelve. For a topic mixing reads and writes, the rules ask for a happy path, a variant exercising optional fields, and an edge case where an obvious one exists. That’s the honest scope of AI-generated edge case test coverage here: an edge case earns a test when the docs imply one, not because a quota demanded one.

The lint is the final cut and the one I consider least optional. The drafting rules already ban cross-test dependencies, wording that assumes a resource was created in a prior step, and hedged fallbacks about whether a resource exists without a setup prompt to guarantee it. A model can still slip. So a deterministic lint re-reads every drafted flow and drops the ones that assume data another test created, or that grade against a named resource nothing guarantees exists, recording the test and the reason. A flow like that isn’t a weak test, it’s a false alarm on a timer, destined to fail for environment reasons and to erode trust in the verdicts that matter. A bad test is worse than a missing one, and the lint is that sentence enforced in code.

What do different generation approaches accept, and what do they emit?

Manual authoring accepts whatever the author knows and emits documents, record/replay accepts a click-through and emits a recording, test-case-text generators accept a requirements doc and emit more text, and TestAutomate accepts a docs URL and emits runnable tests. The input contract is the clearest axis I know for comparing these, because the input decides how much filtering you get for free and how much junk you inherit downstream.

Every row has an honest advantage the table undersells. Manual authoring is the only approach where judgment about what matters commercially arrives for free, and no filter substitutes for a person who knows which flow bleeds revenue when it breaks. Record/replay needs no documentation at all, which makes it the fastest route to covering one flow that exists today, precisely where a docs-first pipeline has nothing to read. Text generators do fine when a current requirements document exists, and if you’re evaluating a PRD to test cases workflow while your PRDs are genuinely maintained, that route is defensible. A companion post examined why that document so often doesn’t exist for the features that need regression coverage most. And our own row inherits the quality of your docs wholesale. A page describing a flow wrong yields a test that faithfully checks the wrong thing, and a thin page describing a genuinely important task gets skipped, because no filter can know it mattered.

What’s left after the funnel is an editor’s job, not an author’s. On the one ingestion run we’ve recorded, a real docs site came out the far end as roughly 40 skills and a clustered draft suite, and the draft still included one test whose expected outcome was too loose until I tightened it by hand. I’d rather tell you that than pretend the funnel ends at perfect. Filters decide what’s worth drafting. Generation drafts it. A skeptical verifier grades every run afterward. Your judgment stays where it belongs, on the one question no filter can answer: whether the test that survived is checking something worth knowing.