Generate Test Cases From Documentation

I wanted to see how much of a regression suite I could get without writing a single test by hand, so I pointed TestAutomate at linear.app/docs and clicked ingest. That recorded run is the spine of this post, because the honest way to find out whether you can generate test cases from documentation is to watch one real attempt end to end, including the part where a drafted test had to be fixed.

You can generate test cases from documentation when two conditions hold. The docs describe discrete tasks a user performs in the UI, and something downstream turns each task into an intent-based test with an expected outcome a verifier can grade. TestAutomate does its AI test case generation in two stages, an ingestion pipeline that crawls a docs site and distills task-shaped skills, and a planner that clusters those skills into topics and drafts runnable test flows from them.

What follows is the mechanism behind each stage, the Linear run as the worked example, and the two places where a human reviewer still earns their keep.

Can you really generate test cases from documentation?

Yes, if the documentation actually describes tasks, and only up to a draft that deserves review. Testers have a name for whatever artifact tests are derived from, the test basis, and most teams reach for a requirements doc or a stack of user stories by reflex. I built TestAutomate’s ingestion around a different bet. Published product documentation is the one artifact that describes what actually shipped, in the vocabulary users see on screen, and it stays maintained because users depend on it.

The pipeline behind that bet runs in four stages: crawl, extract, classify, emit.

The crawl is deterministic, with no model involved. It fetches the site’s pages at a polite pace, stays inside the documentation section you point it at, and caches what it fetches, so re-running ingestion over unchanged pages downloads nothing twice. The whole walk is bounded, a run can’t wander off and crawl forever. Pages too thin to carry a real procedure, nav indexes and redirect stubs, get skipped before any model sees them, and very long pages are truncated rather than swallowed whole.

Extraction is where the reading happens. A model works through one page at a time and pulls out three things. Intents, the user-facing actions the page describes. Procedures, the numbered steps behind each intent, with the doc’s button and field names preserved rather than paraphrased. And entities, the app’s domain objects and how they relate to each other. The extraction step is deliberately conservative, built to come back empty-handed rather than guess, and strict about what counts as a task. A discrete action a user takes in the UI qualifies. Concept overviews, marketing material, API reference, and release notes don’t.

A classification pass then sorts the extracted material by the kind of handling it needs at runtime, and the emit stage writes each procedure out as a skill, one distilled task per skill, with its source page recorded. Every stage is independently re-runnable and cached, and one page failing mid-run gets logged and skipped instead of killing the ingestion. The design goal I held onto while building it was that a successful crawl should always produce something useful.

If you’re evaluating tools that generate test cases from documentation, AI extraction quality is where I’d aim your skepticism first. Ask what the tool refuses to extract. A pipeline that turns a pricing page into a test case isn’t being thorough, it’s being credulous.

What did the crawl keep from Linear’s docs?

Task pages, and very little else. I picked Linear because its docs are clean, an actual <article> tag around every page, none of the marketing-chrome-around-content mess you get on some doc sites. I figured if the ingestion pipeline was going to fall over anywhere, it wouldn’t be here. It didn’t. Within a few minutes I had a set of extracted skills, each one a distilled procedure: “create an issue,” “assign a project lead,” “set a cycle deadline.”

What surprised me was watching what it skipped. There’s a whole page in Linear’s docs about their API reference, and none of it showed up as a skill. Part of that is the crawl itself, whose source configuration keeps it on the end-user docs and excludes changelog and marketing paths outright. The rest is the extraction step, which throws out anything that isn’t a discrete UI action. Marketing copy, changelog entries, conceptual overviews with no steps, all filtered before a model ever gets asked to turn them into a procedure. That’s the right call. I didn’t want a test called “learn about Linear’s philosophy.”

Each surviving page went through the extraction pass with the actual button and field text preserved, plus the domain entities the page mentions, Project, Issue, Cycle, and how they relate, followed by the classification pass that sorts each intent by what kind of handling it needs at runtime.

The detail worth pausing on is that nothing in this stage invents anything. Extraction is a compression step, not a creative one. If a page describes no steps, no steps come out of it, and a page can legitimately contribute zero skills. That restraint matters two stages later, because a test drafted from invented steps doesn’t fail loudly. It fails by quietly testing something your product never claimed to do.

How do extracted skills become an executable regression suite?

In two stages you can watch from the dashboard, clustering and drafting, and none of it runs until you ask. Generation is triggered lazily, the first time you request a suite for an app that doesn’t have one, rather than as a side effect of ingestion. Progress streams to the dashboard as it works, so you see topics forming and tests landing while it happens.

Then came the part I actually cared about, turning ~40 skills into a runnable suite. I clicked “Generate suite” and watched it cluster the skills into topics. It grouped things like “Cycles & Planning” and “Issue Triage” on its own, without me defining categories anywhere. For each topic it drafted a handful of test flows, each with a prompt carrying the natural-language instruction and an expected_outcome that a separate verifier checks the run against afterward.

A drafted test is structurally identical to one you’d write by hand. It carries the same three required fields, an id, a prompt, and an expected outcome, plus optional setup and cleanup prompts, and the runner refuses to load any test missing its prompt or expected outcome. The drafts are plain natural language by hard rule. The generation contract bars code, JSON, YAML fragments, and selectors from every field, so what you review reads like instructions to a careful colleague, not a script. Expected outcomes come out as itemized end-state assertions, including negative ones such as confirming nothing else was modified, and any bullet prefixed [incidental] is barred from failing the run, so a missing success toast can’t outrank the point of the test.

One thing it got right without me asking was silently dropping every skill that was purely about logging in or signing up. Makes sense, since the runner assumes you’re already authenticated in your bound browser profile, so a generated test that tries to log in would just be broken by design. It didn’t drop SSO setup, though, because that’s an admin configuring something rather than a user authenticating. It’s a distinction I wouldn’t have thought to encode into a filter, but it’s exactly right.

The strictest rule in generation is self-containment. No draft may assume a resource was created in a prior step, because tests run independently and in any order. Anything that must already exist goes into an idempotent setup prompt that checks for the resource and creates a minimal one if it’s missing. And because a model can slip past its own instructions, a deterministic lint pass runs over every draft and drops flows that assume prior-step data or hedge about resources nothing guarantees. I wrote that lint after watching what hidden data dependencies do to chained tests. A bad test is worse than a missing one.

Executability is the other half of the phrase “executable regression suite,” and it’s the half most docs-to-test-cases tooling quietly drops. What lands here isn’t a spreadsheet of test titles. Each draft joins the same list as your hand-written tests and runs through the same machinery. An agent drives a real browser, a separate skeptical verifier grades the run against the expected outcome item by item, every run ends passed, failed, skipped, or blocked, and a failed first attempt is retried once on a stronger model before the red is treated as real. That’s the full agentic testing loop, and generated tests get no special leniency inside it. AI regression testing earns trust at the verdict, not at the drafting step.

Which source of truth should you generate tests from?

The one that’s still true on the day the suite runs. A regression suite exists to re-verify shipped behavior after every change, so the artifact you generate it from should describe shipped behavior, not intended behavior. That’s the lens I’d use on the four sources teams actually reach for.

Honest caveats, because that table flatters my choice. Requirements docs and user stories are better than product docs at capturing intent, what a feature is for and what must never happen, and the trade-offs between those inputs get a full treatment in generating test cases from requirements with AI. Docs lag releases at plenty of companies, and a new feature can be live for a sprint before its page exists. Internal tools often have no public docs at all, in which case this whole approach has nothing to eat. Recordings deserve their due too. For a stable flow that never changes, a recording is an honest, if brittle, source. My claim isn’t that docs are perfect. It’s that among the four, they’re the only source that is simultaneously maintained, written in the user’s vocabulary, and specific enough to execute from.

Where does generating tests from docs fall short?

On assertion strictness, and on anything your documentation doesn’t say. The recorded run produced a clean example of the first. One of the drafted tests for “Cycles” had an expected_outcome that was too loose, something like “the cycle is created successfully,” and it passed on a run where the cycle got created with the wrong date range. That’s a generated-test problem, not a bug in Linear, and it’s exactly the kind of thing the schema expects a human to tighten before trusting it in a real suite. I rewrote the expected outcome to name the actual date range and re-ran it, and it caught the same issue immediately.

That incident is why I treat generation as a drafting step, never a publishing step. The review I’d actually do on a generated suite is narrow. Read every expected outcome and ask whether a wrong-but-plausible result could still satisfy it, then tighten the ones where it could. Check that the setup prompts guarantee whatever the prompts assume. Cut the flows you know don’t matter, applying the same editorial instinct behind deciding which generated test ideas deserve to survive. Writing assertions that can actually lose is its own craft, and there’s a separate checklist for expected results that hold up on this blog.

The second limit has no workaround inside the product. Coverage is bounded by what’s documented. An undocumented admin corner produces no skills, so it produces no tests, and no generation pass can know that your revenue depends on a flow your docs never mention. The suite you get is a floor, the documented core of the product, executable on day one. Someone who knows the product still decides what matters above that floor.

It’s not a hands-off “regression suite in one click,” but it is a genuinely useful first draft. It turned documentation I would’ve had to read anyway into 15 minutes of review-and-edit instead of a day of writing test prompts from scratch. Docs to test cases is a real pipeline now, not a metaphor. Just remember that the draft’s confidence and the draft’s correctness are two different things, and a skeptical review pass is what turns the first into the second.