Checksum alternative: Two agent stories. Two different oracles.
Checksum's pitch — "tests that generate, run, and fix themselves" — is one of the most complete agent stories in AI testing, and this page takes it seriously. The real split is what the agent treats as truth: where expectations come from, who may rewrite a failing test, and whose browser holds the keys.
Last reviewed · See a real graded run
What Checksum genuinely gets right
Real Playwright, in your repo
Standard Playwright committed to your repo — "no lock-in, no proprietary format," per their pricing page — and their docs say the tests can be run with Playwright directly. Rarer still, intent lives beside the code as a human-readable story file. If you eject, you keep working tests on their open-source runtime package, with the AI layer switched off. Few rivals can say that.
An agent that ships verified work
Their pipeline plans, implements, reviews its own output, then verifies the tests pass by actually running them before opening a PR. The CI Agent claims 50–200 tests per pull request, targeting exactly what changed — their number, and a differentiated one.
Triage that names the culprit
On failure, their auto-heal agent reads screenshots, error messages and stack traces plus the test and app code, and decides: broken test, or real bug — tagging genuine bugs in source, with videos, screenshots and Playwright traces attached. More honest failure handling than most of this category.
"Healed" is a verdict that edits the test
Checksum's runs report three outcomes: pass, fail, and healed. Per their auto-healing docs, when the triage agent decides a failure is a test issue — drifted selector, timing, stale setup, assertion drift — it fixes the test and opens a PR. Their platform page claims roughly 70% of failures resolve autonomously.
But notice what healing an assertion means. Their own docs describe the fix as updating "expected values to match current app behavior." The assertion is the oracle — the recorded definition of correct. An agent that rewrites it until the run goes green has quietly changed what correct means, checked only by whoever reviews the heal diff. The maintenance burden didn't vanish; it became code review of an agent's edits to your safety net.
- Their grammar: pass / fail / healed — where "healed" means the agent modified the test and it now passes.
- Our grammar: pass / fail / blocked. The runner never rewrites an expectation. Renamed-but-equivalent UI passes with an advisory flag; a pass on retry is labeled "passed on retry", never laundered.
- Failure confirmation: no failure is reported until a second, stronger model re-drives the journey and agrees. Nobody else does two-model failure confirmation.
A blocked environment gets its own verdict, too: a dead staging server can never be a false fail — or get "healed" into a different test.
Every documented input is the implementation
Let's be accurate here, because Checksum's story has matured. Per their docs, detection draws on three inputs: source-code analysis of your connected repo; an optional-but-recommended JavaScript SDK recording real user sessions — clicks, navigation, input patterns; and direct scanning of the live app for key journeys. The session data, their docs note, feeds both detection and generation.
The structural point survives the update: all three inputs observe the implementation — the code as written, the app as running, users using the app as it is. Whatever the software currently does becomes the expected behavior. A feature that shipped computing the wrong thing on day one produces tests that enshrine the wrong thing — passing, verified, healed forever after.
- Their sources: repo code, live-app scanning, and (optionally) recorded user sessions — the app grading itself.
- Our sources: your docs and help center as the expectation authority, your repo for enumeration, the live accessibility tree for reality — expectations independent of the implementation under test.
- Why it matters: verdicts grade the app against stated intent, not its own current behavior. No commercial rival ingests documentation.
Day one, we hand you a reconciliation findings report — everywhere your docs, code and running app disagree. That report is only possible when expectations come from somewhere other than the app.
Whose browser, whose credentials
Checksum documents two execution modes: your own CI runners (npx checksumai test in GitHub Actions or GitLab), or their cloud executing the run. Either way, test generation — including the verify-by-running step before each PR — happens in their cloud, which their docs describe as running "24/7, generating and verifying tests autonomously." Sign-in rests on "test users": credentials entered in their environment settings, which Checksum stores and uses to log into your application. Whether their cloud can reach a VPN'd private environment is not documented, so we won't assert it either way; the documented path to a private network is your own CI.
- TestAutomate drives your actual Chrome. Real logged-in sessions, behind your VPN and SSO, on real staging data — the browser doing the testing already has access. No vendor-held test accounts, no security-exception ticket.
- Unattended runs get both paths. A named Auth Catalog per app — email/password, Google, Okta, magic links, TOTP — encrypted per org, drives scheduled runs on a self-hosted runner or cloud browsers, with a live view and click-to-take-over when SSO or 2FA needs a human.
- Mid-run questions go to you. The agent asks a human when reality is ambiguous, and every tracker write is confirm-gated — an ask-user channel, not an autonomous guess.
Their your-own-CI mode is reasonable pipeline design. It is still a scripted sign-in with test-user credentials — held in their settings, or handed to their CLI as your CI secrets — into an environment your own browser was already inside.
Four design choices, one theme
Expectations from the docs
Draft suites come from documentation, repo and runtime — docs as the authority on intended behavior. The app is graded against a promise, not a recording of itself.
Nothing rewrites the test
Intent is a prompt plus an expected outcome in versionable YAML — no selector layer, so nothing to heal and no heal diff to review. Drift surfaces as an advisory, not an edit.
Test from inside the wall
Your Chrome, your live session, your VPN. The test-user credential handoff their environment config requires does not occur for attended runs.
Charge nothing on the meter
Bring your own key and model spend passes through at provider list price with a live dollar meter and a hard ceiling. No key? Managed credits at provider price + 15%, both numbers shown.
Side by side
| Checksum | TestAutomate | |
|---|---|---|
| How tests are authored | Background agents generate Playwright tests from repo analysis, live-app scanning and an optional user-session SDK; delivered as pre-verified PRs | Plain English — typed or voice-dictated; instant draft suites from docs, repo and runtime |
| What is stored | A story file (intent, markdown) + a Playwright spec with "grounded selectors" (test IDs, roles, text), per-action metadata and recorded network HARs | Intent: prompt + expected outcome in versionable YAML — no selector layer exists in the artifact |
| Where expectations come from | The implementation — source code, the running app, and optionally recorded user sessions | Docs as an independent expectation authority; repo for enumeration; runtime tree for reality |
| Where tests execute | Your CI runners or their cloud; test generation and its verify-by-running step happen in their cloud in both modes | Your own Chrome; self-hosted runner or cloud browsers for unattended runs |
| VPN'd staging & SSO | Reachable when runs execute on your own CI runners; their cloud's private-network reach is not documented | Already inside — your browser, your session, your network |
| Sign-in model | "Test users" — credentials entered in their environment settings; Checksum stores them and logs into your app during execution | Your real logged-in session; named Auth Catalog (encrypted per org) for unattended runs |
| Verdict grammar | Pass / fail / healed — a healed run is one the agent modified until it passed | Pass / fail / blocked, advisory flags, "passed on retry" labels — expectations are never rewritten |
| Failure handling | Auto-heal triage: test issue → auto-fix + PR; app bug → bug verdict and @bug tag; "~70% of failures resolving autonomously" (their figure) | A stronger model re-drives every failure and must agree before it is reported |
| Bug-tracker loop | No Jira or Linear integration documented; Slack / Teams / Discord / Google Chat notifications, @bug-tagged tests, health dashboard | Dedupe against the live tracker, confirm-gated filing with evidence attached, close-fixed transitions after re-verification |
| Pricing model | Sales-led — no public dollar figures (their pricing page, as of Sep 1, 2026); priced per maintained workflow ("up to 30 actions") in tiers of 50 / 200 / 400+ | BYOK at provider cost — never marked up — with live meters and hard budget ceilings; managed keys at provider price + 15%, shown |
| Test export | Genuinely strong: standard Playwright in your repo, atop their open-source runtime package; healing, AI recovery and mock playback require their CLI + cloud | Portable human-readable YAML, one-click full-suite export; the entire stack self-hosts |
| Platform coverage | Web via Playwright; an API Agent on the top tier (their claim); no mobile/native documented | Web-first — your Chrome, your sessions; no mobile/native either |
One footnote in fairness: their no-per-seat, no-per-run pricing shape is customer-friendly — what is absent is any number before a sales call, and with the workflow unit capped at 30 actions, how many workflows your suite needs is itself a sizing conversation. Their export story earns its credit twice: the code is genuinely portable; the maintenance layer is the subscription.
TestAutomate vs Checksum, asked directly
Doesn't Checksum already avoid lock-in?
For the test code, yes — standard Playwright in your repo, on an open-source runtime package, is one of the best export stories in the category. But auto-healing, AI recovery and HAR-based mocking require their CLI, API key and cloud: you keep the tests and lose the thing keeping them alive. TestAutomate's artifact is intent in portable YAML with one-click full-suite export, and the whole stack (Node + Postgres) self-hosts — the maintenance model leaves with you too.
Checksum fixes failing tests automatically — isn't that less work?
Less work than hand-repairing selectors, certainly. But the heal loop exists because their artifact contains something that decays — a Playwright spec bound to the app's structure — and "healed" means an agent edited the test, assertions included, reviewed afterward in a PR diff. Ours stores no selector layer, so there is nothing to heal: every run re-reads the live page against the stated expectation, renamed-but-equivalent UI passes with an advisory flag, and no failure reaches you until a second, stronger model re-drives it and agrees.
Can Checksum test apps behind a VPN?
On your own CI runners, yes — your runners are inside your network, and that mode is documented. Whether their cloud execution can reach a private environment is not documented, so we won't claim it either way; test generation and its verify-by-running step happen in their cloud in both modes, and their environment settings hold the "test user" credentials used to sign in. TestAutomate runs in your own Chrome — already behind the VPN, already signed in, already looking at real staging data.
Which one costs less?
We can only compare what is published, and as of September 1, 2026 Checksum publishes no dollar figures — pricing is per maintained workflow (an E2E test of up to 30 actions), in tiers of 50, 200 and 400+, numbers arriving on a sales call. TestAutomate passes model spend through at provider list price on your own key — no commercial QA platform offers BYOK at all — with a live dollar meter and a hard ceiling; managed credits show provider price + 15% explicitly. Ours is arithmetic you can check today.
We already have a Playwright suite. Why would we look at TestAutomate?
Keep it — and let us audit it. Our free existing-suite assessment reads a Playwright or Cypress suite and reports coverage gaps and flake patterns, alongside a reconciliation report of where your docs, code and running app disagree. That second document is the argument: Checksum's inputs can only tell you whether the app still does what it did. Docs-grounded expectations tell you whether it ever did what you promised.
Get a verdict no agent rewrote
Point TestAutomate at the staging app behind your VPN, in your own logged-in Chrome, and get a verdict graded against what your docs promised — replay attached, at provider list price.