How we decide a feature is “verified working”

Our operating model behind the launch-readiness percentage on the roadmap.

Plain version: a shipped feature only counts toward our headline percentage once an automated end-to-end test has passed at least once, a manual smoke-run has been recorded in the last 30 days, and there are no open regressions against it.

The three gates

Every feature in the roadmap passes through the same three checks before it earns the prod badge:

  1. End-to-end test exists — a spec file under tests/e2e/<feature_key>.spec.ts drives the real UI or API in a clean environment.
  2. Auto-test is green on main — the latest run in CI is pass and the consecutive-failure count is zero.
  3. Manual smoke-run inside 30 days — a human has clicked through the documented happy path on staging and recorded the result. After 30 days without a smoke-run, the feature drops back to untested until someone re-runs it.

What each badge actually means

prod  Auto-test passed on the most recent CI run, smoke-run inside 30 days, no open regressions. Counts toward the headline launch-readiness percentage.

known issue  Auto-test or smoke-run is failing, or a customer-visible regression has been reported and accepted. Shown on the public status page and excluded from the percentage.

untested  No automated spec yet, or the 30-day smoke-run window has lapsed. The feature still exists in the product — we just won’t claim it’s verified.

Why we set the bar here

RMM tools have a quiet history of shipping features that “technically work” in a demo but fail in production. We don’t want to be one of those. The 30-day window forces us to re-prove the basics regularly — it’s the easiest way to catch a feature that silently broke during a refactor.

The downside is honest: our percentage will sometimes drop after a release, not because we shipped bugs, but because a smoke-run lapsed or a flaky test landed. We’d rather have that conversation than hide it.

Where the numbers come from

The headline percentage on /roadmap reads from /api/public/launch-readiness. That endpoint counts only shipped features whose latest test run is green and whose smoke-run is inside the 30-day window. Anyone can call it — no sign-in required.

If you find a feature labelled prod that doesn’t actually work for you, please tell us. The fastest way is the “Report a bug” button on any dashboard page, or email support@patchpilot.co.uk.

Back to PatchPilot home