Experiments
Success metrics and guardrails for product experiments
Updated
Most experiments do not fail because the idea was wrong. They fail because nobody agreed in advance what “worked” would mean, so any result can be read as a win. This article covers the three decisions to make before you start: the primary metric, the target, and the guardrails.
Decide before you measure
Once results start coming in, everyone involved wants the idea to have worked. That is human, and it is exactly why the success criteria must be written down and locked before the experiment starts. Changing the target after seeing the data is not analysis; it is choosing the answer.
1. One primary metric
Pick the single number that tells you whether the hypothesis was right. It should come straight from the hypothesis’s outcome. If the hypothesis is about prescribing time, the primary metric is prescribing time, not page views or satisfaction.
- One, not five. With five metrics, one will usually move by chance, and that one becomes the story.
- Close to the behaviour. “Median minutes to complete an infusion order” beats “engagement”.
- Measurable now. If you cannot measure the baseline today, fix that before you start.
2. A target with a baseline and a timeframe
Primary metric: median infusion prescribing time Baseline: 15 minutes (last 30 days, two pilot units) Target: under 5 minutes Timeframe: 6 weeks after release
A target without a baseline is meaningless, and one without a timeframe never fails. Set the target at the level that would change your decision. If a drop to 12 minutes would not be worth building, do not count 12 minutes as success.
3. Guardrails that must not get worse
A guardrail is a metric you are not trying to improve but must not harm. Faster prescribing is no win if errors rise. Guardrails catch the cases where the primary metric improves for the wrong reason.
Guardrail: infusion orders corrected after submission Baseline: 3.1% Limit: must stay at or below 3.5%
- Quality and safety: errors, corrections, incidents
- Experience: support tickets, abandonment, complaints
- Business health: retention, revenue, cost to serve
Reading the result honestly
With the criteria set in advance, the verdict is almost mechanical:
- Validated: the primary metric hit its target and no guardrail was breached.
- Invalidated: the primary metric missed its target.
- Not validated, with a warning: the primary metric hit its target but a guardrail was breached. Something worked, but at a cost you said you would not pay.
Whatever the verdict, write down what you learned as a finding and update your confidence in the hypothesis. An invalidated hypothesis that saves you three months of building is a good outcome.
Not every experiment is an A/B test
The same discipline applies to small bets. A prototype test with five users can have a primary metric (“at least 4 of 5 complete the order without help”) and a guardrail (“nobody picks the wrong drug”). Match the size of the experiment to the risk of the decision, and save full A/B tests for decisions that are expensive to reverse.
Keep your evidence and decisions linked
Groundwork takes your team from customer insights to evidence-based requirements. Free to start, with your whole team.
Get started