Design an A/B test for a product feature

Design a statistically rigorous A/B experiment to validate product decisions with real data before full rollout.

Workflow · Product DevelopmentRole · Product Manager●●● AdvancedUpdated 2026-07-31

The prompt

Copy and customize

prompt.txt
**Role:** You are a Product Manager designing an A/B test for {feature_being_tested} on {product_name}.

**Context:**
Poorly designed A/B tests lead to false conclusions that drive the product in the wrong direction. A rigorous experiment starts with a clear hypothesis, defines success metrics before running the test, calculates the required sample size to achieve statistical significance, and establishes guardrails to protect against harm during the test.

**Task:**
Design a complete A/B test plan. Apply step-by-step reasoning: (1) define the hypothesis, (2) determine the primary and guardrail metrics, (3) calculate required sample size, (4) define the control and variant, (5) establish stopping rules.

**Input Available:**
- {feature_being_tested}: What is being tested
- {product_name}: Product name
- {hypothesis}: Your belief about what will happen and why
- {primary_metric}: The one metric that determines success or failure
- {current_baseline}: Current value of the primary metric
- {minimum_detectable_effect}: The smallest improvement that would be worth shipping
- {daily_users}: Number of users who encounter this surface daily
- {guardrail_metrics}: Metrics that must not degrade during the test

**Output Format:**
1. Hypothesis statement: If [we do X], then [Y metric] will [increase/decrease] by [Z%] because [mechanism]
2. Test design: Control (what users see today) vs. Variant (what you're testing)
3. Primary success metric with definition and measurement method
4. Guardrail metrics with acceptable variance thresholds
5. Sample size calculation: Required users per variant to achieve 95% confidence
6. Estimated test duration: Based on daily users and required sample size
7. Audience allocation: What % traffic to each variant and why
8. Stopping rules: Pre-defined conditions to stop early (positive and negative)
9. Analysis plan: How you will analyze results and interpret them
10. Rollout plan: What happens if the variant wins / loses / is inconclusive

**Guardrails & Quality Control:**
- Do not peek at results before reaching required sample size — this inflates false positive rates
- Guardrail metrics must be defined before the test starts, not after analyzing results
- Run the test for a minimum of 1 full business cycle (usually 2 weeks) to avoid day-of-week bias
- An inconclusive result is valuable information — build a clear interpretation for this case

How to use

Run this prompt in four steps

  1. 1Have a data analyst review the sample size calculation and stopping rules before launching.
  2. 2Document the full test design before launching — do not change the success metric mid-test.
  3. 3Set up monitoring alerts for guardrail metrics from day 1 of the test.
  4. 4Share the test plan with stakeholders before launch to prevent interference with results.

When to use

When to use this prompt

Use before shipping major UI changes, new features, or pricing experiments that affect conversion or retention.

Limitations · Worth knowing

This prompt has limitations you must understand.

A/B tests require sufficient traffic to reach significance. For products with <1,000 daily active users, testing timelines become impractically long.