How to start
Install the skill in Claude Code, then trigger it with any of these phrases:
/ab-test design an A/B test plan a split test set up an experiment A/B Test
Design statistically valid A/B tests with clear hypotheses, proper sample sizing, and analysis plans.
A Claude Code skill that builds complete split test plans — hypothesis, sample size, variant design, traffic allocation, and pre-committed analysis criteria.
/ab-test design an A/B test plan a split test How it works
1
Test Context
The skill asks what you want to test, why, where, current performance baseline, success metrics, and minimum detectable effect.
2
Hypothesis & Sample Size
Builds a structured hypothesis and calculates required sample size and test duration based on your traffic and baseline conversion rate.
3
Variant Design & Allocation
Designs control and variant(s) with specific changes, traffic allocation strategy, and guardrail metrics to monitor.
4
Analysis Plan & Export
Pre-committed analysis plan with decision criteria, saved as a structured markdown document — no post-hoc rationalization.
Why use it
Structured Hypotheses
Clear 'if-then-because' hypotheses that make test results actionable regardless of outcome.
Sample Size Calculations
Proper statistical sizing based on baseline rate, desired lift, and significance level — no guessing.
Pre-committed Analysis
Decision criteria defined before the test runs — preventing p-hacking and post-hoc rationalization.
Guardrail Metrics
Secondary metrics monitored to catch unexpected negative effects on other parts of the funnel.
Use Cases
Headline Testing
Test headline variations with proper statistical rigor instead of gut feelings.
CTA Optimization
Test button copy, placement, and design with pre-committed success criteria.
Pricing Experiments
Test pricing changes, tier structures, or billing options with controlled experiments.
Layout Changes
Test page layout and information hierarchy changes with proper sample sizing.
Onboarding Flows
Test different onboarding sequences with activation rate as the success metric.
Email Subject Lines
Test email subject lines with statistically valid sample sizes and clear winners.