Purpose
Turn raw experiment output into a defensible decision by interpreting the p-value, confidence interval, effect size, and power together — not in isolation — and stating whether the result is trustworthy, practically meaningful, and ready to act on. Solves the two most common experiment-analysis failures: shipping on noise, and killing winners that were merely underpowered.
When to Use
Invoke this skill when the user asks to:
- Interpret A/B test or experiment results and decide whether to ship
- Explain what a p-value or confidence interval actually means for a business decision
- Assess whether a "non-significant" result means no effect or an underpowered test
- Sanity-check whether an observed lift is real or likely noise
- Translate a statistics-heavy experiment readout into a stakeholder summary
- Determine whether an experiment should be stopped, extended, or called
Inputs Required
Before starting, collect:
- Metric tested: The primary success metric (conversion rate, revenue per user, retention, etc.) and whether it's a proportion or a continuous mean
- Variant results: Control and treatment values — rates or means, plus sample size per group
- Statistical output: p-value and/or confidence interval, and effect size if available (absolute and/or relative lift)
- Test setup: One-tailed or two-tailed, significance threshold (default 0.05), and whether the test was pre-registered or peeked at
- Business context: What decision hinges on this — ship, roll back, iterate — and the cost/risk of being wrong
- Practical threshold: The minimum lift that would justify shipping (the "minimum detectable effect" that matters to the business)
- Test duration & guardrails: How long it ran, and any secondary/guardrail metrics to watch
If the practical threshold or test setup is missing, assume a two-tailed test at α = 0.05, treat any positive lift as directionally interesting, and flag that a business-defined minimum effect is needed for a confident call.
Instructions
Step 1: Establish What Was Actually Measured
Restate the experiment in one sentence: metric, direction of the hypothesized effect, and observed values for control vs. treatment. Compute the observed effect if not given:
- Proportions: absolute lift = treatment rate − control rate; relative lift = absolute lift / control rate
- Means: absolute difference and % change Flag immediately if sample sizes are lopsided, tiny (<100 per arm for proportions near typical rates), or if the metric is a ratio-of-ratios that inflates variance.
Step 2: Interpret the p-value Correctly
State what the p-value does and does not mean, applied to this test:
- A p-value is the probability of seeing this result (or more extreme) if there were no true effect — it is not the probability the hypothesis is true, nor the probability the result will replicate.
- Compare to the threshold: below α = statistically significant; above α = not significant at this bar.
- Peeking penalty: if the user checked results repeatedly without a sequential-testing correction, the true false-positive rate is inflated — treat a borderline p-value (0.01–0.05) with suspicion.
- Multiple comparisons: if many metrics or variants were tested, note that one "significant" result among many is expected by chance and recommend a correction (Bonferroni or Benjamini-Hochberg).
Step 3: Read the Confidence Interval as the Real Answer
The CI is more informative than the p-value — interpret it directly:
- State the interval in business units (e.g., "true lift is between −0.4% and +3.1% conversion").
- If the CI crosses zero: the direction of the effect is uncertain — cannot claim a win.
- If the CI is entirely positive but includes trivial values below the practical threshold: statistically real but possibly not worth shipping.
- If the entire CI clears the practical threshold: strong evidence of a meaningful effect.
- Width tells power: a very wide CI signals an underpowered test — the honest conclusion is "we don't know yet," not "no effect."
Step 4: Assess Effect Size and Practical Significance
Separate statistical significance from business relevance:
- Compare the observed effect (and the CI's lower bound) against the practical threshold from inputs.
- For continuous metrics, note the standardized effect size (Cohen's d) if derivable: ~0.2 small, ~0.5 medium, ~0.8 large.
- Explicitly call out the big-sample trap: with enough traffic, trivial effects become "significant" — significance ≠ importance.
- Explicitly call out the small-sample trap: a large observed lift with a wide CI is likely regression-to-the-mean and should not be trusted.
Step 5: Evaluate Statistical Power (Especially on Null Results)
For any non-significant result, do not conclude "no effect" without checking power:
- Estimate whether the sample size was sufficient to detect the practical threshold at ~80% power.
- If the test was underpowered, the correct conclusion is "inconclusive — the test could not have detected the effect we care about," and the recommendation is to extend or re-run with a proper sample.
- If the test was well-powered and still null, that is meaningful evidence of no practical effect — a legitimate "don't ship."
Step 6: Render the Decision
Classify the result into one of four verdicts and justify it:
- Ship it: CI clears the practical threshold, well-powered, no guardrail regressions.
- Don't ship: well-powered, effect is null or below threshold, or guardrails degraded.
- Keep running: promising direction but underpowered / CI too wide / duration too short to trust.
- Ship with caution: significant but marginal, borderline power, or peeking occurred — recommend a confirmation test or staged rollout. Then check guardrail/secondary metrics for regressions that would override a positive primary result.
Step 7: Write the Plain-Language Business Summary
Produce a stakeholder paragraph with zero jargon: what was tested, what happened, how confident we are, and what to do next — expressed in dollars, users, or percentage points, not p-values.
Output Format
- Structure: Diagnostic walkthrough followed by a one-line verdict banner and a jargon-free stakeholder summary
- Length: 600–900 words
- Format: Structured document with short labeled sections; the interval and verdict presented as a compact table or bolded callout
- Sections: What Was Measured → p-value Reading → Confidence Interval → Effect Size & Practical Significance → Power Check → Verdict (Ship / Don't Ship / Keep Running / Ship with Caution) → Plain-Language Summary → Recommended Next Step
- Tone: Analytical and plain-spoken — a senior analyst explaining the result to a PM, confident enough to make a call and honest about uncertainty
Quality Checklist
Before delivering output, verify:
- [ ] The p-value is interpreted as a false-positive probability — not as the probability the hypothesis is true
- [ ] The confidence interval is stated in business units and its relationship to zero AND the practical threshold is addressed
- [ ] A non-significant result is not called "no effect" without an explicit power assessment
- [ ] Statistical significance and practical significance are distinguished, with the observed effect compared to the business threshold
- [ ] Exactly one of the four verdicts is stated clearly, with justification tied to the numbers
- [ ] The plain-language summary contains no statistical jargon and expresses impact in business terms
Notes
- The confidence interval, not the p-value, is the most useful single number — a p < 0.05 with a CI spanning trivial-to-huge lift is far weaker than the p-value suggests.
- Peeking and multiple comparisons are the most common silent inflators of false positives — always ask whether results were checked repeatedly or across many metrics.
- "Underpowered null" and "well-powered null" lead to opposite recommendations (keep running vs. don't ship); the power check in Step 5 is the pivot point and should never be skipped.
- When the business threshold is unknown, the interpretation still holds but the ship/don't-ship line is the analyst's weakest link — press for that number.
How to use this skill
This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.
Installing in Claude Code:
~/.claude/skills/<folder-name>/SKILL.md— auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)<your-project>/.claude/skills/<folder-name>/SKILL.md— auto-loads only when working in that project (recommended for project-specific skills)
Restart Claude Code or open a new session to register.
Using in Claude.ai, ChatGPT, Gemini, or other AI tools:
- Drag this file into a new chat, or copy and paste its contents
- Send your task or request as your next message — the AI will apply the skill where it fits
For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills