Statistical significance checker for low-conversion experiments

Purpose

Determine whether a low-conversion-rate experiment has actually reached a trustworthy conclusion — or whether the observed lift is noise dressed up as significance. Produces a rigorous read on statistical power, required sample size, minimum detectable effect, and the specific failure modes (peeking, high variance, multiple comparisons) that mislead teams testing rare events at 0.5%–5% base rates.

When to Use

Invoke this skill when the user asks to:

• Check whether an A/B test result is statistically significant with a low base conversion rate
• Determine if an experiment has enough sample size or statistical power to call
• Validate a suspiciously large lift on a rare-event metric (signup, deposit, fraud flag, checkout)
• Assess how long a test must run given current traffic and conversion volume
• Diagnose why two tests keep flipping between "winner" and "no effect"
• Sanity-check a p-value before shipping a change to production

Inputs Required

Before starting, collect:

• Metric under test: The conversion event and its type (binary rate, revenue-per-user, count)
• Base (control) conversion rate: The historical or control-arm rate — critical for rare-event math
• Sample sizes per arm: Users/sessions in control and each treatment
• Observed conversions per arm: Raw event counts, not just percentages
• Observed lift/effect: The measured difference (relative % or absolute)
• Minimum meaningful effect: The smallest lift worth shipping (MDE the business cares about)
• Test setup: Number of variants, whether results were checked mid-flight, significance level and power targets (default α=0.05, power=0.80 two-sided)

If MDE or power targets are missing, default to α=0.05, power=0.80, and infer a business-relevant MDE from the base rate (typically a 5–15% relative lift for rare events) and state the assumption.

Instructions

Step 1: Reconstruct the Raw Data

Never trust reported percentages alone.

• Convert every arm to raw counts: successes and total N per arm
• Recompute observed rates from counts to catch rounding or reporting errors
• Flag if any arm has fewer than ~30 conversions — normal-approximation math becomes unreliable below this and exact/Bayesian methods are needed
• For revenue or count metrics, note the mean and variance separately (high variance is the silent killer in these tests)

Step 2: Assess Sample Size Adequacy

Compute the sample size the test should have had.

• Required N per arm (two-proportion z-test approximation): n = (Z_α/2 + Z_β)² · [p₁(1−p₁) + p₂(1−p₂)] / (p₂ − p₁)² where p₁ = base rate, p₂ = p₁ × (1 + MDE), Z_α/2 = 1.96, Z_β = 0.84 at 80% power

• Compare required N to actual N per arm → report % of required sample achieved

• For rare events, note that required N scales roughly with 1/base_rate — a 1% base rate needs ~5× the sample of a 5% base rate for the same relative MDE

Step 3: Compute Statistical Power (Post-Hoc, With Caveats)

Estimate whether the test could have detected the effect it aimed for.

• Given actual N, base rate, and target MDE, back out achieved power
• Label the test:
• Adequately powered (≥80%): a null result is meaningfully informative
• Underpowered (50–79%): a null result is inconclusive, not evidence of no effect
• Severely underpowered (<50%): the test cannot reliably detect the target effect — any "winner" here is high-risk
• Warn against interpreting post-hoc power as validation of a significant result; use it only to interpret null results and future planning

Step 4: Test Significance Correctly

Run the appropriate test for the data shape.

• Binary rate, adequate counts: Two-proportion z-test → report z-statistic, two-sided p-value, and 95% CI on the absolute difference
• Binary rate, low counts (<30 conversions per arm): Recommend Fisher's exact test or Bayesian Beta-Binomial; report a credible interval on lift
• Revenue/count metric: Welch's t-test (unequal variance) or Mann-Whitney; flag that outliers/whales distort means and recommend capping or log-transform
• Always report the confidence interval on lift, not just the p-value — a CI spanning zero or spanning both trivial and huge effects tells the real story

Step 5: Screen for the Four Rare-Event Failure Modes

Explicitly check each:

• Peeking / optional stopping: If results were checked mid-flight and stopped at significance, the true false-positive rate is inflated well above 5%. Recommend a sequential correction (alpha-spending / SPRT) or discount the p-value
• Multiple comparisons: If ≥3 variants or ≥2 metrics were tested, apply Bonferroni (α/m) or note the family-wise error inflation and give the adjusted threshold
• High variance / low signal: If per-arm variance is large relative to the effect (common in revenue-per-user with heavy tails), the nominal p-value is optimistic
• Novelty / ramp effects: For fintech/e-commerce, flag if test duration is under one full business cycle (weekly seasonality, payday cycles, weekend patterns)

Step 6: Render the Verdict and Decision

Deliver a clear call, not a hedge.

• One of: Ship, Do Not Ship, Keep Running, or Inconclusive — Redesign
• State required additional sample and estimated additional runtime at current traffic to reach adequate power
• Give the MDE the current sample can actually detect (so the team knows what effect size, if any, is trustworthy today)

Output Format

• Structure: Diagnostic report with a headline verdict at top, then the six analysis sections, then a decision block
• Length: 700–1,000 words
• Format: Structured document with a summary table (per-arm N, conversions, rate, CI) plus prose reasoning
• Sections: Verdict headline → Raw data reconstruction → Sample size adequacy → Statistical power → Significance test → Failure-mode screen → Decision & required runtime
• Tone: Rigorous experimentation lead — precise, numeric, unwilling to over-claim; explains the math but always lands on an actionable call

Quality Checklist

Before delivering output, verify:

• [ ] Raw conversion counts (not just percentages) are reconstructed and rates recomputed for every arm
• [ ] Required sample size per arm is calculated with stated α, power, base rate, and MDE
• [ ] Achieved power is reported with an explicit adequacy label (adequate / underpowered / severely underpowered)
• [ ] The significance test chosen matches the data shape (z-test vs. Fisher's exact vs. Welch's), and a confidence interval on lift is reported alongside the p-value
• [ ] All four rare-event failure modes (peeking, multiple comparisons, high variance, novelty) are explicitly checked
• [ ] A single clear verdict is given, with required additional sample and runtime when the test isn't ready

Notes

• The single most common error in low-conversion testing is calling a winner on an underpowered test after peeking — treat any mid-flight-stopped significant result as suspect until corrected
• At base rates under 2%, favor Bayesian Beta-Binomial or Fisher's exact over normal approximations; the CI is more honest and doesn't break with low counts
• Revenue-per-user in e-commerce is heavy-tailed — a single whale can manufacture "significance"; always inspect variance and consider a capped or log-transformed metric alongside the raw one
• Runtime estimates should use realistic daily traffic and conversion volume, not idealized numbers — a test needing 400k users per arm at 5k daily eligible users is a 3-month commitment worth surfacing up front

How to use this skill

This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.

Installing in Claude Code:

• ~/.claude/skills/<folder-name>/SKILL.md — auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)
• <your-project>/.claude/skills/<folder-name>/SKILL.md — auto-loads only when working in that project (recommended for project-specific skills)

Restart Claude Code or open a new session to register.

Using in Claude.ai, ChatGPT, Gemini, or other AI tools:

1. Drag this file into a new chat, or copy and paste its contents
2. Send your task or request as your next message — the AI will apply the skill where it fits

For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills