Purpose
Turn a proposed product change into a launch-ready experiment design specification — falsifiable hypothesis, unit of randomization, metric tree, powered runtime, and a ranked risk register — so analysts stop reinventing the design doc for every test and stop shipping experiments that can't actually answer the question.
When to Use
Invoke this skill when the user asks to:
- Design an A/B test or controlled experiment for a product change
- Write a hypothesis and pick primary, secondary, and guardrail metrics
- Calculate required sample size or estimate how long a test must run
- Decide the unit of randomization (user, session, cluster, geo)
- Pressure-test or review an experiment plan before launch
- Build a reusable experiment design template for a team
Inputs Required
Before starting, collect:
- Product change: What is being changed and why — the intervention being tested
- Expected mechanism: How the change is supposed to move user behavior
- Baseline metrics: Current value and daily/weekly volume of the metric expected to move (and traffic available)
- Business goal: The outcome the team ultimately cares about (revenue, retention, activation, etc.)
- Minimum detectable effect: The smallest effect worth shipping for — relative or absolute
- Constraints: Max acceptable runtime, risk tolerance, regulatory or UX limits
- Population: Where the change applies — all users, a segment, a surface, a geo
If MDE is missing, default to a relative 2% lift on the primary metric and flag it. If baseline volume is missing, request it before estimating runtime — runtime cannot be honest without it.
Instructions
Step 1: Frame the Hypothesis
Write a single falsifiable hypothesis in the form: "Because [mechanism], [change] will cause [primary metric] to [direction] by at least [MDE] for [population]."
- The hypothesis must name the causal mechanism — not just "we think this is better"
- It must be falsifiable: state what result would make you reject the change
- Separate the business hypothesis (what we believe about users) from the measurement hypothesis (what the experiment can actually detect)
- Call out the leap of faith — the single assumption most likely to be wrong
Step 2: Choose the Unit of Randomization
Select and justify the randomization unit based on interference and metric alignment:
- User-level: Default. Use when treatment effects are independent across users and metrics are per-user
- Session-level: Use only when there is no cross-session learning and you accept contaminated within-user comparisons
- Cluster/geo-level: Use when network effects, marketplace spillover, or shared resources cause SUTVA violations — note the variance cost (fewer units = less power)
- Switchback/time-based: Use for systems with strong temporal interference (pricing, dispatch, ranking)
State explicitly: does the chosen unit risk SUTVA violation (one unit's treatment affecting another)? If yes, escalate to cluster or switchback design.
Step 3: Build the Metric Tree
Define four tiers, no overlap:
- Primary metric (exactly 1): The decision metric. Must be sensitive, attributable to the change, and move within the runtime window. Avoid lagging metrics (e.g., 90-day retention) as primary if runtime is short — use a validated proxy and label it as such
- Secondary metrics (2–4): Explain why the primary moved or didn't — mechanism evidence
- Guardrail metrics (2–4): Metrics that must NOT degrade (latency, error rate, revenue, complaints, churn). Define the degradation threshold that would stop the test
- Counter-metrics: Metrics that catch gaming or harmful trade-offs (e.g., engagement up but unsubscribes up)
For each metric: name it, give its current baseline, its expected direction, and whether it's a ratio metric requiring delta-method or clustered variance correction.
Step 4: Estimate Sample Size and Runtime
Compute power, then translate to calendar time:
- Use α = 0.05, power = 0.80 as defaults unless the user specifies otherwise
- Required sample per arm scales with baseline variance and inversely with MDE² — state the formula assumption (proportion vs. continuous metric)
- For ratio metrics, note that naïve variance underestimates and inflates false positives — recommend delta method or bootstrap
- Translate required sample into runtime in days using daily eligible traffic, and add buffer for weekly seasonality: round up to whole weeks
- Flag the peeking problem: if the team wants to monitor early, recommend sequential testing (e.g., always-valid p-values or group-sequential bounds) rather than fixed-horizon p-values checked daily
Output: required N per arm, estimated runtime in days/weeks, and the MDE actually detectable at that runtime if traffic is constrained.
Step 5: Define Rollout and Allocation
- Allocation: Recommend split (e.g., 50/50) and justify — uneven splits cost power
- Ramp plan: Start at a small % to validate instrumentation and guardrails before full allocation
- Pre-launch checks: A/A test or SRM (sample ratio mismatch) check to confirm randomization is unbiased before trusting results
- Stopping rules: Define exactly when to stop early (guardrail breach) vs. when to wait for full runtime (efficacy)
Step 6: Build the Risk Register (Ranked)
List 5–8 risks ranked by likelihood × impact. For each: the risk, why it threatens validity, and a specific mitigation. Cover at minimum:
- Novelty/primacy effects distorting early results
- Interference / SUTVA violation
- Underpowered design (MDE too small for available traffic)
- Multiple-comparison inflation across secondary metrics
- Instrumentation/logging gaps
- Heterogeneous effects masked by an average treatment effect
Step 7: Decision Rule
State the explicit ship/no-ship/iterate logic before launch: what primary result ships, what guardrail breach blocks, and what ambiguous result triggers an iteration rather than a relaunch.
Output Format
- Structure: Structured design document with labeled sections; metric tree and risk register as tables
- Length: 900–1,300 words
- Format: Sectioned document — prose for hypothesis and rationale, tables for metrics and risks, explicit numbers for sample size and runtime
- Sections: Hypothesis → Unit of Randomization → Metric Tree → Sample Size & Runtime → Rollout & Allocation → Risk Register → Decision Rule
- Tone: Senior experimentation specialist — precise, statistically literate, opinionated about validity threats, no hedging on the design call
Quality Checklist
Before delivering output, verify:
- [ ] The hypothesis names a causal mechanism and states what result would reject it
- [ ] Exactly one primary metric is designated; guardrails have explicit degradation thresholds
- [ ] The unit of randomization is justified and SUTVA risk is explicitly assessed
- [ ] Sample size produces a concrete N per arm AND a runtime in days/weeks using the provided traffic
- [ ] The risk register has 5–8 risks, each ranked and each with a specific mitigation
- [ ] A decision rule is stated before launch — ship/no-ship/iterate is unambiguous
Notes
- If runtime exceeds the user's max constraint, do not silently shrink it — report the detectable MDE at the allowed runtime and let the user decide whether the test is worth running
- The most common failure is choosing a primary metric that can't move within the runtime window; always check metric latency against runtime
- Recommend an A/A or SRM check as non-negotiable — a broken randomizer invalidates every downstream number
- For network-effect products (marketplaces, social, messaging), default skepticism toward user-level randomization and lean toward cluster or switchback designs
How to use this skill
This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.
Installing in Claude Code:
~/.claude/skills/<folder-name>/SKILL.md— auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)<your-project>/.claude/skills/<folder-name>/SKILL.md— auto-loads only when working in that project (recommended for project-specific skills)
Restart Claude Code or open a new session to register.
Using in Claude.ai, ChatGPT, Gemini, or other AI tools:
- Drag this file into a new chat, or copy and paste its contents
- Send your task or request as your next message — the AI will apply the skill where it fits
For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills