Purpose
Turn a vague product or growth idea into a decision-ready experiment: the specific assumption being tested, a falsifiable hypothesis, the smallest test that generates a valid signal, the metrics and thresholds that define a win, and what to do based on each outcome. Prevents the two most common failures — building on untested assumptions and running tests that can't actually produce a decision.
When to Use
Invoke this skill when the user asks to:
- Design an A/B test for a feature, flow, or copy change
- Validate a new feature idea before committing engineering time
- Test a growth, onboarding, activation, or conversion hypothesis
- Figure out the cheapest way to de-risk an idea before building it
- Turn a "I think users want X" statement into a testable experiment
- Set up metrics, sample size, and success criteria for a planned test
Inputs Required
Before starting, collect:
- The idea: What they want to build, change, or try — in their own words
- The belief behind it: Why they think it will work (the underlying assumption)
- Current metric baseline: The metric this is meant to move and its current value (e.g., activation rate 34%, trial-to-paid 18%)
- Traffic/user volume: How many users hit this surface per week or month
- Constraints: Engineering effort available, timeline, and any technical limits on what can be tested
- Stage of confidence: Is this a wild guess, an informed hunch, or something with supporting signal already?
If baseline metric or traffic volume is missing, estimate a realistic range from the context provided and flag the assumption — these directly affect the sample size and duration recommendation.
Instructions
Step 1: Identify the Riskiest Assumption
Extract the assumptions buried in the idea and rank them. The riskiest assumption is the one that, if false, kills the idea — and that you currently have the least evidence for.
- List 3–5 assumptions the idea depends on (demand, willingness to change behavior, technical feasibility, value perception)
- For each, mark evidence level (none / weak / moderate) and impact if wrong (low / medium / fatal)
- Select the single assumption with the highest impact and lowest evidence — this is what the experiment must test first
- State plainly: "The riskiest assumption is that [X]. If it's false, the idea doesn't work."
Step 2: Write a Falsifiable Hypothesis
Reframe the idea as a hypothesis that can be proven wrong. Use the format:
"We believe that [change] for [audience] will cause [measurable outcome], because [reasoning]. We'll know we're right if [specific metric] moves from [baseline] to [threshold] within [timeframe]." - The outcome must be a number, not a feeling - The hypothesis must be capable of failing — if any realistic result would still be interpreted as success, rewrite it
Step 3: Choose the Minimum Viable Test
Match the test type to the riskiest assumption and the effort available — always favor the cheapest test that produces a valid signal.
- Demand exists? → Fake-door / painted-door test, landing page, or feature-interest CTA. No build required.
- Behavior will change? → A/B test on the live flow, or a concierge/manual version delivered to a small cohort.
- Value is perceived? → Wizard-of-Oz test (manual backend), or a prototype usability test with intent-to-pay questions.
- Feasibility works? → Technical spike or single-cell pilot with instrumentation.
State the chosen test type, why it fits this assumption, and the rough build effort (hours/days, not a full feature).
Step 4: Define Metrics
- Primary metric: The single number the decision hangs on. One only.
- Secondary metrics: 1–2 metrics that add context or explain the "why"
- Guardrail metrics: 2–3 metrics that must NOT degrade (e.g., churn, support tickets, load time, downstream conversion). Define the tolerance for each.
- Confirm the primary metric is directly caused by the change and measurable within the test window.
Step 5: Estimate Sample Size and Duration
- State the minimum detectable effect (MDE) worth acting on (e.g., a relative lift of 10%)
- Estimate required sample size per variant using the baseline conversion rate and MDE at 95% confidence and 80% power — provide a concrete number
- Divide by weekly traffic to get estimated duration
- Enforce a minimum runtime of one full business cycle (typically 1–2 weeks) to absorb weekday/weekend variance, even if sample size is reached sooner
- If traffic can't reach significance in a reasonable window, flag it and recommend a lower-effort qualitative test instead
Step 6: Set Decision Thresholds and the Next-Steps Ladder
Define what happens at each outcome — no ambiguous results allowed:
- Win (primary metric hits threshold, guardrails hold): Ship, then specify the next experiment to compound the gain
- Flat/inconclusive (no significant movement): State whether to iterate on the variant, retest with a larger swing, or drop
- Loss (metric drops or guardrail breaks): State the learning and the pivot
- End with a one-line kill criterion: the result that means "stop pursuing this idea."
Output Format
- Structure: Structured experiment brief with labeled sections in the order of the steps
- Length: 600–900 words
- Format: Structured document — short prose per section, hypothesis in blockquote, assumptions and thresholds as bullets/table
- Sections: Riskiest Assumption → Hypothesis → Minimum Viable Test → Metrics (primary/secondary/guardrail) → Sample Size & Duration → Decision Thresholds & Next Steps
- Tone: Product lead reviewing an experiment plan — pragmatic, decisive, allergic to tests that can't produce a decision
Quality Checklist
Before delivering output, verify:
- [ ] Exactly one riskiest assumption is named with evidence level and impact rating
- [ ] The hypothesis includes a baseline number, a target threshold, and a timeframe — and can fail
- [ ] Exactly one primary metric is defined; guardrail metrics have explicit tolerances
- [ ] A concrete sample-size number and estimated duration are provided, tied to the stated baseline and MDE
- [ ] All three outcome branches (win / flat / loss) have specific actions, plus a kill criterion
- [ ] The chosen test is the cheapest option that still tests the riskiest assumption — not an over-built version
Notes
- The most common mistake is testing the idea instead of testing its riskiest assumption — if the assumption is "people want this at all," a polished A/B test is wasteful; a fake-door test answers it in days.
- If traffic is too low for statistical significance, say so directly and shift to a qualitative or concierge test rather than running an underpowered A/B test that produces a false read.
- A hypothesis that can't produce a "loss" outcome is not a hypothesis — it's a launch plan in disguise. Push back and rewrite it.
How to use this skill
This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.
Installing in Claude Code:
~/.claude/skills/<folder-name>/SKILL.md— auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)<your-project>/.claude/skills/<folder-name>/SKILL.md— auto-loads only when working in that project (recommended for project-specific skills)
Restart Claude Code or open a new session to register.
Using in Claude.ai, ChatGPT, Gemini, or other AI tools:
- Drag this file into a new chat, or copy and paste its contents
- Send your task or request as your next message — the AI will apply the skill where it fits
For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills