Purpose
Turn messy pilot and experiment data into a structured, decision-ready read-out that states the hypothesis verdict, separates what worked from what didn't, delivers a clear scale/kill/iterate recommendation, and distills a repeatable playbook. Solves the common problem where pilots conclude but the results never get synthesized into a decision or preserved as institutional knowledge.
When to Use
Invoke this skill when the user asks to:
- Analyze the results of a completed pilot or growth experiment
- Decide whether to scale, kill, or iterate on a tested initiative
- Write a stakeholder read-out or post-mortem for an experiment
- Turn pilot learnings into a repeatable playbook or SOP
- Interpret whether a test result is statistically meaningful or noise
- Compare a pilot cohort against a control or baseline
Inputs Required
Before starting, collect:
- Experiment overview: What was tested and why — one paragraph
- Hypothesis: The specific prediction being tested (e.g., "reducing onboarding steps will lift 7-day activation by 10%")
- Primary metric + target: The metric the experiment was designed to move, and the success threshold
- Results data: Actual numbers — test vs. control/baseline, sample sizes, secondary metrics
- Duration and scope: How long it ran, and which segment/geography/cohort it covered
- Qualitative signals: User feedback, support tickets, operational friction, team observations
- Cost/effort to run: What it took to execute, and what it would take to scale
If sample size or control data is missing, flag the confidence limitation explicitly rather than overstating the result. If cost-to-scale is missing, note it as a required input for the final recommendation.
Instructions
Step 1: State the Hypothesis Verdict
Open with a single, unambiguous verdict before any detail:
- Confirmed: Primary metric hit or exceeded the target
- Partially confirmed: Directional lift but below target, or lift in a sub-segment only
- Not confirmed: No meaningful movement or movement in the wrong direction
- Inconclusive: Insufficient sample, contaminated control, or too short a run to trust
State the verdict in one sentence, then give the headline number (e.g., "Confirmed — 7-day activation rose from 34% to 41%, a 20.6% relative lift against a 10% target").
Step 2: Assess Confidence and Data Quality
Before interpreting results, pressure-test whether they can be trusted:
- Sample size: Is each arm large enough? Flag if primary metric has fewer than ~100 conversions per arm — results are directional at best
- Statistical significance: State whether the difference is likely real or within noise. If exact stats aren't available, estimate: small samples + small lifts = treat as directional
- Duration validity: Did it run long enough to capture a full behavior cycle (e.g., a weekly usage pattern, a full sales cycle)?
- Confounders: Seasonality, concurrent launches, novelty effect, selection bias in the pilot cohort
- Assign a confidence rating: High / Medium / Low with one line of reasoning
Step 3: What Worked
Extract the specific, causal drivers of positive results — not just "it went up":
- Identify which changes or mechanics most plausibly drove the movement
- Separate the primary metric win from secondary metric wins (e.g., activation up AND support tickets down)
- Call out any segment where results were disproportionately strong — this is often where scale value hides
- Distinguish signal from anecdote: anchor each claim to a number or a repeated qualitative pattern
Step 4: What Didn't Work
Be equally rigorous about failures and side effects:
- Metrics that stayed flat or moved the wrong way
- Negative externalities (e.g., activation up but churn up, or lift concentrated in low-value users)
- Operational friction that would worsen at scale (manual steps, support load, unit economics)
- Segments where the experiment underperformed or backfired
Step 5: Scale / Kill / Iterate Recommendation
Deliver one clear recommendation with reasoning:
- Scale: Result confirmed, confidence Medium+, and cost-to-scale is justified by projected impact. Include a projected impact estimate and the rollout guardrails
- Iterate: Directional or partial result worth another round — specify exactly what to change and what to test next
- Kill: Not confirmed, or confirmed but uneconomical to scale — state the reason plainly
- Include a projected impact if scaled: rough extrapolation of the lift applied to the full addressable population, with the assumption stated
- Include rollout guardrails: the metric threshold or leading indicator that would trigger a pause during rollout
Step 6: Playbook Distillation
Convert the learning into a reusable asset so the next operator doesn't start from zero:
- The repeatable mechanic: The core change described so it can be re-run in a new context
- Preconditions: What must be true for this to work (segment, stage, platform state)
- Setup steps: The operational steps to deploy it, in order
- Metric to watch: The single leading indicator that tells you it's working within days, not weeks
- Known failure modes: What to avoid, drawn from what didn't work in this run
Output Format
- Structure: Six labeled sections in the order of the instructions, opening with the verdict
- Length: 700–1,000 words
- Format: Structured document — verdict as a bolded lead line, confidence as a rating, what-worked/what-didn't as bulleted lists, recommendation as a callout, playbook as a numbered/labeled reusable block
- Sections: Hypothesis Verdict → Confidence & Data Quality → What Worked → What Didn't Work → Scale/Kill/Iterate Recommendation → Playbook Distillation
- Tone: Growth lead in a leadership read-out — direct, quantitative, willing to call a result inconclusive or recommend a kill without hedging
Quality Checklist
Before delivering output, verify:
- [ ] The verdict is stated in one unambiguous sentence with the headline number
- [ ] Confidence is rated High/Medium/Low with a stated reason tied to sample size or duration
- [ ] What worked and what didn't are each anchored to specific numbers, not adjectives
- [ ] Exactly one recommendation is given (Scale, Iterate, or Kill) — no fence-sitting
- [ ] The recommendation includes a projected impact estimate and a rollout guardrail
- [ ] The playbook section is reusable in a new context — includes preconditions, steps, and a metric to watch
Notes
- A confirmed result with poor unit economics is still a kill — always weigh cost-to-scale against projected impact, not just the metric lift
- Novelty effect is the most common false positive in growth pilots: a lift that fades once users stop noticing the change. If the run was short, flag this explicitly
- Small samples are the norm in early pilots — the honest move is to label results "directional" and recommend a confirmatory round rather than scaling on noise
- The playbook is often the most valuable output even when the recommendation is Kill — a documented failure mode saves the next team a wasted quarter
How to use this skill
This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.
Installing in Claude Code:
~/.claude/skills/<folder-name>/SKILL.md— auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)<your-project>/.claude/skills/<folder-name>/SKILL.md— auto-loads only when working in that project (recommended for project-specific skills)
Restart Claude Code or open a new session to register.
Using in Claude.ai, ChatGPT, Gemini, or other AI tools:
- Drag this file into a new chat, or copy and paste its contents
- Send your task or request as your next message — the AI will apply the skill where it fits
For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills