AI experiment design

Purpose

Produce a complete, statistically defensible experiment design for an AI change — model swap, prompt iteration, pipeline modification, or feature rollout — so teams know exactly what they're testing, how they'll measure it, what sample size they need, and what decision they'll make at the end. Eliminates underpowered tests, vibes-based shipping, and ambiguous "it seems better" outcomes.

When to Use

Invoke this skill when the user asks to:

• Design an A/B test for a prompt change or model swap
• Validate a prompt iteration or pipeline modification before shipping
• Build an evaluation plan for a new AI feature or agent
• Compare two models, providers, or retrieval strategies rigorously
• Create a structured rollout plan with guardrails for an AI change
• Determine sample size or statistical thresholds for an AI experiment

Inputs Required

Before starting, collect:

• Experiment intent: What change is being tested and why (e.g., "swap GPT-4o for Claude Sonnet on support triage")
• System context: Where this AI sits — task type, user-facing or internal, current architecture
• Current baseline performance: Existing metrics if known (accuracy, CSAT, cost, latency, etc.)
• Traffic/volume: Daily or weekly request volume available for the test
• Success definition: What "better" means to the business
• Constraints: Cost ceiling, latency budget, safety requirements, regulatory considerations
• Risk tolerance: Reversibility of the change and acceptable downside

If baseline metrics are unknown, design the experiment to include a baseline measurement phase before the comparison phase.

Instructions

Step 1: Hypothesis Statement

Write a falsifiable hypothesis with three components:

• Change: The specific intervention (not "improve the prompt" but "add chain-of-thought reasoning block before final answer")
• Expected effect: Direction and magnitude (e.g., "increase resolution accuracy by 5+ percentage points")
• Mechanism: Why this change should produce this effect (so failure is interpretable)

Format: "We believe [change] will [effect with magnitude] because [mechanism]. We will know this is true if [observable signal] and false if [observable signal]."

Reject vague hypotheses ("make it better," "see if Claude is good"). Push back and reformulate.

Step 2: Baseline Definition

Specify exactly what "control" means:

• Current system snapshot: Model name + version, prompt version hash, retrieval config, temperature, all relevant params frozen for the test duration
• Baseline metrics: Primary metric value, guardrail metric values, with confidence intervals if available
• Baseline data freshness: Is the baseline from last quarter or last week? Stale baselines invalidate comparisons
• Traffic composition: What user segments, query types, or time windows the baseline reflects

If no clean baseline exists, define a baseline measurement phase (typically 1–2 weeks) before treatment launch.

Step 3: Treatment Specification

Specify exactly what's changing, holding everything else constant:

• Single variable principle: One change per experiment. Model + prompt + retrieval changed simultaneously = uninterpretable result
• Treatment definition: Exact model version, prompt text (or diff vs. control), parameter values, any new tools or context
• Implementation notes: Feature flags, routing logic, fallback behavior on treatment failure
• If multi-arm test is needed, list each arm explicitly (Control, Treatment A, Treatment B) and justify the comparison structure

Step 4: Evaluation Methodology

Choose the right evaluation approach for the task type:

Offline evaluation (use when ground truth exists or can be generated):

• Golden dataset size, source, and labeling methodology
• Eval rubric: exact-match, semantic similarity, LLM-as-judge with rubric, human review
• If LLM-as-judge: specify judge model, rubric, calibration against human labels (target >80% agreement)

Online evaluation (use when real user behavior is the signal):

• Randomization unit (user, session, request) and why
• Assignment mechanism: hash-based, feature flag, traffic split percentage
• Telemetry: what events are logged, where, and how outcomes are attributed to arms

Hybrid (most production AI changes): Offline gate first, then online rollout with traffic ramp.

Step 5: Metrics Hierarchy

Define metrics in three tiers:

Primary metric (1): The single number that determines ship/no-ship. Must be:

• Causally connected to the hypothesis
• Measurable within the experiment window
• Not gameable by the treatment in trivial ways

Secondary metrics (2–4): Mechanism checks — do they move in ways consistent with the hypothesis?

Guardrail metrics (3–6): Cannot get worse beyond threshold, regardless of primary metric:

• Quality guardrails: Hallucination rate, refusal rate, format compliance, citation accuracy
• Safety guardrails: Toxic/unsafe output rate, PII leakage, jailbreak susceptibility
• Cost guardrails: Cost per request, total spend pace
• Latency guardrails: p50, p95, p99 — specify which matters for this surface
• Business guardrails: Conversion rate, retention proxy, escalation rate

For each guardrail, set an explicit threshold: "p95 latency must not exceed 2.4s (currently 2.0s, +20% ceiling)."

Step 6: Sample Size and Power Calculation

Compute required sample size before launch:

• Minimum detectable effect (MDE): Smallest effect worth shipping (e.g., +2pp accuracy, -5% cost)
• Baseline rate: Current value of primary metric
• Statistical power: Default 80%
• Significance level: Default α = 0.05 (two-tailed for primary metric)

For proportion metrics (accuracy, resolution rate): use standard two-proportion z-test sample size formula. As a rough guide: detecting +2pp on a 70% baseline at 80% power needs ~4,300 per arm; +5pp needs ~750 per arm.

For continuous metrics (latency, cost, score): use t-test sample size with estimated standard deviation.

For LLM-as-judge offline evals: typical golden sets are 200–1,000 examples depending on MDE and task difficulty. Smaller sets are acceptable only when effect sizes are large.

Output: required N per arm, expected experiment duration given traffic, and the MDE that's actually detectable if traffic is constrained.

Step 7: Statistical Thresholds and Decision Rules

Specify exactly how the result will be interpreted:

• Primary metric decision: Treatment ships if primary metric improves by ≥ MDE with p < 0.05 (or equivalent Bayesian threshold, e.g., P(treatment > control) > 0.95)
• Guardrail decision: Any guardrail breach beyond threshold = no ship, regardless of primary
• Multiple comparison correction: If testing multiple arms or multiple primaries, apply Bonferroni or Benjamini-Hochberg
• Early stopping rules: Only with proper sequential testing (e.g., always-valid p-values, group sequential designs) — naive peeking inflates false positive rate
• Novelty effects: For user-facing changes, plan for novelty/primacy washout (first 3–7 days excluded or analyzed separately)

Step 8: Safety, Cost, and Latency Guardrails (Operational)

Beyond metric thresholds, specify operational kill switches:

• Automatic rollback triggers: Conditions under which the experiment auto-disables (e.g., error rate >2x baseline for 15 min, safety violation detected, cost pace >150% projected)
• Manual review checkpoints: When a human must review before continuing (e.g., after first 500 requests, after day 1, before ramp to next traffic tier)
• Traffic ramp schedule: 1% → 5% → 25% → 50% → 100% with hold periods at each stage
• Monitoring dashboard requirements: What must be visible in real time

Step 9: Decision Criteria — Ship / Iterate / Kill

State the explicit decision matrix before the experiment runs:

Pre-commit to this matrix in writing. Post-hoc rationalization is the most common failure mode in AI experimentation.

Step 10: Risks, Confounders, and Documentation Plan

Identify what could invalidate the result:

• Confounders to control: Time-of-day effects, day-of-week, traffic source mix, concurrent experiments, model provider outages
• Interaction effects: Other live experiments that could interact with this one
• External validity: Will the result generalize beyond the test population/time window?
• Documentation deliverables: Experiment doc (this design), monitoring plan, results writeup template, decision log

Output Format

• Structure: 10 numbered sections matching the instruction steps, plus a one-paragraph executive summary at the top
• Length: 1,400–1,900 words
• Format: Structured document with headers, tables for metrics hierarchy and decision matrix, code-style blocks for hypothesis statement and exact parameter specs
• Sections: Executive summary → Hypothesis → Baseline → Treatment → Evaluation methodology → Metrics hierarchy → Sample size & power → Statistical thresholds → Operational guardrails → Decision criteria → Risks & documentation
• Tone: Senior staff / director — precise, statistical literacy assumed, no hedging on decision rules

Quality Checklist

Before delivering output, verify:

• [ ] Hypothesis is falsifiable and includes change, expected effect with magnitude, and mechanism
• [ ] Exactly one primary metric is named (not two co-primaries without correction)
• [ ] Every guardrail metric has a numeric threshold, not a vague "shouldn't regress"
• [ ] Sample size calculation shows N per arm, expected duration, and MDE — not just "run for 2 weeks"
• [ ] Decision matrix covers ship, iterate, kill, and inconclusive cases explicitly
• [ ] Single-variable principle is enforced (or multi-arm structure is explicitly justified)
• [ ] Operational rollback triggers are specified with numeric thresholds and time windows

Notes

• The most common failure mode in AI experimentation is shipping on a primary metric improvement while ignoring a guardrail regression — enforce the guardrail veto strictly
• L

How to use this skill

This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.

Installing in Claude Code:

• ~/.claude/skills/<folder-name>/SKILL.md — auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)
• <your-project>/.claude/skills/<folder-name>/SKILL.md — auto-loads only when working in that project (recommended for project-specific skills)

Restart Claude Code or open a new session to register.

Using in Claude.ai, ChatGPT, Gemini, or other AI tools:

1. Drag this file into a new chat, or copy and paste its contents
2. Send your task or request as your next message — the AI will apply the skill where it fits

For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills