Skip to content This library is free, offered as dana. If it helps your work, donate here →
penguin tree ai
← back to libraryDonate here

This is one of 3,622 free prompts in the penguin tree ai library. Sign up once and the whole library opens, with new prompts by email.

open the library →

Program impact evidence evaluator

Evaluates the strength of evidence behind a social program or intervention using a tiered evidence framework, classifying each supporting study by design (RCT, quasi-experimental, pre-post, observational, anecdotal)…

download .md

Purpose

Assess how strong the evidence actually is behind a social program or intervention — classifying each study by research design, scoring validity, and assigning an overall evidence tier — so program leaders can defend impact claims, prioritize interventions, and know exactly what evidence is still missing.

When to Use

Invoke this skill when the user asks to:

  • Evaluate whether a program's impact claims are backed by credible evidence
  • Rate an intervention against an evidence-tier framework before adopting or funding it
  • Distinguish causal evidence (RCTs, quasi-experimental) from weaker correlational or anecdotal claims
  • Prepare an evidence summary for a funder, board, or grant application
  • Identify gaps in the evidence base and plan studies to close them
  • Compare several programs on evidence strength to prioritize investment

Inputs Required

Before starting, collect:

  • Program/intervention name: What is being evaluated and what it aims to change
  • Theory of change or intended outcome: The primary outcome(s) the program claims to affect
  • Evidence sources: The studies, evaluations, reports, or data available — titles, designs if known, sample sizes, findings
  • Population and context: Who the program serves and where (setting, geography, scale)
  • Decision at stake: What this evaluation will inform (funding, adoption, scaling, reporting)
  • Standard to apply: Preferred evidence framework if any (e.g., ESSA tiers, HomVEE, What Works Clearinghouse) — otherwise the default tiered framework is used

If study designs or sample sizes are not provided, infer them from the source descriptions and flag every inference as an assumption to verify.

Instructions

Step 1: Inventory and Classify Each Evidence Source

For every study or source provided, capture and classify:

  • Design type, using this hierarchy from strongest to weakest:
  • RCT (Randomized Controlled Trial): Random assignment to treatment/control — strongest for causal claims
  • Quasi-Experimental (QED): Comparison group without randomization (matched groups, regression discontinuity, difference-in-differences, interrupted time series)
  • Pre-Post / Single-Group: Same group measured before and after, no comparison group — cannot isolate cause
  • Observational / Correlational: Cross-sectional or cohort analysis showing association, not causation
  • Descriptive / Monitoring Data: Outputs, participation counts, satisfaction — describes activity, not impact
  • Anecdotal / Testimonial: Stories, case examples, single-participant reports
  • Outcome measured: Does it match the program's primary intended outcome, or a proxy?
  • Sample size and attrition: Note underpowered samples (small N) and high dropout
  • Independence: Was the study independent or self-conducted/funder-commissioned?
  • Replication: Has the finding been reproduced in another sample or setting?

Step 2: Score Internal Validity (Can We Trust the Causal Claim?)

For each study, flag threats to internal validity:

  • No comparison/control group (cannot rule out that change would have happened anyway)
  • Selection bias (participants differ systematically from non-participants)
  • High or differential attrition (over ~20% dropout, or dropout differing between groups)
  • Confounding events during the study period
  • Outcome measured only immediately, with no durability check Rate internal validity: Strong / Moderate / Weak / None.

Step 3: Score External Validity (Does It Transfer to This Context?)

Assess how well the evidence applies to the buyer's population and setting:

  • Population match: Study sample vs. the population this program serves
  • Setting match: Delivery context, dosage, and scale
  • Implementation fidelity: Was the studied version the same as what will be delivered? Rate external validity: High / Partial / Low.

Step 4: Assign the Overall Evidence Tier

Based on the strongest credible evidence and its consistency, assign one tier:

  • Tier 1 — Well-Built (Strong) Evidence: At least one well-conducted RCT, or two or more strong QEDs, showing a favorable effect on the intended outcome, with acceptable sample size, low attrition, and ideally independent replication
  • Tier 2 — Moderate Evidence: A single well-conducted QED, or an RCT with notable limitations, showing favorable effects but not yet replicated or with external-validity concerns
  • Tier 3 — Preliminary Evidence: Pre-post or correlational studies suggesting a favorable trend, or a promising pilot, but no comparison group to establish cause
  • Tier 4 — No Reliable Evidence: Only descriptive, monitoring, or anecdotal information; or studies with fatal validity threats; or a compelling theory of change with no impact evaluation yet State the tier plainly and give a 2–3 sentence justification tied to the specific studies.

Step 5: Gap Analysis

Identify what is missing to reach the next tier:

  • The single most important design weakness across the current evidence
  • Whether the primary outcome is even being measured (vs. proxies or outputs)
  • Whether findings have been replicated or tested at scale
  • Whether the evidence applies to the actual population served

Step 6: Evidence-Strengthening Plan

Recommend concrete next steps ranked by cost and rigor gain:

  • The most feasible study design to move up one tier (e.g., add a matched comparison group)
  • What to measure and when (baseline, endline, follow-up for durability)
  • Quick wins to improve current data quality (fidelity tracking, standardized outcome measures)
  • What to say honestly to funders/boards given the current tier

Output Format

  • Structure: An evidence-classification table for all sources, followed by narrative scoring, a single overall tier verdict, and forward-looking recommendations
  • Length: 800–1,200 words
  • Format: Table for the source inventory; structured bullets for validity scoring; prose for the tier verdict and plan
  • Sections: Evidence Inventory (table) → Internal Validity → External Validity → Overall Evidence Tier (verdict) → Gap Analysis → Evidence-Strengthening Plan
  • Tone: Rigorous and candid — a MEL specialist who will not overstate weak evidence, delivering an honest tier even when it's Tier 4

Quality Checklist

Before delivering output, verify:

  • [ ] Every source provided is classified by one of the six design types
  • [ ] Each study has both an internal-validity rating and an external-validity rating
  • [ ] A single overall evidence tier (1–4) is assigned with justification tied to specific studies
  • [ ] Any inferred study designs or sample sizes are explicitly flagged as assumptions
  • [ ] The gap analysis names the specific weakness blocking the next tier — not a generic "more research needed"
  • [ ] The strengthening plan specifies at least one concrete, feasible study design to advance one tier

Notes

  • Do not inflate the tier to be encouraging — a Tier 4 verdict delivered honestly is more useful than a Tier 2 verdict that collapses under funder scrutiny
  • A strong theory of change is not evidence of impact; classify it as Tier 4 until an outcome evaluation exists
  • Watch for outcome-swapping: programs often cite strong evidence for a proxy (e.g., attendance) while claiming impact on the real goal (e.g., graduation)
  • When multiple studies conflict, weight by design strength and independence, not by recency or headline effect size
  • If the buyer names a specific framework (ESSA, WWC, HomVEE), map the overall tier to that framework's language in the verdict so it's usable in their reporting

How to use this skill

This file is a Claude Code skill in Anthropic's standard format — markdown with YAML frontmatter at the top.

Installing in Claude Code:

  • ~/.claude/skills/<folder-name>/SKILL.md — auto-loads in every session anywhere on your machine (recommended for skills you'll use across multiple projects)
  • <your-project>/.claude/skills/<folder-name>/SKILL.md — auto-loads only when working in that project (recommended for project-specific skills)

Restart Claude Code or open a new session to register.

Using in Claude.ai, ChatGPT, Gemini, or other AI tools:

  1. Drag this file into a new chat, or copy and paste its contents
  2. Send your task or request as your next message — the AI will apply the skill where it fits

For Claude Code skill details: https://docs.anthropic.com/en/docs/claude-code/skills