Field note: Small programs can still produce useful evidence, but the design should not pretend that a few observations support precise population estimates. Mixed methods can show patterns, implementation differences, mechanisms, and credible cases while making uncertainty visible.
Operational question
What combination of numeric and qualitative evidence will answer the decision question without overstating what a small sample can establish?
A workable method
- Prioritize cases and measures. Choose a small number of outcomes closely tied to the program theory. Avoid a large dashboard that turns random variation into stories.
- Use transparent descriptive analysis. Show counts, denominators, distributions, trajectories, and missingness. Individual-level plots may be more informative than an unstable average.
- Sample qualitative evidence purposefully. Seek variation in role, experience, outcome, location, and implementation exposure. Interview both expected and surprising cases.
- Integrate during interpretation. Use a joint display or case matrix to compare what changed, for whom, under which conditions, and how participants explain the pattern.
What this looks like in practice
A twelve-person maintenance academy can report each participant’s diagnostic performance over time, compare task types, and interview selected high-gain, low-gain, and non-completing participants to understand practice quality and workplace opportunity.
Evidence to collect
Choose a small set of measures before implementation. Record the baseline, the source of each measure, the review cadence, and who is authorized to act on the result.
- completeness and transparency of the case record
- convergence and divergence across evidence sources
- usefulness of findings for the next program decision
Field checklist
- Write the decision, accountable owner, and decision date.
- Describe the current workflow and the conditions that shape performance.
- Confirm the source hierarchy, permissions, and local requirements.
- Test the method under representative—not merely convenient—conditions.
- Review both intended outcomes and burden on the people doing the work.
- Record a change, escalation, and stop rule before results arrive.
Watch-out
Qualitative evidence is not a decorative quote layer. Document sampling, data collection, analytic process, researcher role, and contradictory cases with the same care used for numeric data.