Field note: A demo can show that a chatbot sometimes answers well. An evaluation set asks whether it behaves well enough across the situations that matter. The set should represent real learner language, difficult edge cases, and questions the system must decline.
Operational question
What evidence would justify a limited pilot of a training-support chatbot, and what failures would make the team stop?
A workable method
- Collect authentic question types. Use de-identified help requests, instructor notes, search logs, and learner interviews. Include short questions, ambiguous wording, misconceptions, and requests that cross the system’s scope.
- Write expected behaviors. For each case, define must-have facts, acceptable variations, required citations, and forbidden claims. Some cases should require a clarifying question or a refusal.
- Score multiple dimensions. Separate factual support, completeness, instructional usefulness, tone, accessibility, and safety. A single pass/fail label hides why a response succeeds or fails.
- Version the set and results. Keep a stable core for comparison and add cases when production reveals a new failure. Record model, prompt, retrieval corpus, configuration, date, and evaluator.
What this looks like in practice
A maritime training assistant might be tested on a routine definition, a question with an outdated regulation embedded in it, a request for medical advice, an ambiguous equipment fault, and a question that can only be answered by the vessel’s approved procedure. The expected behaviors differ for each.
Evidence to collect
Choose a small set of measures before implementation. Record the baseline, the source of each measure, the review cadence, and who is authorized to act on the result.
- grounded accuracy on high-priority question types
- appropriate clarification or refusal rate
- severity-weighted failure count by evaluation version
Field checklist
- Write the decision, accountable owner, and decision date.
- Describe the current workflow and the conditions that shape performance.
- Confirm the source hierarchy, permissions, and local requirements.
- Test the method under representative—not merely convenient—conditions.
- Review both intended outcomes and burden on the people doing the work.
- Record a change, escalation, and stop rule before results arrive.
Watch-out
Do not optimize only for average score. A system with excellent routine answers and one dangerous safety failure may be unsuitable. Weight results by consequence and inspect every severe failure individually.