An AI answer can be fluent, factually plausible and still fail the task. It might omit the exception that matters, point to a source the person cannot access or leave them unsure what to do next. Evaluating only the generated text misses the product around it.
Designers and developers need a shared definition of success that includes the output, the interaction and the system's boundaries. An evaluation is a repeatable way to examine those properties, not a score added to a launch presentation.
Define the job in observable terms
Imagine an illustrative assistant that answers questions about a product manual. “Helpful answers” is too broad. Better criteria include answering from the supplied manual, preserving important conditions, identifying the relevant section and acknowledging when the manual does not contain the answer.
Then add experience criteria. Can someone inspect the source? Can they distinguish an answer from a suggestion? If the service fails, can they still find the manual? A correct answer that appears in an unusable interface is not a successful feature.
Use AI to suggest missing criteria, but have the team resolve disagreements. Whether a wrong answer should be blocked, qualified or escalated depends on the product's consequences, not on the model's preference.
Build a small, meaningful evaluation set
Start with representative questions and known source material. Include direct questions, ambiguous wording, absent answers, conflicting passages, long inputs and attempts to obtain unauthorized information. Clearly label synthetic cases; do not present them as observed production behaviour.
For each case, record the expected properties rather than insisting on one exact sentence. A good answer may use different wording while preserving the required condition. Deterministic checks can test source identifiers and output structure. Human review can assess whether the answer is misleading or usable.
Keep some cases separate from prompt tuning. Repeatedly adjusting the system to pass the same visible examples can make progress look stronger than it is. Maintain a baseline and compare changes against the same held-out conditions where practical.
Evaluate variability, not one lucky run
Generative systems can return different results for similar or identical inputs. One successful demonstration does not establish reliability. Repeat important cases and inspect the nature of failures, especially when outputs could trigger actions.
The evaluation best-practices guide emphasizes task-specific tests, representative datasets and calibration of automated scoring against human judgement. Those are methodology recommendations, not a requirement to use a particular vendor's evaluation service. Tool availability and APIs can change; the evaluation plan should remain understandable without them.
If a model grades another model, check agreement with human reviewers. Graders may prefer longer answers or be influenced by presentation order. A high score is not meaningful until the rubric and grader behaviour have been examined.
Connect failures to product decisions
Suppose the assistant consistently answers unsupported questions with confident general advice. Prompt changes may help, but so might a narrower feature promise, a source-coverage indicator or a clear fallback to browsing the manual. Evaluation should inform design, not just model selection.
Record latency and failure conditions alongside quality. A response that is accurate but takes too long may require a different interaction or a smaller scope. Avoid collecting raw sensitive prompts simply because they are convenient test material. Use consent, minimization, access control and retention rules for any production-derived examples.
Evaluation readiness checklist
- Define output, interaction and boundary criteria together.
- Include supported, ambiguous, absent-answer and adversarial cases.
- Distinguish synthetic examples from observed behaviour.
- Use repeatable comparisons and inspect repeated-run variation.
- Calibrate automated graders against human review.
- Turn failures into design and engineering changes.
- Re-run relevant evaluations after prompt, model or retrieval changes.
Evaluation does not prove that an AI feature will never fail. It creates a disciplined way to discover where it fails and decide whether those failures are acceptable. The useful outcome is a better product decision, not a single impressive percentage.
Sources
Key takeaways
- Define success as an observable taskEvaluate whether people can complete the intended job, not just whether a model response looks plausible.
- Use representative evaluation casesInclude normal, ambiguous, unsupported and risky inputs, and repeat runs to expose variability.
- Turn failures into product decisionsTrace evaluation results to interaction changes, source coverage, safety boundaries or explicit limitations.