Model evaluation
Compare models and prompts with real cases and measurable criteria.
Updated on Aug 09, 2026
The best choice depends on your application. An evaluation brings together representative inputs, an objective definition of success, and operational measurements to compare models and prompt versions.
- Common cases that account for most of the volume.
- Hard cases where an error has the greatest impact.
- Incomplete, ambiguous, or out-of-scope inputs.
- Safety, permission, and required-format cases.
- Recent examples, without unnecessary personal data.
| Dimension | Examples |
|---|---|
| Quality | Correctness, coverage, adherence to sources, and usefulness |
| Format | Valid JSON, required fields, limits, and language |
| Operations | Latency, failures, retries, and tokens |
| Business | Resolution, human review, and cost per completed task |
- 1
Create a baseline
Record the result of the model and the prompt you use today.
- 2
Change one variable at a time
Compare models with the same prompt, or prompts with the same model.
- 3
Review samples
Combine automated checks with human review.
- 4
Monitor in production
Track quality, cost, and failures; update the set with new cases.
General benchmarks do not replace your data
Use external results to form hypotheses. The production decision must reflect the inputs, criteria, and risks of your business.