Evals (LLM Evaluations)
Repeatable tests that measure how well an LLM application performs on a defined set of inputs, so changes to prompts, models or data can be judged objectively.
Definition
An eval is a dataset of inputs paired with a way of scoring the output. The inputs are real or realistic cases for the task; the scoring may be exact match (for classification or extraction), a rubric applied by a human, a stronger model acting as judge, or a programmatic check (valid JSON, cites a source, under 200 words). Running the application over the set gives a number you can compare across versions.
Evals exist because LLMs are probabilistic and because small changes have wide effects. A new model, a rewritten system prompt or a different chunking strategy can improve one case and break three others. Without a fixed test set, teams argue from anecdotes rather than evidence. Retrieval has its own metrics (recall, precision, ranking) separate from answer quality.
Example: a support assistant has an eval set of 120 past tickets with approved answers. A prompt change raises the judged accuracy from 81% to 88% but drops "correctly refuses out-of-scope questions" from 97% to 90%. The team sees both numbers and decides with the trade-off in view.
What it means in practice
- They turn "it seems better" into a measured decision, and let you upgrade models with confidence.
- They catch regressions before customers do, including hallucination and safety failures.
- They are evidence for governance and audits that the system was tested against defined criteria.
- They take real effort to build, so their budget belongs in the project plan from the start.
How Teksmobile uses it
Every build in our fine-tuning and evals service starts with an eval set before any tuning, and our AI QA and testing service turns that set into a repeatable pipeline. The same datasets feed production monitoring through our LLMOps and monitoring work.
Related terms
Part of the Teksmobile AI Glossary. For longer reads, see our guides.
Frequently asked questions
How big does an eval set need to be?
Start with 50 to 100 real, representative cases per task and grow it from production failures. A small honest set beats a large synthetic one.
What is LLM-as-judge?
Using a strong model to grade another model's output against a rubric. It scales well but must itself be checked against human judgements on a sample, or it drifts.
When should evals run?
Before launch, on every prompt or model change, and on a schedule in production against sampled live traffic. Treat them like a test suite in software engineering. See evaluating LLM outputs.