Fine-tuning and evals: measure first, then change the cheapest thing
We build the test set before touching the model, then work up the ladder from prompting to retrieval to fine-tuning, stopping at the first rung that passes.
A real input, taken from production or written by the person who handles it today. An expected output, or a rubric that says what a good answer must contain and must not contain. A note on why the case is in the set: a common request, a known failure, an edge case, an adversarial attempt. And a version, so when the case changes you know which scores are comparable. A set built this way is readable by a product owner, not just an engineer, which is what keeps it alive after we leave.
What we deliver
A measurement system you keep
Eval set and harness
Versioned cases drawn from real traffic and expert input, a runner that scores every change in CI, and a dashboard your team reads.
Tuning work, in order
Prompt and output-structure work first, retrieval improvements with our enterprise RAG practice second, fine-tuning only when the tree says so.
Fine-tuning pipeline
Data curation and de-duplication, held-out test split, training runs on open-weight or hosted models, and a comparison against the untuned baseline. See fine-tuning and RLHF in the glossary.
Build the eval set
Real cases, expected outputs, rubrics, and the awkward 20% that breaks things. Agree the pass threshold.
Baseline
Score the current system and two or three candidate models. Now there is a number to beat.
Cheapest fix first
Prompting, then retrieval, then fine-tuning, each measured against the same set.
Gate and monitor
Release only on a passing score; sample production into the set so it keeps up with reality via LLMOps monitoring.
What goes wrong without evals
Example: a team fine-tunes a model on 300 hand-written examples to fix a tone problem. Tone improves; factual accuracy quietly drops, because nobody measured it. Three weeks later the support team notices. A golden-set regression run before release would have shown the trade-off in minutes, and the prompting fix that was tried first would have been kept.
Less often than people expect. Most quality gaps are fixed by better prompting or by retrieval. Fine-tuning earns its place when the problem is style, consistency, latency or cost at high volume, and when you have hundreds to thousands of good examples. The decision tree on this page is how we decide, and our guide to prompt engineering versus fine-tuning goes deeper.
What is an eval set and how big does it need to be?
A versioned collection of real inputs with expected outputs or grading rubrics, scored automatically. Fifty well-chosen cases will catch regressions; a few hundred give confidence for a release gate. Quality and coverage of the awkward cases matter more than size. See evals.
Can an LLM grade another LLM's output?
Yes, with care. A judge model scoring against a written rubric is fast and cheap, and correlates well with human judgement on many tasks. We calibrate the judge against a human-labelled sample before trusting it, and we keep humans on the cases that matter most.
Which models can be fine-tuned?
Open-weight models such as Llama and Mistral can be fine-tuned anywhere, including on your own hardware, and several hosted providers offer fine-tuning on their models. We choose based on your data residency rules, volume and the base model that already scores best on your evals.
How long does an evals engagement take?
Building a first eval set and baseline typically takes two to three weeks. A fine-tuning project, including data curation, training runs and evaluation, usually runs six to ten weeks. The curation step is nearly always the longest.
Ready to make AI work in production?
Tell us about your systems, your data and your constraints. We reply within one business day with a straight answer and a suggested next step.