LLMOps and monitoring so you know what your AI is doing before your customers do
Tracing for every model and tool call, dashboards for latency, cost and quality drift, release gates tied to your evals, and a runbook for the day something changes.
- 2 to 4 weeks to instrument
- Latency, cost, quality, safety
- Monthly operating review
Three panels every AI owner should have open
Latency tells you whether users are waiting. Cost tells you whether finance will call. Quality drift tells you whether the thing still works, which no infrastructure metric can. The third panel is the one most teams are missing.
Signals, alert conditions and who acts
| Signal | Alert condition (set per feature) | Likely cause | Owner and action |
|---|---|---|---|
| Latency p95 | Above the agreed budget for a sustained window | Context growth, provider slowdown, retrieval slow, retries | Engineering: check traces, shrink context, fail over model |
| Cost per request and per day | Above budget, or sudden change in tokens per request | Prompt change, loop in an agent, cache miss, traffic spike | Engineering and product: trace the top spenders, cap or route |
| Token usage per request | Drift upward without a release | Longer inputs, growing conversation history | Engineering: truncation and summarisation rules |
| Quality score (sampled evals) | Below release threshold across a sample window | Data or content change, provider model update, prompt regression | Product owner: review failures, fix content or prompt, re-gate |
| Refusal and safety flags | Rate changes sharply in either direction | Prompt change, abuse campaign, provider policy change | Security and product: inspect samples, tighten or loosen |
| Injection attempts | Any confirmed attempt on an agent with tools | Malicious content in email, tickets, web pages | Security: review with governance controls, add to red-team set |
| Retrieval hit rate | Share of answers with no supporting source rises | Index stale, chunking broke, new topics | Content owner: refresh index, add content |
| Tool-call error rate | Above baseline for any tool | Downstream API change, expired credential, bad arguments | Engineering: fix the integration, check model arguments |
From traces to a monthly review
Tracing layer
Every model call, tool call and retrieval recorded with inputs, outputs, tokens, latency, cost and the version of prompt and model used. Redacted where required.
Dashboards and alerts
The three panels above plus safety and retrieval signals, with alert thresholds agreed per feature and routed to the people who can act.
Release gates
Prompt, model and index changes go through the eval harness in CI. A change ships only if the score holds, and it ships with a version tag you can roll back.
Runbook and review
Who does what when an alert fires, how to fail over between models with model routing, and a monthly review of cost, quality and the next improvement.
Instrument
Wrap model, tool and retrieval calls. Ship traces to your observability stack or one we host.
Baseline
Two weeks of data. Budgets and thresholds set from measured data.
Gate
Eval harness in CI, version tags on prompts and models, rollback rehearsed.
Operate
Alerts routed, runbook in use, monthly review feeding the roadmap.
The provider updates the underlying model; your prompts behave differently; nobody is scoring outputs, so the first signal is a customer complaint three weeks later. Meanwhile an agent has been looping on a broken tool and the bill has doubled. Both are visible within a day on the dashboard above. Read our note on measuring AI ROI for why this data also justifies the next project.
Operations connects to these
Frequently asked questions
What is LLMOps?
The operational discipline for applications built on large language models: tracing, monitoring, evaluation, cost control, release management and incident response. It is MLOps adapted to systems where the model is an external API and quality is judged on text rather than a single metric.
What should we monitor on an LLM feature?
Four families: latency percentiles, cost per request and per day, quality scores from sampled evals, and safety signals such as refusal rate and injection attempts. Retrieval hit rate and tool-call errors matter for RAG and agents. The table on this page lists the alert conditions we set.
How do you detect quality drift if the output is text?
By sampling production traffic into the eval harness and scoring it with the same rubrics used at release, then tracking the score over time. A sustained drop below the threshold raises an alert before customers notice. See fine-tuning and evals.
Can you set this up on something we already built?
Yes. Instrumenting an existing application typically takes two to four weeks: a tracing layer around model and tool calls, dashboards, a first set of alerts and a sampled eval. Nothing in the application logic needs to change on day one.
How do you keep costs predictable?
Budget alerts per feature, caching of repeated prompts, routing of simple tasks to cheaper models, context-size limits and a monthly review. Use the LLM cost calculator for a first estimate and our guide to the cost of LLM applications for the levers.