Where the money goes in an LLM application

The four cost buckets

Every LLM application we have costed falls into the same four buckets. The proportions vary enormously by use case, which is why rules of thumb mislead.

1. Model tokens. What the provider charges for input and output, or what self-hosted inference costs in hardware and power.
2. Retrieval and infrastructure. Vector storage, embedding, search, the application servers, gateways and logging around the model.
3. Evaluation and monitoring. Building and maintaining test sets, running them, sampling production, dashboards, red teaming.
4. People and maintenance. Engineers, product and domain experts to build, tune, operate and change the system over its life.

What drives the token bill

Providers charge per token, with input and output priced differently and output usually costing more. Four things decide the total.

Traffic. Requests per day, multiplied by tokens per request. This is obvious, and it is the number most often underestimated, because successful features get used.

Context length. Every request carries your instructions, any retrieved documents, the conversation history and the user's message. A system prompt of two thousand tokens, sent on every one of fifty thousand daily requests, is a hundred million input tokens a day before anyone has asked anything. Treat the context window as a budget rather than a target.

Model tier. Frontier models cost multiples of smaller ones from the same provider. Many tasks in a real application, such as classification, extraction and routing, do not need the frontier model at all.

Caching. Most providers now offer prompt caching, which charges much less for the repeated prefix of a prompt. Long, stable system prompts and shared document context are exactly the shape caching rewards. Design prompts so the stable part comes first.

Retrieval and infrastructure

For a RAG system, this bucket includes embedding the corpus (a one-off cost, repeated when the embedding model changes), storing and searching vectors, re-embedding on updates, and the reranker if you use one. For most knowledge bases these costs are modest compared with tokens and people; they grow with corpus size and query volume, and they spike when you change embedding models and have to re-index everything. Our RAG cost estimator sizes this for your document counts.

Around the model sits ordinary infrastructure: application servers, a gateway for routing and logging, queues for batch work, storage for traces. None of it is exotic. All of it is forgotten in the first estimate.

Nobody budgets for evaluation and monitoring

You cannot run an LLM system responsibly without knowing whether it is doing the job, and that knowledge costs money. Building the first evaluation set takes domain experts' time. Running it on every prompt, model or retrieval change costs tokens. Sampling production outputs for review costs reviewer hours. Dashboards, alerts and drift detection cost engineering time. Red teaming before launch and after major changes costs specialist time.

Teams that skip this bucket do not save the money; they pay it later in incidents, in arguments about whether the system has got worse, and in the inability to upgrade models safely. Budget it explicitly, as described in evaluating LLM outputs and our LLMOps and monitoring service.

People and maintenance

For an internal tool with modest traffic, people are the majority of the cost over the first year: the engineers who build it, the domain experts who write evaluation cases and review outputs, the person who owns it afterwards. The model bill is a rounding error by comparison. The lesson is to choose projects where the people cost buys something worth having, and to keep scope tight, as our AI project roadmap recommends.

Maintenance is real and ongoing. Providers retire models, documents change, users find new failure modes. Plan for a fraction of the build effort every year, indefinitely.

The levers that cut cost

Routing and caching are the two that matter most. Together they often change the shape of the token bucket entirely. Where a system runs, covered in on-prem vs cloud LLMs, affects the inference cost shape too, but rarely as much as these two levers.

An illustrative example

Illustrative scenario, relative figures only. A support assistant answers customer questions from documentation. In its first version every request sends a long system prompt, ten retrieved chunks and the full conversation history to a frontier model. Call its monthly token cost 100 units.

The team reorders the prompt so the system prompt and shared context come first and switches on caching: the repeated prefix is now charged at a fraction of the rate, and the bill drops to roughly 70. They cut retrieved chunks from ten to five after the evaluation set shows no quality loss, and summarise history beyond the last few turns: roughly 50. They add a router that sends simple "where do I find" questions to a smaller model, which the evaluation set shows handles them as well: roughly 35. Three changes cut the bill to about a third with no loss of measured quality. The evaluation set that made each step safe cost less than one month's saving.

Cost-control checklist

• Estimate tokens per request and requests per day before building, with a growth range.
• Put a gateway in front of every model call to log usage and attribute cost per feature and tenant.
• Set spend alerts and per-user or per-tenant budgets from day one.
• Order prompts for caching; keep the system prompt stable.
• Build the evaluation set early so every cost lever can be pulled safely.
• Route by task; do not send everything to the frontier model.
• Budget evaluation, monitoring and maintenance explicitly for the life of the system.

Pitfalls

If the question behind this guide is "was it worth it", read measuring AI ROI next; cost is only half of that calculation.

What matters most

• There are four buckets: tokens, retrieval and infrastructure, evaluation and monitoring, and people.
• Tokens dominate only for high-traffic customer-facing systems; elsewhere people and evaluation do.
• Context length and traffic drive the token bill; the context window is a budget.
• Routing and prompt caching are the two levers that change the shape of the bill.
• Evaluation is the bucket nobody budgets and the one that makes every saving safe.
• Log and attribute cost per feature from day one.

Related services