Where your LLMs should run, from hosted API to on-premises

The three real options

"Cloud or on-prem" is a false binary. In practice there are three deployment models, and the middle one is where most serious enterprise AI now lives.

Hosted API. You call a model provider's public endpoint. You get the strongest models, zero infrastructure and pay-per-use. Your prompts and data leave your network under the provider's terms.

Private cloud deployment. The model runs inside your own cloud tenancy or a dedicated instance the provider operates for you, often reachable only from your private network. Both frontier models and open-weight models are available this way through the major cloud platforms. Data stays in a region and account you control.

On-premises. You run open-weight models on hardware you own or lease, in your own data centre or a colocation facility. Nothing leaves the building. You own every part of the stack, including the parts that break at 3 a.m.

The decision turns on five criteria, summarised below and then discussed in turn.

The decision table

Data sensitivity and residency

This is the criterion people reach for first, and it is usually less decisive than they expect. The useful question is "what are we obliged to guarantee, and can the deployment guarantee it?" rather than "does the data leave our network?"

A hosted API with a signed data-processing agreement, no training on your data, a stated retention period and a region you can name satisfies many obligations. A private cloud deployment adds network isolation, your own encryption keys and audit logs inside your tenancy, which satisfies most of the rest. On-premises is for the residual cases: classified data, contractual bans on third-party processing, air-gapped environments, or a board that has decided.

Whatever you choose, the controls in our LLM security checklist still apply. Where the model runs says nothing about whether your retrieval layer leaks data between users. Map your obligations with the AI data privacy lens before you let residency decide the architecture, and keep the EU AI Act in view if your use is high-risk.

Model choice

Hosted endpoints give you the newest frontier models first. Private cloud deployments usually follow within weeks and also offer open-weight models. On-premises limits you to open-weight models such as the Llama and Mistral families, which are excellent for focused tasks and still behind on the hardest reasoning work.

The right way to settle this is empirical. Build an evaluation set from your real tasks, run the candidate models on it, and compare. Our guide to evaluating LLM outputs explains how. In our experience, extraction, classification, summarisation and grounded question answering are well served by smaller open models, as discussed in small language models. Open-ended agentic work tends to want a frontier model.

Cost profile

The three options have different shapes of cost, which matters more than any single number. Hosted APIs are pure variable cost: cheap at low volume, linear as you grow, zero when idle. Private cloud is typically a blend of reserved capacity and usage. On-premises is capital expenditure plus steady running costs for power, cooling, maintenance and people, whether or not anything is using the GPUs.

The crossover point depends on how busy the hardware is. A model that is busy around the clock with stable traffic can be cheaper to own; a model that is busy for two hours a day cannot. Before you believe a spreadsheet that says on-prem is cheaper, check the assumption about how busy the GPUs will be and add the people cost. Our cost of LLM applications guide breaks down the buckets, and the LLM cost calculator lets you test your own traffic shape.

Operations and the team you have

On-premises inference is a platform. Someone must choose hardware, install and tune an inference server, manage model versions and quantisation, handle scaling, patch the stack, monitor latency and errors, and be on call. If that team does not exist, the project will build it, slowly, while the business waits.

Private cloud deployments need a competent cloud platform team, which many organisations already have. Hosted APIs need almost nothing beyond good LLMOps and monitoring practice, which you need anyway.

The hybrid pattern that usually wins

Most organisations do not have one data sensitivity level; they have several. The pattern that resolves the decision for them is routing by sensitivity and task.

Put a thin gateway in front of every model call. The gateway classifies each request by data category and task type, then routes it: public or low-sensitivity requests to a hosted frontier model, internal-but-ordinary requests to a private cloud deployment, and the small set of truly restricted requests to an on-premises open-weight model. The same gateway handles logging, cost attribution, retries and fallback, and it is where model routing by cost lives too.

This keeps the expensive, hard-to-staff on-premises estate as small as possible while still honouring the strictest obligations. It also makes the inevitable future migration a configuration change. Our AI integration work usually starts by putting this gateway in place.

An illustrative example

Illustrative scenario. A mid-sized insurer wants three things: a claims-document summariser, an internal policy assistant and a customer-facing quote helper. The claims documents contain health data under strict contractual terms; the policy documents are internal but not sensitive; the quote helper handles public product information.

Rather than put everything on-premises to satisfy the claims use case, they route. Claims summarisation runs on a mid-sized open-weight model in a private cloud tenancy with customer-managed keys, after the privacy team confirms this meets their contracts. The policy assistant uses the same tenancy. The quote helper calls a hosted frontier model through the gateway. Nothing is on-premises, the platform team runs one environment, and the strictest data never touches a public endpoint. If the contract terms later tighten, only the claims route moves.

Pitfalls

What matters most

• There are three options rather than two; private cloud deployment covers most enterprise needs.
• Decide on obligations and controls rather than on where the server physically sits.
• Frontier models are hosted; on-premises means open-weight, which is excellent for focused tasks.
• Cost shape matters more than headline cost; check how busy the hardware would be before buying it.
• On-premises inference is a platform that needs a team you may not have.
• Route by sensitivity through a gateway, and keep the on-premises estate as small as obligations allow.

Related services