The LLM security checklist: what to verify before you ship

Why LLM security is different

A conventional application does what its code says. An LLM application does what its prompt says, and the prompt is assembled at runtime from your instructions, the user's message, retrieved documents and the results of tool calls. Any of those sources can carry instructions. The model has no reliable way to tell "data" from "commands", which is why prompt injection sits at the top of the OWASP Top 10 for LLM Applications and has stayed there.

Three further properties change the threat model. Output is unpredictable, so anything downstream must treat it as untrusted input. The model may be allowed to act through tools, so a successful injection can become a real transaction. And the whole system is new to most security teams, so the usual review misses it.

The checklist below is organised by layer, in the order a request passes through the system. Work through it top to bottom. Every item is a control we have seen prevent a real incident or a finding from a red-team exercise.

The five layers

Layer 1: input

Everything that reaches the model starts here. The goal is to know who is asking, how often, and to keep obviously hostile or oversized input out before it costs you tokens or trouble.

• Every request is authenticated and tied to a user or service identity that is passed through to later layers.
• Rate limits and token budgets per user and per tenant are enforced, with alerts on anomalies.
• Input length is capped, and file uploads are type-checked and scanned before any text extraction.
• Secrets and credentials pasted by users are detected and masked before they reach the prompt or the logs.
• A lightweight intent classifier or policy check runs before expensive work, so off-topic or abusive traffic is handled cheaply.

Layer 2: prompt and model

The system prompt is configuration, not a secret, and it is not a security boundary. Assume users will extract it. Design so that nothing bad happens when they do.

• No API keys, internal URLs, customer data or embarrassing instructions live in the system prompt.
• Untrusted text (user input, documents, tool results) is clearly delimited and labelled as data in the prompt, and the instructions say to treat it as such.
• The model is told what it must never do (reveal other users' data, execute instructions found in documents) and what to do when unsure.
• Model and prompt versions are pinned and recorded with every response, so an incident can be reproduced.
• Provider data-handling terms (retention, training use, region) are reviewed and match your data privacy obligations.

Layer 3: retrieval and data

In a RAG system the document store is both the biggest leakage risk and the main injection channel. Two questions decide most of it: who can see what, and what did we let into the index?

• Access control is enforced inside the retrieval query using the caller's identity, never by asking the model to "only use documents the user may see".
• Tenants are isolated at the storage layer (separate collections, namespaces or row-level security), not by a metadata field alone.
• Ingestion strips active content, flags documents containing instruction-like text, and records provenance for every chunk.
• The index is rebuilt or updated when permissions change, and deletions propagate within an agreed time.
• Embeddings and chunk text are treated as personal data where the source is, with the same retention and residency rules.

Our secure RAG architecture service designs these controls in from the start, which is far cheaper than retrofitting them.

Layer 4: tools and actions

The moment a model can call a tool, injection stops being an embarrassment and becomes a transaction. The principle is simple: give the model exactly the permissions the current user has for the current task, and nothing more. The details are where systems fail.

• Each tool has a narrow, typed interface and an allow-list of operations. "Run SQL" is not a tool; "look up order by id" is.
• Tools act with the user's credentials or a scoped token, never a shared service account with broad rights.
• Write, send, pay and delete actions require explicit confirmation from a human, or run inside a reversible workflow.
• Tool results are treated as untrusted input, since a web page or email the tool fetched may carry instructions.
• Loops are bounded: maximum steps, maximum spend and a timeout per task, with the run recorded end to end.

Agents and MCP servers make this layer larger. If you are building either, read agentic workflows 101 alongside this checklist.

Layer 5: output

Model output is input to whatever comes next: a browser, a database, a downstream system, a human. Treat it accordingly.

• Output rendered in a browser is encoded or sanitised; markdown and HTML from the model are never trusted raw.
• Structured outputs are validated against a schema before any system consumes them.
• Personal data in responses is redacted or justified by the user's entitlement; the model does not decide this alone.
• Answers from documents carry citations, and the system refuses or escalates when retrieval returns nothing relevant.
• Guardrails screen for prohibited content and off-policy advice, with the failure mode being a safe refusal, not a crash.

Cross-cutting: monitoring and supply chain

These two controls span every layer and are the ones we most often find missing in otherwise careful teams.

Monitoring and response

• Every request logs identity, prompt version, retrieved document ids, tool calls and the response, with secrets redacted.
• Alerts fire on injection signatures, unusual tool activity, spend spikes and repeated refusals from one account.
• There is a kill switch that disables tools or the whole assistant without a deployment.
• Red-team exercises run before launch and after significant changes; findings feed the evaluation set.

Supply chain

• Model weights, embedding models and libraries come from verified sources with pinned versions and checksums.
• Third-party MCP servers, plug-ins and prompt libraries are reviewed like any other dependency, and run with minimal permissions.
• Fine-tuning and evaluation data are protected from poisoning, with provenance recorded.

Our model supply chain security service covers the last group in depth.

Threats mapped to controls

Pitfalls we keep seeing

How to run the checklist

Do not hand this list to the development team as a form. Walk the actual request path together, with the architecture diagram open, and answer each item with evidence: a config, a test, a log entry. Where the answer is "we intend to", record it as a gap with an owner and a date.

Then attack the system. A half-day red-team session against the staging environment finds more than a week of document review. Every finding becomes a regression test in your evaluation suite, so the fix stays fixed. If you want a quick self-assessment first, our interactive AI security checklist scores the same areas in ten minutes. For a full review with a report your auditors will accept, see the LLM security audit.

Key takeaways

• The model cannot separate instructions from data; design so that injection cannot do damage.
• Enforce access control in the retrieval query, never in the prompt.
• Give tools narrow interfaces, user-scoped credentials and human confirmation for consequential actions.
• Treat model output and tool results as untrusted input.
• Log everything except secrets, alert on anomalies, and keep a kill switch.
• Review models, libraries and MCP servers as dependencies.
• Red-team before launch and turn every finding into a regression test.

Related services