Context Window
The maximum number of tokens a model can take in at once, including instructions, conversation history, retrieved documents and the answer it is generating.
Definition
The context window is the working memory of an LLM, measured in tokens. Everything the model can see for one request has to fit inside it: the system prompt, any tool definitions, the conversation so far, documents or search results you add, and the response being generated. Nothing outside the window exists for the model; it has no persistent memory between requests unless the application supplies one.
Window sizes have grown from a few thousand tokens to hundreds of thousands, and more than a million on some models. A larger window is useful but not free. Cost rises with every token processed, latency grows, and models can still overlook details in the middle of very long inputs. For most business applications, retrieval and summarisation keep each request well below the limit.
Example: a 200-page contract is around 100,000 tokens. It may fit in one request, but reviewing one clause does not need all of it. RAG sends the dozen relevant passages instead, at a fraction of the cost.
What it means in practice
- It sets the ceiling on how much a model can consider per request, and therefore what the product can promise.
- Every token in the window is billed; careless prompts and long histories quietly multiply cost.
- Conversation memory, document size and tool count are all design decisions constrained by the window.
- Prompt caching on some providers makes repeated long prefixes cheaper, which changes the economics.
How Teksmobile uses it
Context budgeting, deciding what goes into each request and what is retrieved or summarised, is a standard step in our LLM application development work. Our LLM cost calculator shows how window usage drives monthly spend before you commit to a design.
Related terms
Part of the Teksmobile AI Glossary. See also the cost of LLM applications.
Frequently asked questions
If the window is large, do we still need RAG?
Usually yes. Large windows cost more per request, slow responses, and models still attend less reliably to material buried in the middle of a very long prompt. Retrieval keeps prompts focused and cheap.
What happens when the context overflows?
The request is rejected or the application must drop something: trim history, summarise earlier turns or retrieve fewer passages. Designing that policy is part of building the application.
Does the output count towards the window?
Yes. Input and output share the same limit on most models, and providers often set a separate, smaller maximum for output tokens.