What Is a Context Window?

The context window is the model's working memory: everything it can see while it answers. Whatever is not in it does not exist, and everything in it costs money and attention.

What Is a Context Window? — Troiana insight cover

In short

A context window is the maximum number of tokens a language model can process in a single request, covering the system prompt, the conversation so far, any documents or retrieved passages supplied, and the answer it generates. Current models offer windows from around 128,000 tokens to over a million, but the window is shared, costs scale with what fills it, and models attend less reliably to material buried in the middle of a very long context. Products manage it by trimming and summarising history, retrieving only relevant document chunks, and keeping system prompts lean rather than by relying on the maximum.

What it is

A language model has no memory between requests. Each time it is called, it receives a block of text and produces a continuation. The context window is the maximum size of that block, measured in tokens, and it holds everything the model can take into account: the instructions in the system prompt, the messages exchanged so far in a conversation, any documents, search results or data the application inserted, and the answer as it is being written, since output tokens occupy the same window.

Anything outside the window is invisible. A chat assistant that seems to remember an earlier turn does so because the application sent that turn back in with the new message. When the conversation exceeds the window, something has to be dropped, and what is dropped is forgotten.

PromptHistoryDocumentsAnswer
Everything must fit inside one window, including the answer. Anything outside it is invisible to the model.
An illustration of the idea, not real proportions. Every token in the window is also billed as input on every request.

Sizes

Windows have grown quickly: a few thousand tokens in 2020, 128,000 as a common size by 2024, and several models now offering a million tokens or more, enough for entire codebases or hundreds of pages of documents in one request. The numbers are maximums; the practical working size is set by cost, latency and attention, all of which degrade as the window fills.

What happens when it overflows

The API refuses the request, or the application truncates the input first, usually by dropping the oldest conversation turns. Symptoms in a product: the assistant forgets instructions given at the start of a long session, contradicts something it said earlier, loses track of the user's name or goal, or stops following the system prompt because it was pushed out. These are context management failures, and they look to users like the model becoming stupid mid-conversation.

Lost in the middle

A large window does not mean the model uses all of it equally. Research on long-context behaviour has repeatedly found that models attend most reliably to the beginning and end of the context and least to the middle: a fact placed in the middle of a long document set is more likely to be missed than the same fact at either end. Performance also tends to decline as the total context grows, even within the limit. The implication for product design is that filling the window with everything that might be relevant is worse than supplying the small amount that is, placed where the model will attend to it.

Cost

Every token in the window is billed as input on every request. A conversation that resends 50,000 tokens of history with each message costs 50,000 input tokens per message, and a system prompt of 3,000 tokens is 3,000 tokens on every call forever. Long contexts are also slower, since the model must process the whole window before producing output. Prompt caching, offered by several providers, discounts repeated prefixes substantially and is the main mitigation for large stable contexts. The cost model of any LLM feature is mostly a context-size model.

Managing it in a product

Keep the system prompt lean. Instructions, not essays. Every token there is paid on every request.

Trim history. Keep recent turns in full and drop or compress older ones. Most conversations need only the last few exchanges verbatim.

Summarise. Replace old turns with a short running summary the model maintains, so long sessions keep their gist without their bulk.

Retrieve, do not dump. For documents and knowledge, use retrieval to select the few chunks relevant to the current question rather than supplying the whole corpus. This is what RAG is for, and it is the correct answer even when the corpus would fit.

Place important material at the edges. Instructions at the start, the current question at the end, retrieved passages between, with the most relevant nearest the question.

Set explicit limits. Budget tokens per component, system prompt, history, retrieval, answer, and enforce them in code.

Store state outside the model. User preferences, facts established, task progress: keep them in the application and inject a compact version, rather than relying on the model to remember from history.

Watch for silent truncation in libraries that drop tokens without telling you.

When a large window is the right tool

Analysing a whole document, contract or codebase where the question genuinely depends on all of it. One-off tasks where retrieval setup is not worth building. Cases where the relevant material cannot be identified in advance. Large windows make these possible; they do not make them cheap or make the lost-in-the-middle effect disappear, so even then, the question should go at the end and the output should be checked.

The context window is the constraint every LLM product is designed around, whether the team knows it or not. Products that manage it deliberately stay coherent, fast and affordable; products that rely on the maximum discover the limit in production. If you would like an AI feature's context strategy reviewed before it ships, book a call.

Common questions

What counts toward the context window?

Everything in the request and the response: the system prompt, all conversation turns sent, any documents, search results or data inserted by the application, and the output tokens the model generates. They share one limit, so a long history or a large document leaves less room for the answer.

What happens when the context window is full?

The API rejects the request, or the application truncates the input, usually by dropping the oldest conversation turns. In a product this appears as the assistant forgetting earlier instructions, contradicting itself, or ignoring the system prompt after it is pushed out. Managing history and retrieval prevents it.

Is a bigger context window always better?

No. Cost and latency grow with everything in the window, and models attend less reliably to material in the middle of very long contexts, so accuracy can fall as the window fills. A large window makes whole-document tasks possible; for most products, supplying a small amount of relevant context beats supplying everything.

How do chatbots remember previous messages?

The application sends the earlier messages back to the model with each new one, so they are inside the context window. The model itself retains nothing between requests. When the history grows too long, the application trims or summarises it, which is why assistants can forget things from early in a long conversation.

What is the lost-in-the-middle problem?

The finding, from research on long-context models, that they attend most reliably to information at the beginning and end of the context and least to information in the middle, with overall accuracy tending to decline as context length grows. It argues for retrieving only relevant material and placing instructions first and the question last.

Have something worth building right?