AI ENGINEERING | LIVING GLOSSARY | #005
CONTEXT WINDOW
How Much Information Can AI Consider at Once?
AIYING — Accomplishing with AI
Imagine an AI assistant reviewing a lengthy engineering specification, several design documents, and a long conversation with a developer.
At some point, the total information may exceed what the model can accommodate in its active context.
What happens then? Which information should be retained, summarized, retrieved again, or excluded?
These are questions of context window management.
1. What Is a Context Window?
A context window is the maximum amount of tokenized information a language model can accommodate within its active processing context, subject to the model's architecture and provider-specific limits.
The context may include:
- System and developer instructions
- User questions
- Conversation history
- Retrieved documents
- Tool responses
- Other supported input content
- Tokens generated during the interaction
Providers may specify separate input limits, output limits, or combined context limits. Engineers must consult the documentation for the particular model and API.
The context window is sometimes described as a model's working memory. This is a useful analogy, but it should not be confused with permanent storage or human memory.
↑ Back to contents2. Why Context Windows Matter
AI applications often need to process more information than a single user question.
Consider a technical support assistant that receives:
- A user's question
- Previous conversation messages
- Technical documentation
- Error logs
- Tool results
- Application instructions
All this information competes for the available context budget.
| Engineering dimension | Why context matters |
|---|---|
| RAG | Determines how much retrieved evidence can be supplied. |
| Long-document analysis | Influences document selection, segmentation, and processing. |
| AI agents | Affects handling of conversation history and tool outputs. |
| Code assistance | Influences which files and dependencies can be considered together. |
| Cost | Larger inputs may increase token-based usage charges. |
| Performance | Longer inputs can affect latency and answer quality. |
3. How Context Windows Work
In a typical language-model interaction, an application constructs a sequence of information to send to the model.
User Question
+
System Instructions
+
Conversation History
+
Retrieved Information
+
Tool Results
|
v
Active Model Context
|
v
Response Generation
Each component consumes part of the available token budget.
If the request exceeds supported limits, an API may reject it. Some applications or services instead truncate, summarize, or otherwise manage context before sending the request.
Engineers should not assume that older conversation content remains available unless their application explicitly preserves or resupplies it.
↑ Back to contents4. Understanding Token Budgets
Consider a hypothetical model with a combined context capacity of 128,000 tokens.
An application might allocate the budget as follows:
| Component | Tokens |
|---|---|
| Instructions | 3,000 |
| User question and conversation | 5,000 |
| Retrieved documents | 90,000 |
| Reserved response capacity | 20,000 |
| Safety margin | 10,000 |
| Total | 128,000 |
This is an illustrative planning budget, not a requirement or guarantee for any specific model.
The engineer must also consider separate maximum-output limits, hidden overhead, and provider-specific token accounting.
5. Why More Context Is Not Always Better
A larger context window allows more information to be supplied, but it does not guarantee that the model will use every detail equally well.
A. Irrelevant information
Unnecessary documents may distract from the information needed for the task.
B. Conflicting information
Different versions of a policy or specification may create ambiguity.
C. Position effects
Some models and tasks exhibit differences in how effectively information is used depending on where it appears in a long context.
D. Cost and latency
Large contexts may require more computation and increase operating costs.
E. Lost requirements
Important instructions may be omitted by application-level truncation or summarization.
Research on the "lost in the middle" phenomenon illustrates that successful retrieval and use of information within long contexts can depend on position and model behavior.
6. Industry Applications
| Application | Context engineering challenge |
|---|---|
| Enterprise knowledge assistants | Selecting relevant evidence from large document collections. |
| Software engineering assistants | Managing code, dependencies, instructions, and tool results. |
| Customer support | Preserving essential conversation history. |
| Document analysis | Comparing long reports and specifications. |
| AI agents | Maintaining task state across multi-step workflows. |
| Research assistants | Organizing evidence from multiple sources. |
7. What Must an AI Engineer Decide?
A. What information is essential?
Identify instructions, facts, constraints, and evidence necessary for task completion.
B. What should be retrieved on demand?
Use retrieval systems instead of placing every available document into the prompt.
C. What should be summarized?
Compress older information when appropriate, while preserving critical requirements and references.
D. What must never be silently discarded?
Preserve safety constraints, user-approved decisions, and important task requirements.
E. How should context be structured?
Separate instructions, evidence, conversation state, and tool outputs clearly.
F. How should long conversations be handled?
Consider rolling summaries, persistent state, retrieval, and selective history retention.
G. How should performance be measured?
Evaluate answer quality, requirement retention, latency, token consumption, and successful task completion.
H. How should untrusted content be treated?
Retrieved documents and tool outputs may contain malicious instructions. They should be treated as data rather than automatically trusted commands.
8. Practical Industry Case: Long-Running Engineering Assistant
Consider an AI assistant supporting a software engineering team over several weeks.
The assistant works with:
- Product requirements
- Architecture decisions
- Source code
- Test results
- Bug reports
- Conversation history
As the project develops, the accumulated information becomes too large to supply in every request.
Engineering approach
- Identify stable project requirements.
- Maintain a structured record of approved decisions.
- Store documents and code in appropriate external systems.
- Retrieve task-relevant information when needed.
- Summarize older conversations while preserving essential decisions.
- Separate current task instructions from retrieved evidence.
- Reserve sufficient output capacity.
- Check generated work against original requirements.
- Evaluate whether critical information was retained.
- Update persistent project state after approved changes.
Expected benefit: The assistant can continue useful work without repeatedly supplying every historical interaction.
Important limitation: Summaries can omit or distort details. Critical requirements should remain independently accessible and verifiable.
↑ Back to contents9. Try It Yourself
Mini Experiment: More Context Versus Better Context
Objective: Discover whether adding more information always improves AI output.
- Select a question requiring evidence from a short technical document.
- Prepare a prompt containing only the relevant passage.
- Prepare a second prompt containing the same passage plus several unrelated passages.
- Prepare a third prompt containing the relevant passage and a concise summary of supporting information.
- Run all three prompts using the same model and settings.
- Compare factual correctness and completeness.
- Measure input tokens and response time where available.
- Check whether important requirements were followed.
- Repeat the experiment with the relevant passage placed in different positions.
- Record which context design produces the most reliable result.
Reflection questions
- Did more information improve the answer?
- Did irrelevant context create errors or confusion?
- Did summarization preserve essential facts?
- Did the position of relevant information matter?
- Which design provided the best balance of quality, cost, and speed?
Success criterion: Identify a context-selection strategy that reliably supports the task without unnecessary information.
↑ Back to contents10. Connected Glossary Terms
- Token: The unit used to measure context consumption.
- Embedding: Supports retrieval of relevant information.
- Vector Database: Helps store and retrieve external knowledge.
- RAG: Supplies selected external evidence to a model.
- Prompt Engineering: Organizes instructions and context.
- AI Agent: May require context and state management across many steps.
- Evals: Measures whether context design supports successful outcomes.
Previous lessons:
Next lesson: #006 — Prompt Engineering.
↑ Back to contents11. Living Update
Editorial baseline: November 2026
Context engineering has become an important design concern for AI applications involving long documents, retrieval, tools, and multi-step workflows.
Relevant techniques and developments to monitor include:
- Long-context model architectures
- Retrieval-based context selection
- Context compression and summarization
- Prompt caching
- Persistent task state and external memory systems
- Adaptive context construction
- Long-context evaluation
- Tool-output filtering and security
Context-window specifications vary across models, providers, and API configurations. Engineers should verify current limits before designing applications around them.
Living Glossary note: Revisit this entry as model context capacities, effective long-context performance, and application-level context management techniques evolve.
↑ Back to contents12. Key Takeaway
A larger context window is a capability, not a substitute for information architecture.
Think like an AI engineer: What information must the model have now, what can be retrieved later, and what must remain reliably preserved outside the model?
↑ Back to contents13. References and Further Reading
- OpenAI — Model Specifications
- Anthropic — Developer Documentation
- Google — Gemini API Documentation
- Liu et al. — Lost in the Middle: How Language Models Use Long Contexts
- OpenAI — Prompt Caching
