AI Engineering Active Concepts #013 — RAG: How AI Answers Questions Using Your Documents

AIYING | AI ENGINEERING ACTIVE CONCEPTS #013

Retrieval-Augmented Generation (RAG): How AI Answers Questions Using Your Documents

Instead of expecting AI to know everything, give it a way to find relevant information before answering.

Understand → Connect → Apply → Accomplish

1. The Problem: AI Does Not Know Your Documents

Imagine an organization with hundreds of policy documents, technical manuals, project reports, and operating procedures.

An employee asks:

"What is our organization's current policy for approving AI-generated software code?"

A general-purpose language model may not have access to that organization's latest policy.

Even if the model knows common industry practices, that knowledge is not a substitute for the organization's actual approved document.

This creates an important engineering challenge:

How can an AI system answer questions using information it was not specifically trained on?

2. The Active Concept: Retrieval-Augmented Generation

Retrieval-Augmented Generation, commonly called RAG, is an approach that combines information retrieval with language-model generation.

A typical RAG application first searches an external knowledge source for information relevant to the user's question.

It then provides the retrieved information to a language model as context for generating a response.

QUESTION → RETRIEVE → AUGMENT → GENERATE → VERIFY

Verification is shown here as an important application-level engineering step. It is not automatically guaranteed by RAG.

The essential idea is simple:

Do not rely only on what the model learned. Retrieve what the application needs to know.

3. How a Typical RAG System Works

Step 1 — Prepare the knowledge source

Documents are collected, extracted, cleaned, and organized.

Long documents may be divided into smaller passages called chunks.

Step 2 — Index the content

A common approach converts document chunks into numerical representations called embeddings.

These can be stored in a vector index for similarity-based retrieval.

Other implementations use keyword search, metadata filtering, or hybrid retrieval.

Step 3 — Receive the question

A user submits a question. The system prepares it for retrieval.

Step 4 — Retrieve relevant passages

The retrieval component identifies candidate passages related to the question.

Some systems also rerank results to improve relevance.

Step 5 — Generate a grounded response

The application supplies selected passages along with instructions and the user's question to a language model.

The model generates an answer using the supplied context.

Step 6 — Evaluate the answer

A well-designed application checks whether the answer is supported by the retrieved evidence.

It may provide citations, acknowledge missing information, or request clarification.

4. Connect RAG to Other AI Engineering Concepts

Concept	Connection to RAG
Tokens	Retrieved passages consume context-window capacity.
Embeddings	Support semantic retrieval in many RAG architectures.
Tool Calling	Can let an application invoke retrieval tools dynamically.
MCP	Can expose external resources and retrieval tools through standardized integrations.
Evaluation	Measures retrieval relevance, answer correctness, and evidence support.
Observability	Helps trace retrieved passages, model calls, latency, and failures.

5. Try It Yourself — Make a Document Answer Questions

You can explore document-grounded question answering without writing code.

Open Google NotebookLM .

NotebookLM supports source-grounded exploration of uploaded or connected material. It is useful for demonstrating the general idea of retrieval-grounded answering, although its internal implementation should not be assumed to match every RAG architecture.

Step 1 — Create a fictional policy document

Create a short document with the following fictional policy:

Sample AI Code Review Policy

All AI-generated production code must undergo human review before deployment.

Security-sensitive changes require approval from the designated security reviewer.

Automated tests must pass before release.

The engineering lead is responsible for recording the final deployment approval.

This is a fictional learning example, not an actual organizational policy.

Step 2 — Add the document

Create a notebook and add the fictional policy as a source, using an available supported source-upload method.

Step 3 — Ask a question

Who must review AI-generated production code before deployment?

Compare the answer with the original document. Inspect any available source citations.

Step 4 — Ask a question with no answer

What is the maximum permitted deployment budget for AI-generated code?

The fictional policy does not specify a budget.

A well-grounded answer should acknowledge that the source does not provide this information, rather than inventing a number.

Step 5 — Ask for supporting evidence

Answer using only the supplied policy. Identify the supporting passage. If the information is absent, state that it is not specified.

Check whether the response matches the actual source text.

6. What Should an AI Engineer Evaluate?

A functioning RAG demonstration is not automatically a reliable production system.

Engineers should evaluate several dimensions:

• Retrieval relevance: Were the right passages found?
• Answer correctness: Does the response answer the question accurately?
• Groundedness: Are the claims supported by the supplied sources?
• Abstention: Does the system acknowledge missing evidence?
• Access control: Can users retrieve only authorized information?
• Latency and cost: Can the application meet operational requirements?

RAG can reduce some unsupported responses by supplying relevant evidence. It does not eliminate hallucinations.

Retrieved documents can also contain incorrect, outdated, sensitive, or malicious instructions. They must be handled as lower-trust content.

7. A Practical Engineering Decision

When should an application consider RAG?

RAG is particularly useful when answers depend on information that changes, is organization-specific, or must be traceable to external sources.

Examples include:

• Internal knowledge assistants
• Technical documentation search
• Customer support knowledge systems
• Policy and procedure question answering
• Research assistants using selected sources

However, RAG is not always necessary.

Simple deterministic lookups may be better handled through direct database queries or APIs. Tasks requiring exact calculations may need specialized tools.

The engineering question is not "Can we use RAG?"

It is: "Does retrieval improve the reliability, relevance, and maintainability of this application?"

8. Today's Accomplishment

Complete this small experiment:

1. Create one fictional source document.
2. Ask two questions supported by the document.
3. Ask one question the document cannot answer.
4. Check the answers against the original source.
5. Record one strength and one limitation.

You will have explored the difference between generating an answer from general model capabilities and answering with specific external evidence.

The Bigger Insight

A useful AI system does not need to know everything. It needs a reliable way to find, use, and verify what matters.

RAG shifts part of the engineering challenge from storing knowledge in model parameters to designing reliable access to external knowledge.

That shift makes retrieval quality, source governance, evaluation, and system design central to the AI engineer's work.

Further Reading

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• LlamaIndex Documentation
• LangChain Documentation

AIYING — From AI Awareness to Accomplishment

AI Engineering Active Concepts explores the technologies, architectures, engineering decisions, and practices behind modern AI applications.

AIYING | Ravindra Dastikop

AI Engineering Living Glossary #004 — RAG: How AI Grounds Its Answers in External Knowledge

AI ENGINEERING | LIVING GLOSSARY | #004

RAG — RETRIEVAL-AUGMENTED GENERATION

How AI Grounds Its Answers in External Knowledge

AIYING — Accomplishing with AI

Imagine asking an AI assistant about your company's latest travel reimbursement policy.

The language model may not have encountered that policy during training. Even if it has, the policy might have changed.

How can the assistant answer using the organization's actual documents rather than relying only on information encoded in the model?

One important solution is Retrieval-Augmented Generation (RAG).

Engineering insight: RAG connects language generation to external information retrieval. Its purpose is to make relevant evidence available to the model at answer time.

Contents

1. What Is RAG?
2. Why RAG Matters
3. How RAG Works
4. Understanding RAG Architecture
5. Retrieval Quality and Grounding
6. Industry Applications
7. Engineering Decisions
8. Practical Industry Case
9. Try It Yourself
10. Connected Glossary Terms
11. Living Update
12. Key Takeaway
13. References

1. What Is RAG?

Retrieval-Augmented Generation is an approach that combines information retrieval with generative AI.

Instead of generating a response solely from model parameters and the user's question, a RAG application retrieves relevant external information and provides it as additional context.

The model then uses that context to generate a response.

Question: "What is our company's hotel reimbursement limit?"

Retrieved source: The current company travel policy.

Generated response: An answer based on the relevant policy passage, ideally with a source citation.

RAG does not necessarily require a vector database. Retrieval may use keyword search, vector search, structured queries, hybrid search, or other methods.

↑ Back to contents

2. Why RAG Matters

Language models have several limitations when used without access to relevant external information.

• They may lack knowledge of private organizational documents.
• Their learned information may be outdated.
• They may generate unsupported or incorrect statements.
• They may not know which sources justify an answer.

RAG helps address these challenges by making relevant external content available during generation.

Without retrieval	With RAG
Answer relies on existing model context and learned parameters.	Answer can incorporate retrieved external information.
Private documents may be unavailable.	Authorized documents can be retrieved.
Source attribution may be difficult.	Retrieved passages can support citations.
Updates may require changes to supplied context or model behavior.	Updated indexed documents can be retrieved without retraining the model.

Important: RAG can improve factual grounding, but it does not guarantee accuracy. Retrieval errors, outdated sources, misleading documents, and model mistakes can still occur.

↑ Back to contents

3. How RAG Works

The term describes three connected operations.

Retrieve

Find information relevant to the user's question from an external source.

Augment

Add the retrieved information to the context supplied to the generative model.

Generate

Use the model to produce a response informed by the supplied information.

User Question
      |
      v
Retrieve Relevant Information
      |
      v
Combine Question + Evidence
      |
      v
Language Model
      |
      v
Generated Answer
      |
      v
Source References / Verification

Real systems may add query rewriting, metadata filtering, reranking, tool use, validation, and human review.

↑ Back to contents

4. Understanding RAG Architecture

A typical document-based RAG application has two major workflows.

A. Knowledge preparation

1. Collect approved source documents.
2. Extract and clean their content.
3. Divide long documents into suitable passages.
4. Generate embeddings when using vector retrieval.
5. Index passages in a searchable system.
6. Preserve source references and access-control metadata.

B. Question answering

1. Receive the user's question.
2. Identify relevant sources using a retrieval method.
3. Apply access controls and relevance filters.
4. Optionally rerank retrieved passages.
5. Construct a prompt containing the question and selected evidence.
6. Generate an answer using the model.
7. Attach source references and evaluate the result.

Component	Engineering responsibility
Document pipeline	Prepare accurate and current source content.
Retrieval system	Find relevant information.
Context builder	Select evidence that fits the model's context budget.
Language model	Generate a useful response.
Verification layer	Check citations, relevance, and answer support.
Security controls	Prevent unauthorized information disclosure.

↑ Back to contents

5. Retrieval Quality and Grounding

RAG performance depends heavily on the quality of retrieved information.

Consider two possible retrieval results for a travel policy question:

Result A: An outdated travel reimbursement policy.

Result B: The currently approved reimbursement policy.

Both documents may be semantically similar to the question, but only one may be appropriate for the answer.

Therefore, similarity alone is insufficient.

Engineers must consider:

• Document relevance
• Source authority
• Document freshness
• Version control
• Access permissions
• Completeness of retrieved evidence
• Faithfulness of the generated answer

Engineering caution: A fluent answer with a citation is not necessarily a correct answer. The citation must actually support the claim.

Retrieval evaluation and answer evaluation should be treated as distinct activities.

↑ Back to contents

6. Industry Applications

Application	How RAG contributes
Enterprise knowledge assistants	Answers questions using authorized company information.
Customer support	Uses product manuals and support documentation.
Developer assistants	Retrieves relevant code documentation and technical references.
Research assistants	Finds relevant publications and supporting passages.
Financial operations	Retrieves applicable policies and procedural guidance.
Education	Supports answers grounded in selected learning materials.

In regulated or high-stakes applications, retrieval must be accompanied by appropriate verification, controls, and human oversight.

↑ Back to contents

7. What Must an AI Engineer Decide?

A. What information should be retrievable?

Define approved knowledge sources, data ownership, and document update procedures.

B. Which retrieval method should be used?

Choose keyword, vector, hybrid, structured, or other retrieval approaches according to the task.

C. How should documents be divided?

Chunking affects retrieval precision and the completeness of context.

D. How much evidence should be supplied?

Too little evidence may omit necessary facts. Too much can increase cost and introduce irrelevant information.

E. How should conflicting documents be handled?

Define source priority, effective dates, version rules, and escalation procedures.

F. How should access permissions work?

Retrieval must enforce authorization before protected information is exposed to the model or user.

G. How should prompt injection be addressed?

Retrieved documents are data, not trusted instructions. Applications must avoid treating malicious document text as authority over system behavior.

H. How should the system be evaluated?

Measure retrieval relevance, answer faithfulness, citation support, latency, cost, and successful task completion.

Engineering principle: Design RAG as an evidence-handling system, not merely a vector search feature attached to a chatbot.

↑ Back to contents

8. Practical Industry Case: Company Policy Assistant

A company maintains HR, travel, procurement, and IT policies.

Employees ask questions such as:

"What expenses can I claim when traveling to a client location?"

The AI assistant must use the current approved policy and avoid disclosing restricted information.

Engineering workflow

1. Collect current approved policy documents.
2. Preserve version numbers and effective dates.
3. Extract and index relevant passages.
4. Associate each passage with its source and access permissions.
5. Receive the employee's question.
6. Retrieve authorized and relevant policy passages.
7. Prioritize the applicable policy version.
8. Generate an answer using the retrieved evidence.
9. Include citations to the policy passages.
10. Check whether the answer is supported.
11. Escalate ambiguous or conflicting cases to an authorized human reviewer.

Expected benefit: Employees can locate policy information more efficiently while retaining traceability to approved documents.

Important limitation: The assistant should not invent reimbursement limits or assume that retrieved documents are complete.

↑ Back to contents

9. Try It Yourself

Mini Project: Build a Small RAG Demonstration

Objective: Understand how retrieval changes the evidence available to a language model.

1. Prepare five short fictional company policy documents.
2. Include a travel reimbursement policy with a specific limit.
3. Ask an AI model a question about the limit without providing the documents.
4. Record the response and whether it expresses uncertainty.
5. Retrieve the relevant policy passage using manual selection or a search tool.
6. Supply that passage as context and ask the same question.
7. Request a citation to the supporting passage.
8. Check whether the response is fully supported.
9. Replace the policy with an updated version and repeat the test.
10. Try a question that the documents cannot answer.

Reflection questions

• Did retrieval improve the factual support for the answer?
• Did the model cite the correct passage?
• What happened when evidence was missing?
• How did the system respond to an updated policy?
• What controls would be necessary for real company documents?

Success criterion: The assistant should answer supported questions from the provided evidence and acknowledge when the evidence is insufficient.

↑ Back to contents

10. Connected Glossary Terms

• Token: Determines the information budget available to the model.
• Embedding: Represents information numerically for semantic retrieval.
• Vector Database: Stores and searches vectors in many RAG architectures.
• Context Window: Limits how much retrieved evidence can be supplied.
• Prompt Engineering: Helps specify how the model should use evidence.
• Evals: Measures retrieval and generation quality.
• Guardrails: Helps enforce application policies and boundaries.

Previous lessons:

• #001 — Token
• #002 — Embedding
• #003 — Vector Database

Next lesson: #005 — Context Window.

↑ Back to contents

11. Living Update

Editorial baseline: November 2026

RAG engineering continues to develop beyond basic vector retrieval followed by a single model response.

Important design directions include:

• Hybrid retrieval combining lexical and vector search
• Query rewriting and decomposition
• Reranking and relevance optimization
• Adaptive retrieval based on question complexity
• Structured and graph-based retrieval
• Multimodal retrieval
• Source freshness and version-aware retrieval
• Retrieval and answer evaluation
• Security against prompt injection and unauthorized access

These techniques should be selected according to measured application needs rather than adopted solely because they are new.

Living Glossary note: Revisit this entry as retrieval techniques, model capabilities, and evaluation practices evolve.

↑ Back to contents

12. Key Takeaway

RAG makes external evidence available to a generative AI model, enabling responses grounded in information beyond the model's learned parameters.

But evidence must be relevant, current, authorized, and correctly interpreted.

The engineer's responsibility is not simply to retrieve information. It is to design a system that uses evidence appropriately and produces answers that users can verify.

Think like an AI engineer: Can we show not only what the AI answered, but why the available evidence supports that answer?

↑ Back to contents

13. References and Further Reading

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• LangChain — Documentation
• LlamaIndex — Documentation
• OpenAI — Retrieval Guide
• OWASP — LLM Application Security Risks

↑ Back to contents

AI Engineering — Living Glossary

One Term. One Industry Connection. One Practical Insight. One Action.

AIYING — Accomplishing with AI

Episode #004 | RAG

↑ Return to top