AI Engineering Active Concepts #013 — RAG: How AI Answers Questions Using Your Documents

AIYING | AI ENGINEERING ACTIVE CONCEPTS #013

Retrieval-Augmented Generation (RAG): How AI Answers Questions Using Your Documents

Instead of expecting AI to know everything, give it a way to find relevant information before answering.

Understand → Connect → Apply → Accomplish

1. The Problem: AI Does Not Know Your Documents

Imagine an organization with hundreds of policy documents, technical manuals, project reports, and operating procedures.

An employee asks:

"What is our organization's current policy for approving AI-generated software code?"

A general-purpose language model may not have access to that organization's latest policy.

Even if the model knows common industry practices, that knowledge is not a substitute for the organization's actual approved document.

This creates an important engineering challenge:

How can an AI system answer questions using information it was not specifically trained on?

2. The Active Concept: Retrieval-Augmented Generation

Retrieval-Augmented Generation, commonly called RAG, is an approach that combines information retrieval with language-model generation.

A typical RAG application first searches an external knowledge source for information relevant to the user's question.

It then provides the retrieved information to a language model as context for generating a response.

QUESTION → RETRIEVE → AUGMENT → GENERATE → VERIFY

Verification is shown here as an important application-level engineering step. It is not automatically guaranteed by RAG.

The essential idea is simple:

Do not rely only on what the model learned. Retrieve what the application needs to know.

3. How a Typical RAG System Works

Step 1 — Prepare the knowledge source

Documents are collected, extracted, cleaned, and organized.

Long documents may be divided into smaller passages called chunks.

Step 2 — Index the content

A common approach converts document chunks into numerical representations called embeddings.

These can be stored in a vector index for similarity-based retrieval.

Other implementations use keyword search, metadata filtering, or hybrid retrieval.

Step 3 — Receive the question

A user submits a question. The system prepares it for retrieval.

Step 4 — Retrieve relevant passages

The retrieval component identifies candidate passages related to the question.

Some systems also rerank results to improve relevance.

Step 5 — Generate a grounded response

The application supplies selected passages along with instructions and the user's question to a language model.

The model generates an answer using the supplied context.

Step 6 — Evaluate the answer

A well-designed application checks whether the answer is supported by the retrieved evidence.

It may provide citations, acknowledge missing information, or request clarification.

4. Connect RAG to Other AI Engineering Concepts

Concept	Connection to RAG
Tokens	Retrieved passages consume context-window capacity.
Embeddings	Support semantic retrieval in many RAG architectures.
Tool Calling	Can let an application invoke retrieval tools dynamically.
MCP	Can expose external resources and retrieval tools through standardized integrations.
Evaluation	Measures retrieval relevance, answer correctness, and evidence support.
Observability	Helps trace retrieved passages, model calls, latency, and failures.

5. Try It Yourself — Make a Document Answer Questions

You can explore document-grounded question answering without writing code.

Open Google NotebookLM .

NotebookLM supports source-grounded exploration of uploaded or connected material. It is useful for demonstrating the general idea of retrieval-grounded answering, although its internal implementation should not be assumed to match every RAG architecture.

Step 1 — Create a fictional policy document

Create a short document with the following fictional policy:

Sample AI Code Review Policy

All AI-generated production code must undergo human review before deployment.

Security-sensitive changes require approval from the designated security reviewer.

Automated tests must pass before release.

The engineering lead is responsible for recording the final deployment approval.

This is a fictional learning example, not an actual organizational policy.

Step 2 — Add the document

Create a notebook and add the fictional policy as a source, using an available supported source-upload method.

Step 3 — Ask a question

Who must review AI-generated production code before deployment?

Compare the answer with the original document. Inspect any available source citations.

Step 4 — Ask a question with no answer

What is the maximum permitted deployment budget for AI-generated code?

The fictional policy does not specify a budget.

A well-grounded answer should acknowledge that the source does not provide this information, rather than inventing a number.

Step 5 — Ask for supporting evidence

Answer using only the supplied policy. Identify the supporting passage. If the information is absent, state that it is not specified.

Check whether the response matches the actual source text.

6. What Should an AI Engineer Evaluate?

A functioning RAG demonstration is not automatically a reliable production system.

Engineers should evaluate several dimensions:

• Retrieval relevance: Were the right passages found?
• Answer correctness: Does the response answer the question accurately?
• Groundedness: Are the claims supported by the supplied sources?
• Abstention: Does the system acknowledge missing evidence?
• Access control: Can users retrieve only authorized information?
• Latency and cost: Can the application meet operational requirements?

RAG can reduce some unsupported responses by supplying relevant evidence. It does not eliminate hallucinations.

Retrieved documents can also contain incorrect, outdated, sensitive, or malicious instructions. They must be handled as lower-trust content.

7. A Practical Engineering Decision

When should an application consider RAG?

RAG is particularly useful when answers depend on information that changes, is organization-specific, or must be traceable to external sources.

Examples include:

• Internal knowledge assistants
• Technical documentation search
• Customer support knowledge systems
• Policy and procedure question answering
• Research assistants using selected sources

However, RAG is not always necessary.

Simple deterministic lookups may be better handled through direct database queries or APIs. Tasks requiring exact calculations may need specialized tools.

The engineering question is not "Can we use RAG?"

It is: "Does retrieval improve the reliability, relevance, and maintainability of this application?"

8. Today's Accomplishment

Complete this small experiment:

1. Create one fictional source document.
2. Ask two questions supported by the document.
3. Ask one question the document cannot answer.
4. Check the answers against the original source.
5. Record one strength and one limitation.

You will have explored the difference between generating an answer from general model capabilities and answering with specific external evidence.

The Bigger Insight

A useful AI system does not need to know everything. It needs a reliable way to find, use, and verify what matters.

RAG shifts part of the engineering challenge from storing knowledge in model parameters to designing reliable access to external knowledge.

That shift makes retrieval quality, source governance, evaluation, and system design central to the AI engineer's work.

Further Reading

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• LlamaIndex Documentation
• LangChain Documentation

AIYING — From AI Awareness to Accomplishment

AI Engineering Active Concepts explores the technologies, architectures, engineering decisions, and practices behind modern AI applications.

AIYING | Ravindra Dastikop