AI Engineering Active Concepts #005 — Chunking and Reranking: Improving Retrieval Relevance

AI ENGINEERING — ACTIVE CONCEPTS & PRACTICES | #005

Chunking and Reranking: Improving Retrieval Relevance

Chapter 1: Engineering the AI Context

AIYING — Accomplishing with AI

The Big Idea: Effective retrieval depends not only on finding documents, but also on deciding how information is divided and which retrieved passages deserve priority.

Chunking shapes the units of knowledge available for retrieval. Reranking improves the ordering of retrieved candidates. Together, they can improve the evidence supplied to retrieval-augmented generation (RAG) systems.

Engineering question: How can we deliver the most useful evidence to an AI model without losing essential context or wasting tokens?

Contents

1. Why Chunking and Reranking Matter
2. UNDERSTAND — The Two Practices
3. CONNECT — The AI Engineering Ecosystem
4. How Chunking and Reranking Work Together
5. APPLY — Enterprise Technical Documentation
6. Worked Retrieval Example
7. Engineering Decisions and Trade-offs
8. Active Industry Practices
9. ACCOMPLISH — Practical Exercise
10. Student and Practitioner Project
11. Check Your Understanding
12. Active Connections
13. References and Further Reading

1. Why Chunking and Reranking Matter

Imagine an AI assistant helping maintenance engineers troubleshoot industrial equipment.

The organization's knowledge base contains hundreds of technical manuals. Each manual may include operating procedures, troubleshooting tables, warnings, maintenance instructions, and service bulletins.

A technician asks:

"Why does the A100 packaging machine stop after approximately 20 minutes of operation?"

The relevant explanation may appear in a small paragraph inside a 200-page maintenance manual.

Retrieving the entire manual would introduce excessive information. Retrieving a tiny fragment may omit an important warning or condition.

Even after relevant passages are retrieved, the first search result may not be the most useful one.

These are two different engineering problems:

• Chunking: How should the original documents be divided into retrievable units?
• Reranking: Which of the retrieved units best answers the specific question?

Chunking determines what can be found. Reranking helps determine what should be used first.

↑ Back to Contents

2. UNDERSTAND — The Two Practices

What is chunking?

Chunking is the process of dividing larger documents or information sources into smaller units suitable for indexing, retrieval, and downstream processing.

For example, a technical manual might be divided into sections describing installation, operation, troubleshooting, maintenance, and safety.

These sections may then be divided further into passages that fit the application's retrieval and context requirements.

Common chunking strategies

Strategy	How It Works	Typical Consideration
Fixed-size	Divides text by a predefined token or character count	Simple, but may split related information
Overlapping	Repeats some content between adjacent chunks	Preserves boundary context but increases duplication
Recursive	Splits by larger boundaries first, then smaller ones	Can preserve paragraphs and sections
Semantic	Uses content relationships to identify boundaries	Potentially coherent chunks, with added complexity
Structure-aware	Uses headings, tables, sections, and document layout	Useful for structured technical documents
Parent-child	Indexes smaller units but retrieves larger surrounding sections	Combines focused search with broader context

Why chunk size matters

Very small chunks may contain precise information but lose surrounding explanations.

Very large chunks preserve more context but may mix unrelated topics and consume more tokens when supplied to a language model.

There is no universally optimal chunk size. The right configuration depends on document structure, embedding models, query patterns, retrieval methods, and evaluation results.

What is reranking?

Reranking is a second-stage retrieval practice that reorders candidate results according to their estimated relevance to a particular query.

An initial retrieval system might return 20 candidate passages using keyword search, embeddings, or hybrid retrieval.

A reranker examines those candidates more closely and assigns relevance scores or produces a revised ordering.

The application may then select the highest-ranked passages for inclusion in the model's context.

Common reranking approaches

• Cross-encoder reranking: Evaluates query-passage pairs jointly.
• LLM-based reranking: Uses a language model to assess candidate relevance.
• Rule-based reranking: Applies explicit factors such as document authority, effective date, or product applicability.
• Combined ranking: Integrates relevance signals from several methods.

Reranking can improve the ordering of retrieved candidates, but it cannot recover a relevant document that was never retrieved in the first stage.

↑ Back to Contents

3. CONNECT — The AI Engineering Ecosystem

Connected Concept	Engineering Relationship
Context Engineering	Controls which selected passages enter the model context
Token Economics	Balances passage size and number against processing cost
Embeddings	Represent chunks for semantic similarity search
Vector Databases	Index and retrieve chunk representations
Retrieval Engineering	Coordinates candidate retrieval, filtering, and ranking
Hybrid Search	Combines lexical and semantic retrieval candidates
RAG	Supplies selected evidence to a generative model
Evaluation	Measures retrieval relevance and downstream answer quality
Observability	Tracks retrieval behavior, latency, and failures
AI Security	Enforces document permissions and safe handling of retrieved content

The important distinction

Embeddings help retrieve candidates based on learned similarity.

Rerankers can use additional signals or more detailed query-document comparisons to reorder those candidates.

Chunking influences both stages because the boundaries of each passage determine which information is represented and evaluated.

↑ Back to Contents

4. How Chunking and Reranking Work Together

Stage A — Document preparation

1. Collect authorized documents.
2. Extract text, tables, headings, and metadata.
3. Choose a chunking strategy.
4. Preserve document and section identifiers.
5. Create lexical or vector indexes.
6. Maintain source versions and access permissions.

Stage B — Query processing

1. Receive a user question.
2. Apply access and applicability constraints.
3. Retrieve candidate chunks.
4. Rerank candidates according to relevance.
5. Remove duplicates and unsuitable passages.
6. Select the evidence that fits the context budget.
7. Generate a response with source references.

Technical Documents

↓

Structure-aware Chunking

↓

Indexing and Embeddings

---

User Question

↓

Candidate Retrieval

↓

Reranking

↓

Top Relevant Authorized Passages

↓

RAG Context → Generated Answer

Chunking generally occurs during document preparation, while reranking occurs during query-time retrieval.

Some systems dynamically construct or expand chunks at query time, so actual implementations may vary.

↑ Back to Contents

5. APPLY — Enterprise Technical Documentation

Scenario: Industrial Packaging Machine Support

A manufacturer has technical documentation for several packaging machine models.

One service bulletin describes a thermal protection mechanism that can interrupt operation under certain conditions.

The bulletin includes:

• Applicable machine models
• Operating conditions
• Possible symptoms
• Safety warnings
• Authorized inspection procedures

Approach A — Fixed-size chunking without reranking

The document is divided into equal-sized text segments.

A segment describing thermal shutdown may be separated from the machine model identifier or a safety warning.

The retrieval system may return a passage that appears relevant but lacks essential context.

Approach B — Structure-aware chunking with reranking

The engineering team preserves logical sections and document metadata.

Chunks retain their relationship to the original bulletin, including product applicability and section identifiers.

The retrieval system finds candidate passages using semantic and lexical search.

A reranker prioritizes passages most relevant to the specific machine model and reported symptom.

The application then supplies appropriate supporting sections to the generative model.

Engineering outcome

Approach B is designed to improve evidence selection, but its effectiveness must be measured on representative questions.

For safety-critical equipment, retrieved guidance should not replace approved procedures or qualified technical judgment.

↑ Back to Contents

6. Worked Retrieval Example

Suppose a retrieval system returns five passages for the question:

"What could cause an A100 machine to stop after 20 minutes?"

The initial similarity scores are hypothetical and are used only to demonstrate ranking.

Initial Rank	Passage	Initial Score
1	General equipment shutdown causes	0.88
2	A100 thermal shutdown service bulletin	0.84
3	A100 installation checklist	0.79
4	B200 temperature-control guide	0.76
5	Preventive maintenance overview	0.73

The highest similarity score belongs to a generic document, not necessarily the most applicable one.

After reranking, the system might produce:

New Rank	Passage	Illustrative Reranker Score
1	A100 thermal shutdown service bulletin	0.96
2	General equipment shutdown causes	0.83
3	A100 installation checklist	0.70
4	Preventive maintenance overview	0.57
5	B200 temperature-control guide	0.31

Important: Initial retrieval scores and reranker scores may use different scales and are not necessarily directly comparable.

The purpose is to improve ranking quality, not to maximize an arbitrary score.

What changed?

The A100 service bulletin moved to first position because it addresses the exact machine and reported symptom.

The B200 guide moved down because its product applicability is weaker.

This example demonstrates how reranking may prioritize task-specific evidence over broad semantic similarity.

↑ Back to Contents

7. Engineering Decisions and Trade-offs

Engineering Decision	Potential Benefit	Possible Cost or Risk
Smaller chunks	Focused retrieval	Loss of surrounding context
Larger chunks	More complete context	More irrelevant content
Overlapping chunks	Better boundary continuity	Duplicate indexing and retrieval
Semantic chunking	Coherent information units	Additional processing complexity
Structure-aware chunking	Preserves headings and sections	Requires reliable document parsing
More retrieval candidates	Potentially higher recall	Greater reranking cost and latency
Cross-encoder reranking	Potentially stronger relevance estimates	Additional inference cost
LLM reranking	Flexible relevance assessment	Cost, latency, and output variability
Parent-child retrieval	Focused search with wider context	Additional index and retrieval logic

The hidden interaction

Chunking and reranking should not always be optimized independently.

A reranker cannot fully repair chunks that omit essential information. Conversely, excellent chunks may still be poorly ordered by the initial retrieval system.

Engineers should evaluate combinations of chunking and ranking strategies against actual application requirements.

The best retrieval pipeline is the one that consistently delivers sufficient, applicable evidence—not the one with the most sophisticated individual component.

↑ Back to Contents

8. Active Industry Practices

Established practices

• Fixed-size and overlapping chunking
• Recursive text splitting
• Metadata-preserving document ingestion
• Vector and hybrid retrieval
• Second-stage relevance reranking
• Retrieval evaluation using labeled query sets

Evolving practices

• Structure-aware and layout-aware document parsing
• Adaptive chunking based on document type
• Hierarchical and parent-child retrieval
• Contextual enrichment of indexed passages
• Query-aware context expansion
• Cost-aware candidate selection and reranking

Why technical documents require special attention

Technical documents often contain tables, diagrams, exceptions, warnings, and cross-references.

A text-only chunking strategy may separate a table value from its column heading or a safety instruction from the procedure it qualifies.

Production systems may therefore require specialized parsing, layout preservation, and document-specific retrieval methods.

Practice status: Established retrieval engineering techniques with evolving implementation approaches.

Reviewed: November 2026.

↑ Back to Contents

9. ACCOMPLISH — Practical Exercise

Exercise: Design a Better Chunking Strategy

Consider the following fictional technical bulletin:

Document: A100 Thermal Protection Bulletin

Section 1 — Applicability: This bulletin applies to A100 packaging machines manufactured within a specified production range.

Section 2 — Symptoms: Under certain operating conditions, the machine may stop after prolonged operation.

Section 3 — Possible Cause: Thermal protection may activate when monitored temperatures exceed defined limits.

Section 4 — Safety: Inspection must follow approved shutdown and isolation procedures.

Section 5 — Service Guidance: Qualified personnel should consult the applicable maintenance procedure and authorized service documentation.

Your tasks

1. Divide the bulletin into three or four meaningful chunks.
2. Explain why you chose those boundaries.
3. Identify which metadata should accompany every chunk.
4. Determine whether any sections should remain together.
5. Design a retrieval query for the reported A100 shutdown.
6. Propose criteria for reranking candidate passages.
7. Select the evidence that should be supplied to a generative model.
8. Explain how the system should preserve the safety instructions.

Optional AI-assisted prompt

Act as an AI retrieval engineer.

Analyze the technical document below.

1. Identify logical chunk boundaries.
2. Preserve important warnings and conditions.
3. Recommend metadata for each chunk.
4. Explain possible retrieval failure modes.
5. Suggest a reranking strategy.
6. Propose an evaluation test.

Do not invent technical facts.
Use only the supplied document.

Document:
[PASTE DOCUMENT]

Accomplishment: You have designed a chunking and reranking strategy that accounts for meaning, relevance, document applicability, and safety.

↑ Back to Contents

10. Student and Practitioner Project

Mini Project: RAG Retrieval Quality Lab

Objective: Experimentally compare chunking and reranking configurations.

Project requirements

1. Collect 20–30 non-sensitive technical or organizational documents.
2. Prepare at least 15 representative questions.
3. Label relevant source passages for each question.
4. Implement fixed-size chunking as a baseline.
5. Implement a second chunking strategy, such as recursive or structure-aware chunking.
6. Retrieve candidate passages using a consistent retrieval method.
7. Compare results before and after reranking.
8. Measure relevance, latency, and context size.
9. Evaluate downstream answer quality on a small test set.
10. Recommend a production configuration with supporting evidence.

Suggested experimental configurations

Configuration	Chunking	Reranking
A	Fixed-size	No
B	Fixed-size	Yes
C	Structure-aware	No
D	Structure-aware	Yes

Evaluation metrics

• Recall@k: Fraction of relevant passages found in the top k.
• Precision@k: Fraction of top-k passages judged relevant.
• MRR: Mean reciprocal rank of the first relevant result.
• nDCG@k: Ranking quality when relevance has multiple grades.
• Retrieval latency: Time required to retrieve and rerank results.
• Context tokens: Number of tokens passed to the generative model.
• Answer groundedness: Whether the generated response is supported by selected evidence.

Because chunking changes passage boundaries, use a consistent source-level relevance mapping when comparing configurations. Otherwise, passage-level metrics may not be directly comparable.

Deliverable: A prototype or reproducible experiment, comparison table, and one-page engineering recommendation.

↑ Back to Contents

11. Check Your Understanding

1. Why does chunk size affect retrieval quality?
2. How does overlapping chunking differ from fixed-size chunking without overlap?
3. What is the purpose of a reranker?
4. Why can reranking not compensate for missing candidates?
5. How can structure-aware chunking improve technical document retrieval?
6. What trade-offs arise when reranking more candidates?
7. Why must safety warnings and document metadata be preserved?
8. How would you determine whether reranking improves a RAG application?

Reflection: If a RAG assistant repeatedly retrieves the correct document but generates incomplete answers, would you first investigate chunking, reranking, context assembly, or generation? What evidence would guide your diagnosis?

↑ Back to Contents

12. Active Connections

• Lesson #001: Context Engineering
• Lesson #002: Token Economics
• Lesson #003: Embeddings in Practice
• Lesson #004: Retrieval Engineering
• Lesson #006: Context Management — Memory, Compression, and State

These lessons show how AI engineers move from selecting information to representing, retrieving, prioritizing, and managing evidence.

↑ Back to Contents

13. References and Further Reading

1. LangChain — Text Splitters
2. Sentence Transformers — Cross-Encoder Applications
3. Cohere — Reranking Overview
4. Microsoft Learn — Retrieval-Augmented Generation Overview
5. Weaviate — Reranking Search Results

Technical documentation and product capabilities change. Verify current APIs, model availability, and operational guidance before implementation.

---

Final Takeaway

Better RAG is not always about retrieving more information. It is about structuring knowledge so the right evidence can be found, and ranking that evidence so the most useful information reaches the model.

AI Engineering — Active Concepts & Practices
Active Concept #005 — Chunking and Reranking

AIYING — Accomplishing with AI

Understand → Connect → Apply → Accomplish