AI ENGINEERING — ACTIVE CONCEPTS & PRACTICES | #005
Chunking and Reranking: Improving Retrieval Relevance
Chapter 1: Engineering the AI Context
AIYING — Accomplishing with AI
The Big Idea: Effective retrieval depends not only on finding documents, but also on deciding how information is divided and which retrieved passages deserve priority.
Chunking shapes the units of knowledge available for retrieval. Reranking improves the ordering of retrieved candidates. Together, they can improve the evidence supplied to retrieval-augmented generation (RAG) systems.
Engineering question: How can we deliver the most useful evidence to an AI model without losing essential context or wasting tokens?
Contents
- Why Chunking and Reranking Matter
- UNDERSTAND — The Two Practices
- CONNECT — The AI Engineering Ecosystem
- How Chunking and Reranking Work Together
- APPLY — Enterprise Technical Documentation
- Worked Retrieval Example
- Engineering Decisions and Trade-offs
- Active Industry Practices
- ACCOMPLISH — Practical Exercise
- Student and Practitioner Project
- Check Your Understanding
- Active Connections
- References and Further Reading
1. Why Chunking and Reranking Matter
Imagine an AI assistant helping maintenance engineers troubleshoot industrial equipment.
The organization's knowledge base contains hundreds of technical manuals. Each manual may include operating procedures, troubleshooting tables, warnings, maintenance instructions, and service bulletins.
A technician asks:
"Why does the A100 packaging machine stop after approximately 20 minutes of operation?"
The relevant explanation may appear in a small paragraph inside a 200-page maintenance manual.
Retrieving the entire manual would introduce excessive information. Retrieving a tiny fragment may omit an important warning or condition.
Even after relevant passages are retrieved, the first search result may not be the most useful one.
These are two different engineering problems:
- Chunking: How should the original documents be divided into retrievable units?
- Reranking: Which of the retrieved units best answers the specific question?
Chunking determines what can be found. Reranking helps determine what should be used first.
↑ Back to Contents
2. UNDERSTAND — The Two Practices
What is chunking?
Chunking is the process of dividing larger documents or information sources into smaller units suitable for indexing, retrieval, and downstream processing.
For example, a technical manual might be divided into sections describing installation, operation, troubleshooting, maintenance, and safety.
These sections may then be divided further into passages that fit the application's retrieval and context requirements.
Common chunking strategies
| Strategy | How It Works | Typical Consideration |
|---|
| Fixed-size | Divides text by a predefined token or character count | Simple, but may split related information |
| Overlapping | Repeats some content between adjacent chunks | Preserves boundary context but increases duplication |
| Recursive | Splits by larger boundaries first, then smaller ones | Can preserve paragraphs and sections |
| Semantic | Uses content relationships to identify boundaries | Potentially coherent chunks, with added complexity |
| Structure-aware | Uses headings, tables, sections, and document layout | Useful for structured technical documents |
| Parent-child | Indexes smaller units but retrieves larger surrounding sections | Combines focused search with broader context |
Why chunk size matters
Very small chunks may contain precise information but lose surrounding explanations.
Very large chunks preserve more context but may mix unrelated topics and consume more tokens when supplied to a language model.
There is no universally optimal chunk size. The right configuration depends on document structure, embedding models, query patterns, retrieval methods, and evaluation results.
What is reranking?
Reranking is a second-stage retrieval practice that reorders candidate results according to their estimated relevance to a particular query.
An initial retrieval system might return 20 candidate passages using keyword search, embeddings, or hybrid retrieval.
A reranker examines those candidates more closely and assigns relevance scores or produces a revised ordering.
The application may then select the highest-ranked passages for inclusion in the model's context.
Common reranking approaches
- Cross-encoder reranking: Evaluates query-passage pairs jointly.
- LLM-based reranking: Uses a language model to assess candidate relevance.
- Rule-based reranking: Applies explicit factors such as document authority, effective date, or product applicability.
- Combined ranking: Integrates relevance signals from several methods.
Reranking can improve the ordering of retrieved candidates, but it cannot recover a relevant document that was never retrieved in the first stage.
↑ Back to Contents
3. CONNECT — The AI Engineering Ecosystem
| Connected Concept | Engineering Relationship |
|---|
| Context Engineering | Controls which selected passages enter the model context |
| Token Economics | Balances passage size and number against processing cost |
| Embeddings | Represent chunks for semantic similarity search |
| Vector Databases | Index and retrieve chunk representations |
| Retrieval Engineering | Coordinates candidate retrieval, filtering, and ranking |
| Hybrid Search | Combines lexical and semantic retrieval candidates |
| RAG | Supplies selected evidence to a generative model |
| Evaluation | Measures retrieval relevance and downstream answer quality |
| Observability | Tracks retrieval behavior, latency, and failures |
| AI Security | Enforces document permissions and safe handling of retrieved content |
The important distinction
Embeddings help retrieve candidates based on learned similarity.
Rerankers can use additional signals or more detailed query-document comparisons to reorder those candidates.
Chunking influences both stages because the boundaries of each passage determine which information is represented and evaluated.
↑ Back to Contents
4. How Chunking and Reranking Work Together
Stage A — Document preparation
- Collect authorized documents.
- Extract text, tables, headings, and metadata.
- Choose a chunking strategy.
- Preserve document and section identifiers.
- Create lexical or vector indexes.
- Maintain source versions and access permissions.
Stage B — Query processing
- Receive a user question.
- Apply access and applicability constraints.
- Retrieve candidate chunks.
- Rerank candidates according to relevance.
- Remove duplicates and unsuitable passages.
- Select the evidence that fits the context budget.
- Generate a response with source references.
Technical Documents
↓
Structure-aware Chunking
↓
Indexing and Embeddings
User Question
↓
Candidate Retrieval
↓
Reranking
↓
Top Relevant Authorized Passages
↓
RAG Context → Generated Answer
Chunking generally occurs during document preparation, while reranking occurs during query-time retrieval.
Some systems dynamically construct or expand chunks at query time, so actual implementations may vary.
↑ Back to Contents
5. APPLY — Enterprise Technical Documentation
Scenario: Industrial Packaging Machine Support
A manufacturer has technical documentation for several packaging machine models.
One service bulletin describes a thermal protection mechanism that can interrupt operation under certain conditions.
The bulletin includes:
- Applicable machine models
- Operating conditions
- Possible symptoms
- Safety warnings
- Authorized inspection procedures
Approach A — Fixed-size chunking without reranking
The document is divided into equal-sized text segments.
A segment describing thermal shutdown may be separated from the machine model identifier or a safety warning.
The retrieval system may return a passage that appears relevant but lacks essential context.
Approach B — Structure-aware chunking with reranking
The engineering team preserves logical sections and document metadata.
Chunks retain their relationship to the original bulletin, including product applicability and section identifiers.
The retrieval system finds candidate passages using semantic and lexical search.
A reranker prioritizes passages most relevant to the specific machine model and reported symptom.
The application then supplies appropriate supporting sections to the generative model.
Engineering outcome
Approach B is designed to improve evidence selection, but its effectiveness must be measured on representative questions.
For safety-critical equipment, retrieved guidance should not replace approved procedures or qualified technical judgment.
↑ Back to Contents
6. Worked Retrieval Example
Suppose a retrieval system returns five passages for the question:
"What could cause an A100 machine to stop after 20 minutes?"
The initial similarity scores are hypothetical and are used only to demonstrate ranking.
| Initial Rank | Passage | Initial Score |
|---|
| 1 | General equipment shutdown causes | 0.88 |
| 2 | A100 thermal shutdown service bulletin | 0.84 |
| 3 | A100 installation checklist | 0.79 |
| 4 | B200 temperature-control guide | 0.76 |
| 5 | Preventive maintenance overview | 0.73 |
The highest similarity score belongs to a generic document, not necessarily the most applicable one.
After reranking, the system might produce:
| New Rank | Passage | Illustrative Reranker Score |
|---|
| 1 | A100 thermal shutdown service bulletin | 0.96 |
| 2 | General equipment shutdown causes | 0.83 |
| 3 | A100 installation checklist | 0.70 |
| 4 | Preventive maintenance overview | 0.57 |
| 5 | B200 temperature-control guide | 0.31 |
Important: Initial retrieval scores and reranker scores may use different scales and are not necessarily directly comparable.
The purpose is to improve ranking quality, not to maximize an arbitrary score.
What changed?
The A100 service bulletin moved to first position because it addresses the exact machine and reported symptom.
The B200 guide moved down because its product applicability is weaker.
This example demonstrates how reranking may prioritize task-specific evidence over broad semantic similarity.
↑ Back to Contents
7. Engineering Decisions and Trade-offs
| Engineering Decision | Potential Benefit | Possible Cost or Risk |
|---|
| Smaller chunks | Focused retrieval | Loss of surrounding context |
| Larger chunks | More complete context | More irrelevant content |
| Overlapping chunks | Better boundary continuity | Duplicate indexing and retrieval |
| Semantic chunking | Coherent information units | Additional processing complexity |
| Structure-aware chunking | Preserves headings and sections | Requires reliable document parsing |
| More retrieval candidates | Potentially higher recall | Greater reranking cost and latency |
| Cross-encoder reranking | Potentially stronger relevance estimates | Additional inference cost |
| LLM reranking | Flexible relevance assessment | Cost, latency, and output variability |
| Parent-child retrieval | Focused search with wider context | Additional index and retrieval logic |
The hidden interaction
Chunking and reranking should not always be optimized independently.
A reranker cannot fully repair chunks that omit essential information. Conversely, excellent chunks may still be poorly ordered by the initial retrieval system.
Engineers should evaluate combinations of chunking and ranking strategies against actual application requirements.
The best retrieval pipeline is the one that consistently delivers sufficient, applicable evidence—not the one with the most sophisticated individual component.
↑ Back to Contents
8. Active Industry Practices
Established practices
- Fixed-size and overlapping chunking
- Recursive text splitting
- Metadata-preserving document ingestion
- Vector and hybrid retrieval
- Second-stage relevance reranking
- Retrieval evaluation using labeled query sets
Evolving practices
- Structure-aware and layout-aware document parsing
- Adaptive chunking based on document type
- Hierarchical and parent-child retrieval
- Contextual enrichment of indexed passages
- Query-aware context expansion
- Cost-aware candidate selection and reranking
Why technical documents require special attention
Technical documents often contain tables, diagrams, exceptions, warnings, and cross-references.
A text-only chunking strategy may separate a table value from its column heading or a safety instruction from the procedure it qualifies.
Production systems may therefore require specialized parsing, layout preservation, and document-specific retrieval methods.
Practice status: Established retrieval engineering techniques with evolving implementation approaches.
Reviewed: November 2026.
↑ Back to Contents
9. ACCOMPLISH — Practical Exercise
Exercise: Design a Better Chunking Strategy
Consider the following fictional technical bulletin:
Document: A100 Thermal Protection Bulletin
Section 1 — Applicability: This bulletin applies to A100 packaging machines manufactured within a specified production range.
Section 2 — Symptoms: Under certain operating conditions, the machine may stop after prolonged operation.
Section 3 — Possible Cause: Thermal protection may activate when monitored temperatures exceed defined limits.
Section 4 — Safety: Inspection must follow approved shutdown and isolation procedures.
Section 5 — Service Guidance: Qualified personnel should consult the applicable maintenance procedure and authorized service documentation.
Your tasks
- Divide the bulletin into three or four meaningful chunks.
- Explain why you chose those boundaries.
- Identify which metadata should accompany every chunk.
- Determine whether any sections should remain together.
- Design a retrieval query for the reported A100 shutdown.
- Propose criteria for reranking candidate passages.
- Select the evidence that should be supplied to a generative model.
- Explain how the system should preserve the safety instructions.
Optional AI-assisted prompt
Act as an AI retrieval engineer.
Analyze the technical document below.
1. Identify logical chunk boundaries.
2. Preserve important warnings and conditions.
3. Recommend metadata for each chunk.
4. Explain possible retrieval failure modes.
5. Suggest a reranking strategy.
6. Propose an evaluation test.
Do not invent technical facts.
Use only the supplied document.
Document:
[PASTE DOCUMENT]
Accomplishment: You have designed a chunking and reranking strategy that accounts for meaning, relevance, document applicability, and safety.
↑ Back to Contents
10. Student and Practitioner Project
Mini Project: RAG Retrieval Quality Lab
Objective: Experimentally compare chunking and reranking configurations.
Project requirements
- Collect 20–30 non-sensitive technical or organizational documents.
- Prepare at least 15 representative questions.
- Label relevant source passages for each question.
- Implement fixed-size chunking as a baseline.
- Implement a second chunking strategy, such as recursive or structure-aware chunking.
- Retrieve candidate passages using a consistent retrieval method.
- Compare results before and after reranking.
- Measure relevance, latency, and context size.
- Evaluate downstream answer quality on a small test set.
- Recommend a production configuration with supporting evidence.
Suggested experimental configurations
| Configuration | Chunking | Reranking |
|---|
| A | Fixed-size | No |
| B | Fixed-size | Yes |
| C | Structure-aware | No |
| D | Structure-aware | Yes |
Evaluation metrics
- Recall@k: Fraction of relevant passages found in the top k.
- Precision@k: Fraction of top-k passages judged relevant.
- MRR: Mean reciprocal rank of the first relevant result.
- nDCG@k: Ranking quality when relevance has multiple grades.
- Retrieval latency: Time required to retrieve and rerank results.
- Context tokens: Number of tokens passed to the generative model.
- Answer groundedness: Whether the generated response is supported by selected evidence.
Because chunking changes passage boundaries, use a consistent source-level relevance mapping when comparing configurations. Otherwise, passage-level metrics may not be directly comparable.
Deliverable: A prototype or reproducible experiment, comparison table, and one-page engineering recommendation.
↑ Back to Contents
11. Check Your Understanding
- Why does chunk size affect retrieval quality?
- How does overlapping chunking differ from fixed-size chunking without overlap?
- What is the purpose of a reranker?
- Why can reranking not compensate for missing candidates?
- How can structure-aware chunking improve technical document retrieval?
- What trade-offs arise when reranking more candidates?
- Why must safety warnings and document metadata be preserved?
- How would you determine whether reranking improves a RAG application?
Reflection: If a RAG assistant repeatedly retrieves the correct document but generates incomplete answers, would you first investigate chunking, reranking, context assembly, or generation? What evidence would guide your diagnosis?
↑ Back to Contents
12. Active Connections
These lessons show how AI engineers move from selecting information to representing, retrieving, prioritizing, and managing evidence.
↑ Back to Contents
13. References and Further Reading
- LangChain — Text Splitters
- Sentence Transformers — Cross-Encoder Applications
- Cohere — Reranking Overview
- Microsoft Learn — Retrieval-Augmented Generation Overview
- Weaviate — Reranking Search Results
Technical documentation and product capabilities change. Verify current APIs, model availability, and operational guidance before implementation.
Final Takeaway
Better RAG is not always about retrieving more information. It is about structuring knowledge so the right evidence can be found, and ranking that evidence so the most useful information reaches the model.
AI Engineering — Active Concepts & Practices
Active Concept #005 — Chunking and Reranking
AIYING — Accomplishing with AI
Understand → Connect → Apply → Accomplish
