THE EFFECTIVE AI ENGINEER · CASE 006 OF 16
The Engineer Who Helped the Document Explain Itself
Applying Lateral Thinking to Create Meaningful Impact with AI
By Ravindra Dastikop
Engineering focus: Document preparation, retrieval, provenance and freshness.
Case type: Research-grounded fiction. Characters, events and trial observations are invented.
Sources checked: 5 November 2026.
1. The situation and impact sought
Ananya was building an internal knowledge assistant for a software company.
Support engineers used it to find explanations, operating instructions and product limitations across technical manuals. The assistant used retrieval-augmented generation, or RAG: it searched a document collection and supplied selected passages to a language model.
The documents contained the answers. Yet the assistant sometimes retrieved the wrong passage—or retrieved the right words without the information needed to interpret them.
A section said that a feature was unsupported. But the product name appeared several pages earlier. Another passage described a workaround whose version restriction appeared only in its heading.
The existing system divided documents into passages and searched their text using embeddings—numerical representations used to find semantic similarity.
Ananya’s task was to improve the usefulness of retrieved evidence within a two-week test. She could change document preparation and retrieval, but she could not rewrite the underlying manuals.
The useful outcome: Help support engineers find applicable, verifiable evidence with less searching and correction.
Retrieving something plausible would not be enough.
2. The story
The answer was in the manual.
Ananya knew because she had found it herself.
The assistant had been asked whether a particular export option was available in the self-hosted edition. It returned a confident answer supported by a passage about the hosted edition.
Both passages discussed the same feature. Both used similar language.
Only one applied.
“We could try a different embedding model,” said Vikram, the support lead.
They added that option to the experiment list. But Ananya first opened the retrieved passage without the rest of its document.
It began: “This option is available after the administrator enables the setting.”
Which option? Which edition? Which release?
The passage made sense to someone who had read the preceding pages. The search system encountered it alone.
Ananya copied the document title and section heading above the passage.
Now its meaning was clearer.
At first, she considered adding headings to every passage and stopping there. Then she paused.
What was the heading actually doing?
It was helping a fragment retain the identity and scope it had possessed inside the complete document.
That was a broader idea than “add a heading.”
She wrote:
Help each fragment carry the information needed to recognise where it belongs.
From that idea, several possibilities emerged.
One was straightforward: attach reliable metadata such as document identity, edition, version and section path.
Another was to search small passages but retrieve the surrounding section when necessary.
A third was to generate a short, passage-specific explanation using the complete source document.
An AI assistant helped Ananya expand these possibilities. Several suggestions were merely different names for the same mechanism. She removed the duplicates.
Then she asked it to draft a contextual explanation for a passage.
The explanation looked useful. It also described the limitation as applying to every edition.
The manual did not say that.
Ananya marked the generated explanation as incorrect.
Helping a document explain itself could easily become helping a model invent what the document meant.
She changed the design.
Original text would remain unchanged. Generated explanations would be stored separately. Each explanation would be linked to its source version and treated as a derived annotation requiring checks.
For the first experiment, the team compared the existing passages, passages with verified headings and metadata, and passages with additional generated context. They kept the initial retriever and retrieval depth comparable.
Some ambiguous passages became easier to find with simple metadata. Some seemed to benefit from more specific explanations. Others gained little.
A larger section helped with one exception but brought irrelevant material into another answer.
Then a support engineer noticed something else: the correct old passage was still being retrieved after the manual had been updated.
The context was well written.
It was also obsolete.
Ananya added an update test. A source change had to trigger reconsideration of its passages and annotations. A deleted source had to disappear from active search results according to the system’s deletion process.
A separate permission test exposed another issue. An explanation generated from an entire document contained information from a restricted section.
The team stopped that route until they could preserve the appropriate access boundary. Restricting the original passage alone would not protect information copied into an annotation.
By the end of the fictional trial, Ananya had no universal winner.
She proposed verified metadata as the first change for the selected document collection, with generated context reserved for a narrower experiment on passages that remained difficult to retrieve.
Hybrid search and reranking remained separate candidates for later comparisons.
Vikram looked at the revised plan.
“So the answer wasn’t simply a better search model?”
“That was one possibility,” Ananya said. “First we needed to understand what the searchable passage had lost.”
3. The thinking tool explained
The tool is concept extraction followed by generating alternatives.
In Serious Creativity, Edward de Bono describes pulling back from a specific idea to identify its underlying concept. The concept becomes a point from which other practical ideas can be developed.
The relevant discussion appears in “From Idea to Concept,” printed pages 139–141, with a further summary under “Concepts,” pages 315–316, in the supplied edition. [1]
In this case:
- Specific idea: Add the document title and heading to a passage.
- Extracted concept: Preserve the identity and scope a fragment needs when separated from its source.
- Alternative implementations: Metadata, surrounding-section retrieval, or a verified contextual annotation.
The concept must be broad enough to generate alternatives and specific enough to guide useful work.
“Improve information” is too broad.
“Always prepend the section heading” is already an implementation.
This case does not require PO. Concept extraction is a distinct Lateral Thinking practice. Calling every change of perspective a provocation would obscure the method being used.
The retrieval application is our contemporary adaptation of de Bono’s thinking tool.
4. The thinking made visible
| Thinking step | Application in this case |
|---|---|
| Focus | Find evidence that applies to the user’s actual question. |
| Initial idea | Add a title and heading to an isolated passage. |
| Pull back | Ask what useful function that addition performs. |
| Extracted concept | Preserve a fragment’s identity and scope outside its original surroundings. |
| Generate alternatives | Supply structured metadata, recover surrounding sections, or add verified explanatory context. |
| Judge | Compare retrieval quality, applicability, annotation errors and maintenance effort. |
| Act | Test a bounded change against the existing system. |
AI’s contribution
AI helps propose implementations and identify possible failure modes. It can also draft contextual annotations.
The engineer must check whether the alternatives differ in mechanism, whether annotations follow from the source, and whether access and update rules remain intact.
Reusable prompt
We are exploring this specific idea: [idea].
Explain the useful function it performs. Extract two or three possible underlying concepts at different levels of generality.
Choose a concept that is broad enough to generate alternatives but specific enough to guide this task: [task].
Generate three materially different implementations. For each, explain the mechanism, required information, likely failure, maintenance burden and smallest useful test.
Keep source facts separate from generated interpretation. Do not invent missing document details or assume that adding text improves retrieval.
5. The engineering decision
Ananya separates improvements to the searchable material from changes to the search method.
Changing both at once might improve the system, but it would make the reason harder to identify.
| Alternative | Potential benefit | Cost or difficulty | Critical uncertainty | Decision |
|---|---|---|---|---|
| Existing passage search | Provides a baseline. | May lose surrounding meaning. | Which failures come from missing context? | Retain for comparison. |
| Verified headings and metadata | Preserves explicit identity and scope. | Extraction and version mapping. | Are the required details present and reliable? | Test first. |
| Small passages with surrounding-section retrieval | Recovers nearby qualifications. | More text and retrieval logic. | Does the extra material resolve ambiguity? | Test separately. |
| Generated contextual annotations | Can explain a passage’s role. | Generation, verification and refresh. | Does the annotation introduce unsupported meaning? | Narrow experiment. |
| Hybrid search or reranking | Changes how candidates are found or ordered. | Additional configuration and evaluation. | Is the bottleneck retrieval method rather than representation? | Separate comparison. |
Keep evidence and annotation distinct
An original passage, authoritative metadata and a generated explanation have different origins. Preserve that distinction in storage, inspection and answer presentation.
A citation should lead to a source location supporting the claim. A generated description of a passage is not independent evidence.
Treat updates as part of the design
Record which source version produced each derived artifact. When relevant source content changes, refresh or invalidate dependent material.
Access changes need their own tests. A generated annotation must not become a route around restrictions on its source information.
These are design requirements for the fictional system, not guarantees supplied automatically by a retrieval product.
6. The evidence behind the case
What research supports
Anthropic’s Contextual Retrieval approach adds passage-specific context before indexing. Its published experiments also explore lexical retrieval and reranking. The reported gains belong to those configurations and datasets; they are not guarantees for another organisation’s manuals. [2]
Anthropic’s accompanying cookbook provides an implementation example and a retrieval evaluation using expected passages. This supports testing the retrieval stage directly, rather than judging it only through polished final answers. [3]
Dense Passage Retrieval demonstrates learned passage retrieval in open-domain question answering. It establishes an important retrieval approach, but its benchmark results do not settle which method best suits this case’s version-sensitive documents. [4]
Microsoft’s documentation describes document-level access controls, including security filtering and several approaches marked as preview. Its Azure Storage indexing guidance also distinguishes change detection from deletion detection: deletion requires an appropriate configured process. These illustrate why permissions and source lifecycle need explicit engineering attention. [5] [6]
What this case adds
The protagonist, failure sequence, comparison plan and proposed metadata-first decision are fictional teaching choices.
What remains uncertain
The appropriate passage boundaries, annotation method, retrieval strategy and refresh process depend on the collection and user questions. Better retrieval does not automatically establish correct answers or lower total operating cost.
No claim is made that the cited organisations use de Bono’s methods.
7. Try the move
Accessible exercise
Choose an approved document with headings, qualifications or version-specific instructions.
- Select a passage that becomes ambiguous when read alone.
- Add information that makes it interpretable.
- Ask what useful function your addition performs.
- Express that function as a
