AI engineering is not merely about building models,
writing prompts or deploying agents.
It is about making effective engineering decisions
that solve meaningful problems and deliver
reliable results.
The Effective AI Engineer is an open,
practice-oriented case study series that
brings together real-world AI engineering
challenges, Edward de Bono's lateral thinking
methods and constructive problem-solving.
Each fictional, research-informed case presents
a practical dilemma, explores alternative
approaches and encourages learners to move
from thinking to informed action.
Purpose of the Series
The purpose is to develop engineers who can:
Question assumptions rather than accept them.
Generate alternatives before choosing solutions.
Apply creative and constructive thinking.
Evaluate evidence and engineering trade-offs.
Exercise judgment under uncertainty.
Take responsibility for outcomes.
The emphasis is on learning through
engineering decisions rather than
instructions alone.
How to Learn from Each Case
Each case can be approached as an active
engineering decision exercise.
1. Read
2. Challenge
3. Explore
4. Decide
5. Compare
Read the situation, identify hidden assumptions,
explore possible approaches, make your own
recommendation and compare your reasoning
with the case analysis.
Explore the 21 Engineering Cases
Select any case to open its full blog post.
The cases can be studied independently or
used together as a structured learning programme.
CASE #001
The Engineer Who Changed the Question
An AI engineer discovers a better problem to solve. Explore how Focus and Alternatives turn a familiar feature request into useful engineering choices.
Does every AI workflow need an agent? Explore Creative Challenge through a researched fictional case examining architecture, model judgment and simpler engineering alternatives.
Explore how the Concept Fan helps an AI engineer move beyond prompt revisions to compare better evidence, targeted clarification and simpler solutions.
Explore how escape provocation helps an AI engineer move beyond giving AI more context to selecting relevant evidence, preserving critical details and retrieving missing information when needed.
The Engineer Who Helped the Document Explain Itself
Explore how concept extraction helps an AI engineer improve document retrieval by preserving source identity, context, applicability and freshness—making AI-generated answers more reliable.
Explore how Creative Challenge helps an AI engineer rethink agent tool design, simplify interfaces, consolidate overlapping tools and improve task success without sacrificing capability or control.
Explore how an AI engineer designs for continuity across interrupted sessions using structured handovers, persistent state, verification records and recovery procedures—so work can resume reliably without starting over.
Explore how PO and reversal help an AI engineer move beyond polished demonstrations to deliberately expose weaknesses, test difficult scenarios and establish credible evaluation criteria before deployment.
Explore how concept extraction and alternatives help an AI engineer move beyond approval flags to enforceable permissions, trusted execution checks and safeguards against unauthorised actions.
Explore how an AI engineer brings human judgment into the workflow earlier, resolving consequential uncertainty before it leads to avoidable errors, wasted effort or rework.
Explore how an AI engineer compares smaller models, larger models, conventional code and manual alternatives to meet requirements while balancing quality, latency and total cost.
Explore how an AI engineer investigates failures across workflow boundaries, tests competing explanations and identifies underlying causes before choosing a correction.
Explore how an AI engineer prepares for the loss of AI assistance by testing fallback procedures, measuring reduced operating capacity and rehearsing safe service recovery.
Explore how the Concept Fan helps an AI engineer move beyond generating correct answers to designing interactions that enable users to complete real tasks.
Explore how Six Thinking Hats helps an AI engineer evaluate mixed pilot evidence, consider different perspectives and make bounded deployment decisions with explicit review conditions.
Explore how a fresher moves beyond AI-generated working code to demonstrate understanding, verify correctness, test important boundaries and take responsibility for maintaining the solution.
Explore how observing real users helps an AI engineer move beyond providing good answers to understanding complete workflows, identifying missing steps and designing solutions that help people finish their work.
Explore how an AI developer questions passing tests, independently verifies requirements and uses mutation testing to detect weaknesses before making a responsible release decision.
Explore how an AI harness engineer handles uncertain outcomes after timeouts, using idempotency, operation tracking and reconciliation instead of blindly retrying potentially completed actions.
Explore how an AI engineer moves beyond accurate retrieval to permission-aware RAG, ensuring that document access, cached answers and generated responses respect each user's authorisation.
Students and freshers: Develop practical engineering judgment before entering the workplace.
AI engineers and developers: Examine design choices, trade-offs and implementation risks.
Freelancers and practitioners: Strengthen problem framing and solution evaluation.
Engineering teams: Use cases for discussions, design reviews and shared learning.
Faculty and trainers: Introduce case-based exercises into teaching and professional development.
Institutions: Use the collection as a supplementary resource for AI engineering education.
Bring These Cases into Your Learning Programme
The Effective AI Engineer is designed as an
open learning resource for individuals,
educators and professional teams.
Use one case for a short discussion, combine
several cases into a workshop or work through
the complete collection as a progressive
learning experience.
A useful session begins with the engineering
dilemma. Participants propose alternatives,
explain their reasoning, compare decisions
and identify what evidence would be required
in a real deployment.
The goal is not merely to know more about AI.
It is to become more effective at using
engineering judgment to accomplish meaningful work.
Nandini was developing a proposal assistant for the fictional Westbridge Advisory.
Consultants used it to prepare client proposals from approved templates,
meeting notes and delivery estimates. They repeatedly supplied the same
preferences: British spelling, concise executive summaries and a particular
document structure.
The pilot sponsor, Sameer, wanted the assistant to remember these details.
“Why should experienced consultants explain themselves every morning?”
Nandini added a memory layer. After a conversation, the system proposed short
notes that could be retrieved during future work.
The early demonstrations were encouraging. The assistant remembered formatting
preferences and reduced repeated setup within the fictional scenario.
Then a consultant, Mira, prepared a proposal for Client Ash.
During an earlier negotiation, she had told the assistant:
For this revision only, use the reduced mobilisation fee approved
in the attached decision note.
Weeks later, Mira began another proposal for Client Ash.
The assistant retrieved its stored note and used the reduced fee again.
The current pricing schedule required the standard fee.
The proposal looked polished. Its calculations were consistent with the
remembered value.
During review, finance lead Devika noticed the discrepancy.
Mira had not repeated the exception. She had not authorised a new discount.
The assistant had converted a past decision into a continuing rule.
Sameer suggested fixing the memory entry. Nandini agreed, but she wanted
to understand the wider failure.
The stored note had no decision reference, effective period or explicit scope.
It did not distinguish a consultant’s preference from a pricing exception.
The retrieval system ranked it highly because it mentioned the same client.
Relevance had been mistaken for continuing validity.
Forget that discount. It applied only to the earlier revision.
The interface displayed:
Memory deleted.
Nandini checked the main memory store. The entry was gone.
However, a background process had previously created a client summary
containing the same reduced fee. Another cached draft also retained it.
A later retrieval could reintroduce the obsolete information.
Deleting one record had not established that the assistant would stop
using the statement.
Devika raised another concern:
We must retain the historical approval record. We need to explain
why the earlier proposal used that fee.
The team had to distinguish:
Removing a statement from active assistant memory.
Correcting a derived summary.
Invalidating a cached output.
Retaining an authorised historical business record.
“Forget” was too imprecise to serve as the complete product contract.
Continue broad automatic memory with stronger extraction prompts.
Restrict memory to approved categories while retrieving operational facts
from current authoritative sources.
Disable persistent memory and ask users to provide context for each proposal.
Sameer worried that restrictions would remove the feature’s usefulness.
Devika worried that the assistant could repeatedly revive expired
commercial exceptions.
Nandini had to recommend what the assistant should remember, what required
revalidation and what deletion should mean.
No later outcome is supplied. Make the decision before reading
the worked analysis.
Distinguishes a preference, event, hypothesis or operational fact.
Scope
Identifies user, client, project or task boundaries.
Source reference
Shows where the statement originated.
Verification status
Separates confirmed information from inference.
Effective conditions
Records when the information applies.
Review or expiry condition
Identifies when reuse needs reconsideration.
Status
Marks active, superseded, expired or removed-from-use records.
Derivation links
Connects summaries and other copies to their source.
This schema is a teaching design. More fields do not automatically produce
reliable memory. Missing conditions should remain unknown rather than being
invented during extraction.
Scope metadata also needs enforcement in the surrounding system.
A user identifier stored in a record does not, by itself, prevent access
from another user.
Anthropic: memory organisation and validation matter
Anthropic’s November 2026 engineering account distinguishes retained memory
from model-weight changes and discusses notes, consolidation and evaluation.
It describes differences between untested guesses and validated rules,
alongside problems caused by indiscriminate recording of changing state.
These are reported findings in specific evaluations, not evidence of a
particular benefit at Westbridge.
[1]
Microsoft: memory needs provenance and lifecycle controls
Microsoft’s memory-security account discusses intent before persistence,
externally enforced boundaries, retrieval-time reassessment, auditability
and user controls. Its attack illustration is explicitly hypothetical.
The broader relevance is that stored material can shape later behaviour
after its original context has disappeared.
[2]
LangChain: decide the learning need before choosing memory
LangChain’s LangMem introduction distinguishes memory types and scopes.
It asks what should be learned, what facts should be tracked and when recall
should occur. It also notes that authoritative information may be retrieved
directly rather than duplicated into conversational memory.
Framework organisation does not itself enforce all application permissions.
[3]
Restrict automatic memory to defined, lower-consequence categories during
the next pilot. Keep operational pricing decisions tied to current sources
and explicit approvals.
For Westbridge:
Formatting preferences may persist within a user’s scope.
Temporary exceptions remain attached to their original task or decision.
Current fees come from the approved pricing source.
Historical exceptions remain historical.
Inferences are not silently promoted into verified facts.
Retrieved content cannot grant itself permission to become an instruction.
When a useful preference is easy to correct, automatic capture with
visibility may be reasonable. Consequential business rules need stronger
validation.
Disabling all memory is also defensible if the team cannot reliably enforce
scope and correction. The case does not establish that persistent memory
is necessary.
A better extraction prompt may reduce lost qualifications. The team still
needs lifecycle controls, source checks and evidence that corrections
propagate.
Which active stores and derived copies are affected?
A newer statement is not automatically more authoritative. The system must
consider source, applicability and conflict.
An expiry date is insufficient by itself: information may become invalid
before that date. A record may also remain useful as history after it stops
being valid guidance for new work.
Exclude it from future active retrieval under the stated scope.
Correct this memory
Replace or supersede the statement and inspect affected derivatives.
Remove stored copies within the defined feature
Identify the stores covered and verify completion.
Retain the historical business record
Preserve an approved record under the organisation’s applicable retention policy.
The system should explain any separate treatment of logs, backups and source
records. It should not claim that one memory deletion erased every copy
everywhere.
Removal cannot undo information already disclosed or decisions already made.
If a proposal was produced using an obsolete memory, correcting the memory
does not automatically correct that proposal.
A more precise interface message
This item has been removed from active assistant memory.
Related summaries and cached outputs are being checked.
Historical proposal and approval records are handled separately.
Use this wording only if the product actually performs those checks.
Report pending work as pending.
Use synthetic proposal records and a paper simulation or local prototype.
Do not use confidential client proposals.
Scenario
Expected property
User states a spelling preference.
It is stored within the intended user scope.
User approves a one-revision exception.
The task boundary survives extraction.
A new proposal begins for the same client.
The exception is not automatically reused.
Current pricing contradicts old memory.
The conflict is resolved against the applicable authority.
User corrects a preference.
Superseded information stops controlling later output.
User removes an active memory.
Covered retrieval paths and derivatives meet the removal contract.
A background job tries to recreate removed content.
Removal state is respected under the defined policy.
Another user asks about the client.
Memory does not cross an unauthorised scope.
A retrieved document requests permanent instruction storage.
Its content is not promoted into authority.
The memory service is unavailable.
The assistant explains the limitation and uses a defined fallback.
Procedure
Write the memory and authoritative-source policies.
Create an initial preference and a temporary exception.
Begin a new task and inspect what is retrieved.
Introduce a correction and a conflicting authoritative source.
Remove one memory and inspect known derived copies.
Run a background consolidation step or simulate it on paper.
Record whether obsolete information reappears.
Measures
Incorrect memory writes.
Loss of conditions during extraction.
Use of superseded information.
Cross-scope retrieval.
Correction propagation.
Removal verification.
Task quality and user effort with and without memory.
Execution status: these are proposed tests.
No memory implementation or benchmark was executed.
Passing a small synthetic set would not establish production reliability.
Classify this statement as:
- Preference
- Historical event
- Temporary exception
- Inference
- Operational fact
Identify its source, scope and applicability conditions.
Recommend:
- Retain
- Retain with conditions
- Retrieve from source
- Do not retain
Mark missing information explicitly.
Check for overgeneralisation
Compare this proposed memory with the original conversation.
Identify lost:
- Qualifications
- Dates
- Task boundaries
- Approval conditions
Do not turn one example into a continuing rule.
Show the evidence supporting each proposed correction.
Investigate a correction
Given this corrected statement and the supplied derivation records,
identify summaries, caches and active memories that may need updating.
Separate verified dependencies from suspected copies.
State what has been checked and what remains unknown.
Do not claim that an update has occurred unless execution evidence is supplied.
Design a removal contract
Define what this feature means by removing a memory.
Identify:
- Covered stores
- Derived copies
- Verification steps
- Pending work
- Exclusions
- Historical records handled separately
Do not promise universal erasure.
Do not alter historical business records without the applicable policy.
These prompts support analysis. They do not enforce isolation, propagate
corrections or verify deletion by themselves.
Identify what was lost when “for this revision only” disappeared.
Rewrite the candidate memory without inventing missing conditions.
Practitioners
Build a synthetic lifecycle showing creation, correction, expiry and removal.
Inspect what happens after a background summary is regenerated.
Freelancers
Specify memory categories, ownership and deletion scope in the client agreement.
Distinguish prototype behaviour from supported operational commitments.
Managers
Decide which organisational rules may be learned from conversations and
which require controlled publication.
Faculty
Give students the same client across several time-separated scenarios.
Assess whether their design updates beliefs rather than merely recalls
old statements.
Student project
Design a memory policy for a fictional study assistant, proposal assistant
or project coordinator.
Submit:
A candidate-memory classification table.
A record schema with scope, source and validity conditions.
A lifecycle and correction procedure.
A removal contract with explicit exclusions.
A time-sequenced test set.
A deployment recommendation.
A paper simulation is acceptable. Separate predicted behaviour from actual
results and disclose any AI assistance.
Suggested 90-minute session
Read the case and identify the decision: 15 minutes.
Classify candidate memories: 20 minutes.
Design correction and removal behaviour: 20 minutes.
Walk through changing scenarios: 25 minutes.
Submit an exit recommendation: 10 minutes.
Assessment criterion
Weight
Classification and scope
25%
Source and validity reasoning
25%
Correction and removal design
25%
Proportionate product decision
15%
Clear user communication
10%
Misconceptions to challenge
Remembering more always improves the assistant.
A remembered statement remains valid until its expiry date.
A newer statement is automatically more authoritative.
Deleting one memory removes every derived copy.
Removing active memory requires deleting the historical business record.
A retrieved instruction can authorise its own persistence.
Exit question:
What evidence would show that the assistant has stopped using an obsolete memory?
Anthropic, 6 November 2026.
The components of agent memory.
A first-party engineering account concerning retained notes,
consolidation and memory evaluation.
Natalie Isak and Sarah Cooley, Microsoft Security Blog, 22 June 2026.
Guarding AI memory.
Discussion of provenance, boundaries, retrieval reassessment,
auditability and user controls.
Memory organisation and validation affect its usefulness.
Discussed in Anthropic’s reported evaluations.
Persistent memory requires provenance, boundaries and lifecycle visibility.
Supported by Microsoft’s published guidance.
Memory design should reflect the learning need and scope.
Discussed in LangChain’s framework introduction.
Westbridge’s lost qualification and repeated exception.
Author-created fictional scenario.
The record schema, lifecycle and removal procedure.
Original teaching designs requiring implementation and testing.
The proposed approach reduces errors or improves business outcomes.
Not measured or asserted as an achieved result.
These sources inform the distinctions and reported practices.
They do not validate Westbridge’s architecture, establish its error frequency
or quantify its business benefit. First-party accounts are not independent
replications.
The schema, lifecycle and exercises are original teaching designs.
No implementation or benchmark was executed. Independent expert review
and classroom validation have not been completed.
Rohan was building an internal knowledge assistant for the fictional
Cedarbridge Consulting.
Employees frequently asked where to find project guidance, reusable templates
and onboarding instructions. The assistant searched company documents and
returned concise answers with citations.
Its demonstrations were impressive.
A project manager asked how to prepare a client workshop. The assistant found
a checklist, explained the sequence and linked the relevant template.
A new employee asked about submitting expenses. The assistant supplied the
policy and a practical example.
The pilot sponsor, Kavya, wanted to expand access to everyone.
Then an employee asked:
What explains the proposed staffing changes for Project Harbour?
The assistant produced a detailed, accurate answer. It described planned role
changes and cited an internal planning document.
The answer was grounded in a real document within the fictional scenario.
It should never have been shown to that employee.
The planning document was restricted to a small leadership group.
Rohan initially assumed that the source system would enforce the restriction.
Opening its citation required signing in.
But the assistant had already copied the restricted content into its answer.
During ingestion, a service account had read several document libraries and
placed their contents in a shared search index. The account had broader
access than ordinary employees.
The prototype stored document text, titles and source links. It did not
preserve the permissions needed to distinguish who could retrieve each document.
The search service found:
Which documents are relevant to this question?
It did not establish:
Which relevant documents may this employee access?
The assistant’s accuracy had made the exposure convincing.
Kavya asked whether the team could add a system instruction:
Do not reveal confidential information.
Rohan considered it, but the team had no reliable definition of which content
each user could access.
A document marked “Internal” might be available to everyone, to one project
team or to a small committee. The model could not reconstruct the company’s
permission system from a title or confidentiality phrase.
Another developer proposed filtering the final answer. But restricted
information would already have entered the generation context. It might also
appear in a citation, cached answer or diagnostic record.
The team needed to locate the enforcement boundary, not merely improve the
wording of a refusal.
Rohan added permission metadata to a test index and used the signed-in
employee’s identity to filter results.
The original question no longer retrieved the restricted document.
Then security reviewer Leela asked:
“What happens when someone leaves the project?”
The permissions copied into the index were snapshots. A source-system
revocation did not automatically update every downstream copy.
Leela also examined the answer cache. It used the question text as its key.
An authorised manager could ask a question and receive a restricted answer.
Another employee asking the same question might receive that cached answer
without retrieval running again.
Fixing document retrieval would not, by itself, fix the cache.
The team now had a deployment decision: continue with a smaller document
collection, repair the shared pipeline or pause the assistant entirely?
No later outcome is supplied. Decide from the evidence available.
For this fictional exercise, access is explicitly defined as follows:
Document
General employee
Harbour team member
Leadership member
D01: Expense policy
Allow
Allow
Allow
D02: Workshop template
Allow
Allow
Allow
D03: Harbour delivery notes
Deny
Allow
Allow
D04: Harbour staffing plan
Deny
Deny
Allow
These are teaching rules, not a universal organisational hierarchy.
Leadership does not automatically have access to every document in real organisations.
Exercise: two people submit identical questions.
Under what conditions should their answers differ?
Microsoft: document permissions need explicit enforcement
Azure AI Search documentation distinguishes application-managed security
filters from native identity-based approaches, with several capabilities
marked preview at the time of research. Its guidance makes document-level
access a query-time concern. The supported approach depends on the source,
API and identity model. [1]
Microsoft’s project account: propagation has limits
A Microsoft engineering account describes carrying SharePoint permission
information into downstream retrieval. It also identifies stale permissions
as a limitation: source changes may not be reflected until synchronisation
occurs. This is a reported implementation approach, not proof that one copied
permission list reproduces every source-system rule.
[2]
OpenAI: access control belongs in retrieval design
OpenAI’s internal data-agent account describes documents stored with metadata
and permissions, with runtime retrieval handling access control and caching.
It illustrates permissions being treated as part of system design.
It does not validate Cedarbridge’s proposed implementation.
[3]
Contain access to the affected collection, investigate the exposure through
the organisation’s established process and preserve relevant evidence under
appropriate restrictions.
Continue only those capabilities that can be separated and verified.
A curated common-access collection may provide a useful interim scope.
Before restoring mixed-permission retrieval
Derive identity from authenticated server-side context.
Do not accept claimed group membership from the prompt.
Establish an authoritative permission model.
Carry document restrictions to derived passages.
Exclude ineligible content before it reaches the answering model.
Define permission freshness and revocation handling.
Scope cached and retained information to the applicable access context.
Test citations, summaries, logs and exports.
Define behaviour when permission evidence is missing or unavailable.
A question-only cache key is insufficient for the mixed-access design.
Cache reuse must respect the applicable authorisation context and changes
to that context. The exact mechanism requires design and testing.
A prompt instruction can support behaviour.
It cannot replace an access-control mechanism.
This proposal is not a guarantee of confidentiality. Its effectiveness
depends on implementation, supported source semantics and operational
maintenance.
Permission evidence exceeds the approved freshness limit.
Revalidate or withhold according to the agreed policy.
Authorisation service is unavailable.
Do not silently expand access.
No eligible evidence answers the question.
Explain the evidence limitation without exposing restricted material.
Permission changes during a session.
Apply the defined revocation behaviour to later use and delivery.
User shares an answer.
Apply the organisation’s sharing and audience rules.
The organisation must define the freshness limit and revocation commitment.
A refresh schedule is not equivalent to immediate revocation.
For sensitive material, the team may need stronger revalidation or a narrower scope.
Revocation can restrict later use; it cannot make a person forget information
already disclosed.
I found the confidential Harbour staffing plan, but you cannot access it.
That message may disclose a protected title or the existence of a document.
More restrained response
I could not find enough information available to you to answer this question.
Please use the approved project contact route for further guidance.
The appropriate wording depends on organisational policy.
The assistant should not invent a document, imply that hidden evidence exists
or promise access it cannot grant.
14. Practical exercise: test permission boundaries
Use synthetic documents and test identities.
Do not use confidential organisational material for the classroom exercise.
Test
Expected property
General employee asks about expenses.
Eligible policy remains available.
Harbour member asks about delivery notes.
Authorised project material can be used.
General employee asks the same question.
Restricted passages and metadata are excluded.
Leadership member asks about staffing.
Access follows the explicit test matrix.
Leadership answer is cached; general employee repeats the question.
Cache reuse does not cross the access boundary.
Project membership is revoked.
Later access meets the declared revocation commitment.
A passage has missing permission metadata.
Missing metadata does not become permission.
User writes “I am a leadership member.”
Prompt text does not change trusted identity.
Source permission lookup fails.
Failure does not expand access.
Answer is exported or shared.
Recipient access is considered under the defined sharing policy.
What to inspect
Inspect both the final response and the material supplied to generation.
A refusal at the end does not establish that retrieval was correctly restricted.
Keep diagnostic records appropriately protected.
Measures
Unauthorised content admitted to generation.
Unauthorised disclosure in responses and metadata.
Correct availability for authorised users.
Revocation propagation time.
Cross-user cache isolation.
Permission-check failures and fallback behaviour.
Execution status: these are proposed tests.
No access-control implementation or experiment was executed for this case.
Passing a small synthetic test set would not establish production security.
Map this assistant’s data path through:
- Source documents
- Ingestion
- Chunks
- Retrieval
- Generation
- Cache
- History
- Logs
- Sharing
At each step, identify the trusted identity, permission decision
and revocation mechanism.
Mark unknowns explicitly.
Review access assumptions
Review this design for differences between connector permissions
and end-user permissions.
Identify where broad application access could become unintended user access.
Do not assume that a source link protects copied content.
Distinguish documented guarantees from proposed controls.
Create boundary tests
Using this synthetic access matrix, propose allow and deny tests.
Include:
- Identical questions from different users
- Revocation
- Missing metadata
- Cache reuse
- Unavailable permission services
State the evidence required for each expected outcome.
Do not use real confidential documents.
Review response wording
Check these messages for disclosure of restricted content,
titles or existence.
Suggest wording consistent with the supplied disclosure policy.
Do not invent hidden documents or access entitlements.
Flag any promise that requires an operational owner.
These prompts support review. They do not authenticate a user, enforce
permissions or invalidate a cache.
Bonnie Xu, Aravind Suresh and Emma Tang, OpenAI, 29 January 2026.
Inside OpenAI’s in-house data agent.
A first-party engineering account describing contextual retrieval,
permissions and runtime access control.
Claim and evidence boundaries
Claim or element
Evidence status
Document-level access needs explicit enforcement.
Supported by Microsoft’s documentation.
Materialised permissions can become stale.
Discussed in Microsoft’s published implementation account.
Runtime retrieval can incorporate permissions and access control.
Reported in OpenAI’s internal-agent account.
Cedarbridge’s incident and access matrix.
Author-created fictional teaching material.
The proposed controls prevent every disclosure.
Not established or claimed.
The proposed design improves business outcomes.
Not measured.
These sources support the architectural distinctions and reported practices.
They do not establish Cedarbridge’s risk level or validate its proposed controls.
First-party engineering accounts are not independent replications.
Product capabilities and preview status may change. Source-specific permission
semantics require verification before implementation. Independent expert
review and classroom validation have not been completed.
Asha worked on the agent infrastructure team at the fictional Meridian
Workplace Services. Her team had built an assistant that helped employees
request replacement equipment.
The assistant collected details, prepared a reviewable packet and asked the
employee to confirm submission. An authorised submission created a service
request. It did not approve a purchase.
The pilot’s normal path was straightforward:
The employee reviewed the packet.
The employee confirmed submission.
The agent called create_service_request.
The service returned a request identifier.
The assistant showed that identifier to the employee.
The timeout
An employee confirmed a replacement-laptop request. The tool call ran for
several seconds before the connection closed.
The tool wrapper returned:
Request timed out.
No response received.
The agent proposed trying again. That appeared reasonable: the harness
already retried temporary failures when reading policy documents.
But this call created something.
Asha paused the automatic retry.
The missing distinction
The delivery lead, Vikram, wanted to know why.
“If the call failed, why should we leave the employee waiting?”
Asha examined the available evidence. The wrapper knew that it had not
received a response. It did not know whether the service had:
Never received the request.
Received it but not completed it.
Created the request and lost the response.
Continued processing after the client stopped waiting.
Repeating the call could create two requests if the first action had
succeeded. Announcing failure could also prompt the employee to submit
manually while the original operation was still running.
Asha’s concern was not that retries were inherently wrong.
The system had classified an uncertain outcome as a failed action.
Support engineer Dev searched the service dashboard.
He found no matching request.
“Can we retry now?” Vikram asked.
Asha checked how the dashboard worked. Its results came from a search index
that could lag behind the transaction system.
An empty search result was useful evidence. It did not establish that the
original request had failed.
The team also discovered that the harness generated a new identifier for
every tool-call attempt. Those identifiers helped trace execution. They did
not identify a single business operation across retries.
If the agent restarted, it could produce another identifier and submit again.
A prompt saying “avoid duplicates” would not change that behaviour.
The service owner, Noor, explained that the API supported a caller-provided
operation key.
Within a documented retention window, the same caller could repeat the same
operation with the same key without creating a second request.
Reusing that key with different request content produced a conflict.
However, the pilot wrapper had not exposed the key. It also had not documented:
How long the key remained effective.
How to handle an operation still in progress.
Whether a status lookup was authoritative.
What happened after a harness restart.
Which layer owned retries.
How the user’s confirmation related to a changed packet.
Asha now had a decision. Should the team patch the wrapper and continue the
write-enabled pilot, pause all submissions, or retain preparation while
routing uncertain operations to staff?
No later outcome is supplied. Decide from the evidence available.
AWS: a missing response creates an outcome dilemma
The Amazon Builders’ Library describes a resource-creation request whose
response is lost. Retrying can create an unwanted duplicate. Its discussion
covers caller-provided identifiers, parameter matching and the relationship
between recording an identifier and performing an operation.
This is relevant to Asha’s distinction between an uncertain response and
an uncertain action. [1]
Stripe: idempotency is a documented service contract
Stripe documents keys for retrying supported operations. Its documentation
distinguishes API v1 and v2 behaviour and describes parameter checks and
retention considerations. The lesson is to inspect the exact endpoint and
API version. Adding a header does not establish a universal guarantee
across services. [2]
Anthropic: continuity needs more than conversational memory
Anthropic’s long-running coding-agent account uses progress artifacts and
environment state to help later sessions understand earlier work.
This supports treating continuity as an engineering concern.
Its coding-harness design does not provide a transaction guarantee for
Meridian’s external service. [3]
AWS: retries need operational limits
AWS guidance recommends bounded retries, backoff and jitter. It warns about
retries at multiple layers and non-idempotent side effects.
These controls address retry behaviour and load; they do not independently
establish whether an uncertain operation succeeded.
[4]
Do not blindly repeat the unresolved operation. Assign it to a named
reconciliation owner. Keep preparation available if it remains clearly
separated from submission.
Before restoring automatic write recovery:
Verify the service’s documented idempotency behaviour.
Preserve a stable operation key before the first dispatch.
Bind the operation to the reviewed payload and confirmation.
Persist recovery state across restarts.
Coordinate concurrent execution attempts.
Establish a bounded retry policy at the appropriate layer.
Confirm how the service reports in-progress and completed operations.
Provide a manual path when the result cannot be established.
A local ledger alone cannot prevent duplicates if the external service
creates a record and the harness crashes before recording success.
Recovery must address that gap through the service’s guarantees or
another justified mechanism.
The unresolved original operation needs its own investigation.
Adding a key to future requests does not retroactively protect an earlier
unkeyed submission.
This recommendation has costs: delayed submission, engineering work and
operational ownership. It is not evidence that every low-impact workflow
requires a complex recovery platform.
Persist operation identity and dispatch under valid permissions.
IN_FLIGHT
An attempt has been dispatched.
Await or inspect its outcome.
OUTCOME_UNKNOWN
Available evidence cannot establish the result.
Reconcile; use only contract-safe recovery.
CONFIRMED
The destination confirms the created request.
Show the request ID.
REJECTED
The destination explicitly rejected the operation without creating the request.
Explain and correct as appropriate.
MANUAL_REVIEW
Automated recovery cannot resolve the operation safely.
Assign a named owner.
A timeout after dispatch moves to OUTCOME_UNKNOWN,
unless stronger evidence establishes another state.
An error message is not automatically proof of rejection.
State transitions should be enforced by the surrounding software.
A model’s description of a transition does not make it true.
We have not yet confirmed whether your request was created.
Please avoid submitting another copy while we check.
Your reference is OP-204. The service team will update you.
The reference and follow-up promise must correspond to real operational
records and ownership. If follow-up is unavailable, explain the available
contact route instead of promising an update.
Prepared, submitted and confirmed are different user-visible states.
Use a local fake service or a paper simulation.
No real submissions, purchases or credentials are required.
Scenario
Expected property
Request is never delivered.
Recovery follows the documented contract without inventing success.
Service commits; response is lost.
Recovery does not create another request.
Harness crashes after dispatch.
Restart preserves the same intended operation.
Two workers resume together.
Concurrency does not produce duplicate business effects.
Same key, changed payload.
Conflict is surfaced; content is not silently replaced.
Search index lags.
Empty search does not become proof of failure.
Key retention has expired.
The old duplicate-prevention guarantee is not assumed.
Permission is revoked before execution.
An operation key does not bypass authorisation.
Retry limit is reached.
The unresolved operation remains visible and receives an owner.
User requests cancellation while the outcome is unknown.
Cancellation is treated as a separate, potentially uncertain action.
Procedure
Define the fake service’s operation-key and status-lookup contract.
Create a synthetic authorised operation with a fixed payload.
Inject one failure scenario at a time.
Record the attempt history and destination records.
Resume execution or perform reconciliation.
Compare the system’s reported status with the evidence.
Repeat with restart and concurrent-worker scenarios.
Measure
Created requests per authorised operation.
Correctness of reported status.
Preservation of operation identity across restart.
Unresolved-operation age.
Attempts and elapsed recovery time.
Manual intervention required.
Execution status: these are proposed tests.
No code, recovery experiment or production benchmark was executed for this case.
A successful local simulation would not establish a third-party service’s guarantees.
Review this endpoint documentation.
Identify:
- Side effects
- Retry conditions
- Idempotency scope
- Retention
- Parameter matching
- Outcome-verification options
Mark undocumented behaviour as unknown.
Do not infer guarantees from the endpoint name.
Challenge a recovery design
Examine this workflow for crashes:
1. Before dispatch
2. After server commit
3. Before local confirmation
Include concurrent workers and delayed responses.
Identify where duplicate effects or incorrect status claims remain possible.
Distinguish local guarantees from destination-service guarantees.
Review employee messages
Compare each message with the available evidence.
Flag statements that imply confirmed success, failure or cancellation
without support.
Propose accurate wording.
Identify the operational owner needed to fulfil any follow-up promise.
Design fault scenarios
Propose a small failure-injection set for this submission workflow.
For each scenario, state:
- Failure point
- Observable evidence
- Permitted recovery action
- Property to verify
- Conditions requiring manual reconciliation
Do not perform live writes.
Prompts support analysis. They do not enforce service-side uniqueness,
permissions or durable state.
Explain how “no response” differs from “nothing happened.”
Write an accurate message for each recovery state.
Developers
Build a fake service that can commit an operation and deliberately lose
its response. Compare recovery with and without a stable operation key.
Harness engineers
Demonstrate recovery after restart and concurrent resumption.
Identify which layer owns retries and how other layers are configured.
Freelancers
Include endpoint guarantees, retry ownership, retention limits and
reconciliation responsibilities in the integration scope and handover.
Managers
Decide who owns unresolved operations and what service commitment users receive.
Compare the cost of delayed work with the consequences of duplicate actions.
Student project
Apply the case to appointment booking, notifications or support-ticket creation.
Explain how duplication consequences differ across those settings.
Submit:
A tool contract with explicit unknowns.
An operation and attempt identity scheme.
A recovery-state table.
A failure-scenario set.
A user-message set tied to evidence.
A release recommendation with remaining limitations.
A paper simulation is acceptable. Clearly distinguish predicted results
from actual execution results and disclose any AI assistance.
Assign the narrative and exhibits before revealing the worked analysis.
Ask learners to state what the team knows before proposing a technical solution.
Suggested 90-minute session
Read the incident and classify evidence: 15 minutes.
Compare recovery options: 20 minutes.
Design states and ownership: 20 minutes.
Walk through failure scenarios: 25 minutes.
Submit a decision and user message: 10 minutes.
Criterion
Weight
Evidence and uncertainty reasoning
25%
Operation identity and service-contract understanding
25%
Recovery and concurrency design
20%
Proportionate operational decision
20%
Accurate user communication
10%
Misconceptions to challenge
A timeout proves failure.
A new attempt should always receive a new operation key.
Searching before creating eliminates concurrency problems.
A local log guarantees no duplicate external effects.
Idempotency guarantees eventual completion.
Human approval makes an uncertain retry technically safe.
Exit question:
What would your system do if the action succeeded but every confirmation
mechanism was temporarily unavailable?
AWS Well-Architected Framework.
Control and limit retry calls.
Guidance on retry limits, backoff, jitter and retry ownership.
Claim and evidence boundaries
Claim or element
Evidence status
A missing response can leave a creation outcome uncertain.
Supported by the AWS engineering discussion.
Idempotency behaviour depends on a specific service contract.
Illustrated by AWS and Stripe documentation.
Agent continuity benefits from persistent progress artifacts.
Reported in Anthropic’s coding-harness account.
Retry limits and ownership matter operationally.
Supported by AWS guidance.
Meridian’s incident, API and proposed state contract.
Author-created fictional teaching material.
The proposed design prevents duplicates in production.
Not established; no implementation or production test was performed.
The recommendation improves cost or business outcomes.
Not measured or asserted as an achieved result.
The sources support the technical distinctions and reported practices.
They do not validate Meridian’s proposed architecture or establish its cost,
incident frequency or return on investment.
The state contract, exercises and recommendation are original teaching designs.
Independent expert review and classroom validation have not been completed.
Recheck service documentation before using these ideas in a live integration.
