The Engineer Who Designed for Failure
Why a deliberately broken AI workflow revealed a better way to build reliable systems.
An AI Engineering Case Study in Lateral Thinking, Reliability, and Human Judgment
By Ravindra Dastikop | November 2026
1. The Scenario: A System That Worked Until It Mattered
Maya was an AI engineer working on an internal procurement assistant.
The assistant answered questions about purchasing policies, summarized supplier information, and helped employees prepare purchase requests.
Its architecture looked reasonable:
- Receive the employee's question.
- Retrieve relevant policy documents.
- Ask a language model to interpret the information.
- Call an internal service when additional data was needed.
- Return a helpful answer.
In demonstrations, the assistant performed well.
It answered routine questions clearly. It produced convincing summaries. Its interface looked polished.
Then the team tested an unusual situation.
An employee asked whether an urgent purchase could proceed without normal approval.
The assistant retrieved an older policy excerpt, but the service containing current approval rules was temporarily unavailable.
The model still generated a confident answer.
The answer sounded plausible. It was also unsupported by the available evidence.
The team had discovered a serious design weakness: the system could continue sounding successful after the information needed for a reliable answer had become unavailable.
2. The Deadlock: More Testing or More Safeguards?
The team proposed several responses.
- Add more examples to the test set.
- Improve the retrieval configuration.
- Strengthen the system prompt.
- Add a warning that AI can make mistakes.
- Increase monitoring of the external service.
Each proposal had potential value.
But Maya noticed that the discussion assumed the assistant's job was to keep answering.
The team was asking:
They were not asking an equally important question:
That difference became the turning point.
3. The Lateral Thinking Move
Maya used a deliberate provocation inspired by Edward de Bono's Lateral Thinking approach.
At first, the statement sounded absurd.
Why would an engineer deliberately make a system fail?
But a PO provocation is not intended to be accepted literally. Its purpose is to interrupt an established pattern and allow movement toward alternative designs.
Movement 1 — Extract the principle
Perhaps failure should not be treated only as an unexpected event.
Perhaps it should be treated as an explicit condition that the architecture is designed to recognize.
Movement 2 — Focus on the difference
A conventional assistant tries to produce an answer.
A failure-aware assistant first determines whether it has sufficient evidence and functioning dependencies to produce the kind of answer requested.
When it cannot, the correct behavior may be to stop, qualify, ask for clarification, or escalate.
Movement 3 — Discover a practical possibility
The engineer arrived at a new design question:
Could we make the system's inability to answer safely a first-class, testable outcome?
This was more useful than merely asking the model to be careful.
4. The Engineering Decision: Design the Failure Contract
Maya proposed a failure contract.
For every critical dependency, the team would specify:
- What successful operation looks like.
- How failure is detected.
- Whether the failure can be safely retried.
- What information remains trustworthy.
- Which actions must be blocked.
- What the user should see.
- When human intervention is required.
The central idea was simple:
For the procurement assistant, the contract separated two types of response.
| Situation | Required system behavior |
|---|---|
| Current policy retrieved and approval service available | Provide a source-backed explanation within the system's authorized scope |
| Relevant document missing | State that the answer cannot be established from available evidence |
| Approval service unavailable | Do not invent approval status; provide an escalation route |
| Conflicting policy versions | Identify the conflict and request authoritative clarification |
| User requests an unauthorized approval action | Block the action and explain the required approval process |
The architecture was no longer designed around one happy path.
It was designed around legitimate outcomes, including safe non-completion.
5. Connect: What This Means in Modern AI Engineering
The case connects to several established engineering practices.
Failure isolation
Distributed systems use mechanisms such as timeouts, circuit breakers, and bulkheads to prevent one failing component from destabilizing others.
In AI systems, similar principles can help prevent tool failures from producing uncontrolled downstream behavior.
Grounding and evidence checks
Retrieval-augmented systems can use external information to support answers, but retrieval alone does not guarantee correctness.
The system must distinguish between evidence that supports a claim and material that merely appears relevant.
Human oversight
When a decision exceeds the system's authority or available evidence, escalation can be the intended behavior rather than a product defect.
Evaluation beyond answer quality
A useful evaluation set should include not only questions the assistant can answer, but also situations where it should abstain, ask for clarification, or refuse an unauthorized action.
These practices are consistent with broader reliability and AI risk-management principles. Their effectiveness still depends on implementation and testing.
6. The Unexpected Discovery
The team had initially treated an unanswered question as a failed user experience.
Maya's provocation exposed a more important distinction.
| Old interpretation | New interpretation |
|---|---|
| The assistant failed because it did not answer. | The assistant may succeed by refusing to invent an unsupported answer. |
| Every interruption is undesirable. | Some interruptions prevent consequential mistakes. |
| Reliability means continuous response. | Reliability includes correct behavior under degraded conditions. |
| A fallback must always provide an answer. | A fallback may provide a safe next step instead. |
This was not a discovery that failure is good.
It was a discovery that correctly handled failure is part of successful system behavior.
7. The Architecture: A Failure-Aware AI Workflow
Capture the user's request and its intended action.
Gather relevant evidence and dependency results.
Check availability, freshness, permissions, and evidence sufficiency.
Select answer, clarification, abstention, or escalation.
Communicate the result without disguising uncertainty.
Capture the outcome for evaluation and improvement.
Notice the architectural change.
The model is no longer the only component determining whether a response is acceptable.
Explicit checks and system-level controls participate in that decision.
For consequential actions, authorization and validation should be enforced by trusted application logic rather than relying solely on instructions to the language model.
8. Practical Application: Design a Failure Contract
Objective
Create a failure-aware design for a fictional AI assistant using a visual diagramming tool.
Tool: diagrams.net — https://app.diagrams.net/
Time: Approximately 20–30 minutes.
Prerequisites: A web browser. No coding or account is required for basic local diagram creation.
Step 1 — Open the tool
Visit diagrams.net and choose a local storage option when prompted.
Step 2 — Create the happy path
Draw five boxes:
User Request
|
v
Retrieve Policy
|
v
Check Approval Service
|
v
Generate Response
|
v
Show Answer
Step 3 — Introduce deliberate failures
Add three failure scenarios:
- The policy document cannot be retrieved.
- The approval service times out.
- Two retrieved policy documents contradict each other.
Step 4 — Add decision points
After retrieval, insert a decision:
Is the evidence sufficient and authoritative?
After the approval-service call, insert another:
Is the required service available and its result valid?
Step 5 — Design safe outcomes
For each negative branch, choose a behavior:
- Ask for clarification.
- State that evidence is insufficient.
- Provide an authorized escalation route.
- Retry only when safe and appropriate.
- Stop an unauthorized or unsupported action.
Step 6 — Add observability
Draw an outcome-recording component.
Capture the scenario category, dependency status, selected response path, and whether escalation occurred. Avoid logging sensitive content unnecessarily.
Step 7 — Review the diagram
Ask whether any failure path could still lead to a confident but unsupported answer.
If so, revise the design.
Expected output
A one-page Failure-Aware AI Assistant Blueprint with:
- One successful path.
- Three explicit failure scenarios.
- Two or more decision points.
- Safe response or escalation paths.
- An outcome-recording step.
The exercise produces an architectural proposal, not a verified production implementation.
9. Reusable AI Prompt: Failure-Contract Reviewer
Use this prompt with ChatGPT or another suitable AI assistant to critique your design.
Act as a senior AI reliability engineer. SYSTEM: A fictional procurement assistant that retrieves policy documents and checks approval rules through an internal service. MY DESIGN: [Describe or paste my workflow] Review the design using this sequence: 1. Identify the happy path. 2. List five realistic failure modes. 3. Identify where the system might produce an unsupported answer. 4. Specify failure-detection checks. 5. Recommend safe fallback or escalation behavior. 6. Distinguish model-level safeguards from application-enforced controls. 7. Design five evaluation scenarios, including at least two cases where the correct behavior is not to answer. 8. Propose observable pass/fail criteria. Use this PO provocation to challenge the design: "The system must fail before it can succeed." Apply movement to derive practical engineering improvements. Do not assume that generated recommendations are validated. Identify what requires testing.
10. Evaluate: How Would We Know It Works?
A failure-aware design must be tested against concrete expectations.
| Test | Expected behavior | Pass condition |
|---|---|---|
| Approval service unavailable | Do not claim approval is granted | No unsupported approval statement |
| No relevant policy evidence | State the evidence gap | No invented policy rule |
| Conflicting policy versions | Identify conflict and escalate | No unsupported resolution |
| Normal policy question | Answer using available authoritative evidence | Answer is supported by cited material |
| Unauthorized action request | Block the action | No unauthorized tool execution |
These are proposed test cases, not completed test results.
A real deployment would require more comprehensive testing, including adversarial cases, integration tests, access-control checks, and operational monitoring.
11. Today's Accomplishment
By completing the exercise, you will have converted an ordinary AI workflow into a design with explicit failure detection, controlled outcomes, and evaluation criteria.
The observable result is a diagram that another engineer could review, challenge, and potentially implement.
12. The Bigger Insight
The effective AI engineer does not merely make a system work when everything goes right.
The effective AI engineer defines what responsible behavior looks like when something goes wrong.
Reliability is not the absence of failure.
It is the ability to recognize, contain, communicate, and respond appropriately to failure.
The engineer who designed for failure was really designing for trust.
13. References and Further Reading
- Google SRE Book — Handling Overload
- Google SRE Workbook — Implementing SLOs
- Microsoft Azure Architecture Center — Circuit Breaker Pattern
- AWS Well-Architected — Reliability Pillar
- NIST — AI Risk Management Framework
- OpenAI — Evaluation Guidance
- Edward de Bono, Serious Creativity: Using the Power of Lateral Thinking to Create New Ideas (1992).
Editorial note: This is an independent educational case study. The fictional scenario and PO provocation are used to illustrate engineering judgment. References support the underlying reliability and evaluation concepts, not the fictional events.
.png)
