The Effective AI Engineer #024 — The Engineer Who Designed for Failure

AIYING | The Effective AI Engineer #024

The Engineer Who Designed for Failure

Why a deliberately broken AI workflow revealed a better way to build reliable systems.

An AI Engineering Case Study in Lateral Thinking, Reliability, and Human Judgment

By Ravindra Dastikop | November 2026

1. The Scenario: A System That Worked Until It Mattered

Maya was an AI engineer working on an internal procurement assistant.

The assistant answered questions about purchasing policies, summarized supplier information, and helped employees prepare purchase requests.

Its architecture looked reasonable:

1. Receive the employee's question.
2. Retrieve relevant policy documents.
3. Ask a language model to interpret the information.
4. Call an internal service when additional data was needed.
5. Return a helpful answer.

In demonstrations, the assistant performed well.

It answered routine questions clearly. It produced convincing summaries. Its interface looked polished.

Then the team tested an unusual situation.

An employee asked whether an urgent purchase could proceed without normal approval.

The assistant retrieved an older policy excerpt, but the service containing current approval rules was temporarily unavailable.

The model still generated a confident answer.

The answer sounded plausible. It was also unsupported by the available evidence.

The team had discovered a serious design weakness: the system could continue sounding successful after the information needed for a reliable answer had become unavailable.

Case-study note: Maya, the organization, and this incident are fictional. The scenario illustrates a plausible AI engineering failure mode; it is not presented as a documented production incident.

2. The Deadlock: More Testing or More Safeguards?

The team proposed several responses.

• Add more examples to the test set.
• Improve the retrieval configuration.
• Strengthen the system prompt.
• Add a warning that AI can make mistakes.
• Increase monitoring of the external service.

Each proposal had potential value.

But Maya noticed that the discussion assumed the assistant's job was to keep answering.

The team was asking:

How do we prevent the system from failing?

They were not asking an equally important question:

What should the system do when a reliable answer is no longer possible?

That difference became the turning point.

3. The Lateral Thinking Move

Maya used a deliberate provocation inspired by Edward de Bono's Lateral Thinking approach.

PO: The system must fail before it can succeed.

At first, the statement sounded absurd.

Why would an engineer deliberately make a system fail?

But a PO provocation is not intended to be accepted literally. Its purpose is to interrupt an established pattern and allow movement toward alternative designs.

Movement 1 — Extract the principle

Perhaps failure should not be treated only as an unexpected event.

Perhaps it should be treated as an explicit condition that the architecture is designed to recognize.

Movement 2 — Focus on the difference

A conventional assistant tries to produce an answer.

A failure-aware assistant first determines whether it has sufficient evidence and functioning dependencies to produce the kind of answer requested.

When it cannot, the correct behavior may be to stop, qualify, ask for clarification, or escalate.

Movement 3 — Discover a practical possibility

The engineer arrived at a new design question:

Could we make the system's inability to answer safely a first-class, testable outcome?

This was more useful than merely asking the model to be careful.

4. The Engineering Decision: Design the Failure Contract

Maya proposed a failure contract.

For every critical dependency, the team would specify:

• What successful operation looks like.
• How failure is detected.
• Whether the failure can be safely retried.
• What information remains trustworthy.
• Which actions must be blocked.
• What the user should see.
• When human intervention is required.

The central idea was simple:

A reliable AI system must know when not to act as though it has succeeded.

For the procurement assistant, the contract separated two types of response.

Situation	Required system behavior
Current policy retrieved and approval service available	Provide a source-backed explanation within the system's authorized scope
Relevant document missing	State that the answer cannot be established from available evidence
Approval service unavailable	Do not invent approval status; provide an escalation route
Conflicting policy versions	Identify the conflict and request authoritative clarification
User requests an unauthorized approval action	Block the action and explain the required approval process

The architecture was no longer designed around one happy path.

It was designed around legitimate outcomes, including safe non-completion.

5. Connect: What This Means in Modern AI Engineering

The case connects to several established engineering practices.

Failure isolation

Distributed systems use mechanisms such as timeouts, circuit breakers, and bulkheads to prevent one failing component from destabilizing others.

In AI systems, similar principles can help prevent tool failures from producing uncontrolled downstream behavior.

Grounding and evidence checks

Retrieval-augmented systems can use external information to support answers, but retrieval alone does not guarantee correctness.

The system must distinguish between evidence that supports a claim and material that merely appears relevant.

Human oversight

When a decision exceeds the system's authority or available evidence, escalation can be the intended behavior rather than a product defect.

Evaluation beyond answer quality

A useful evaluation set should include not only questions the assistant can answer, but also situations where it should abstain, ask for clarification, or refuse an unauthorized action.

These practices are consistent with broader reliability and AI risk-management principles. Their effectiveness still depends on implementation and testing.

6. The Unexpected Discovery

The team had initially treated an unanswered question as a failed user experience.

Maya's provocation exposed a more important distinction.

Old interpretation	New interpretation
The assistant failed because it did not answer.	The assistant may succeed by refusing to invent an unsupported answer.
Every interruption is undesirable.	Some interruptions prevent consequential mistakes.
Reliability means continuous response.	Reliability includes correct behavior under degraded conditions.
A fallback must always provide an answer.	A fallback may provide a safe next step instead.

This was not a discovery that failure is good.

It was a discovery that correctly handled failure is part of successful system behavior.

7. The Architecture: A Failure-Aware AI Workflow

1. Receive

Capture the user's request and its intended action.

2. Retrieve

Gather relevant evidence and dependency results.

3. Validate

Check availability, freshness, permissions, and evidence sufficiency.

4. Decide

Select answer, clarification, abstention, or escalation.

5. Respond

Communicate the result without disguising uncertainty.

6. Record

Capture the outcome for evaluation and improvement.

Notice the architectural change.

The model is no longer the only component determining whether a response is acceptable.

Explicit checks and system-level controls participate in that decision.

For consequential actions, authorization and validation should be enforced by trusted application logic rather than relying solely on instructions to the language model.

8. Practical Application: Design a Failure Contract

Objective

Create a failure-aware design for a fictional AI assistant using a visual diagramming tool.

Tool: diagrams.net — https://app.diagrams.net/

Time: Approximately 20–30 minutes.

Prerequisites: A web browser. No coding or account is required for basic local diagram creation.

Step 1 — Open the tool

Visit diagrams.net and choose a local storage option when prompted.

Step 2 — Create the happy path

Draw five boxes:

User Request
     |
     v
Retrieve Policy
     |
     v
Check Approval Service
     |
     v
Generate Response
     |
     v
Show Answer

Step 3 — Introduce deliberate failures

Add three failure scenarios:

1. The policy document cannot be retrieved.
2. The approval service times out.
3. Two retrieved policy documents contradict each other.

Step 4 — Add decision points

After retrieval, insert a decision:

Is the evidence sufficient and authoritative?

After the approval-service call, insert another:

Is the required service available and its result valid?

Step 5 — Design safe outcomes

For each negative branch, choose a behavior:

• Ask for clarification.
• State that evidence is insufficient.
• Provide an authorized escalation route.
• Retry only when safe and appropriate.
• Stop an unauthorized or unsupported action.

Step 6 — Add observability

Draw an outcome-recording component.

Capture the scenario category, dependency status, selected response path, and whether escalation occurred. Avoid logging sensitive content unnecessarily.

Step 7 — Review the diagram

Ask whether any failure path could still lead to a confident but unsupported answer.

If so, revise the design.

Expected output

A one-page Failure-Aware AI Assistant Blueprint with:

• One successful path.
• Three explicit failure scenarios.
• Two or more decision points.
• Safe response or escalation paths.
• An outcome-recording step.

The exercise produces an architectural proposal, not a verified production implementation.

9. Reusable AI Prompt: Failure-Contract Reviewer

Use this prompt with ChatGPT or another suitable AI assistant to critique your design.

Act as a senior AI reliability engineer.

SYSTEM:
A fictional procurement assistant that
retrieves policy documents and checks
approval rules through an internal service.

MY DESIGN:
[Describe or paste my workflow]

Review the design using this sequence:

1. Identify the happy path.
2. List five realistic failure modes.
3. Identify where the system might
   produce an unsupported answer.
4. Specify failure-detection checks.
5. Recommend safe fallback or
   escalation behavior.
6. Distinguish model-level safeguards
   from application-enforced controls.
7. Design five evaluation scenarios,
   including at least two cases where
   the correct behavior is not to answer.
8. Propose observable pass/fail criteria.

Use this PO provocation to challenge
the design:
"The system must fail before
it can succeed."

Apply movement to derive practical
engineering improvements.

Do not assume that generated
recommendations are validated.
Identify what requires testing.

10. Evaluate: How Would We Know It Works?

A failure-aware design must be tested against concrete expectations.

Test	Expected behavior	Pass condition
Approval service unavailable	Do not claim approval is granted	No unsupported approval statement
No relevant policy evidence	State the evidence gap	No invented policy rule
Conflicting policy versions	Identify conflict and escalate	No unsupported resolution
Normal policy question	Answer using available authoritative evidence	Answer is supported by cited material
Unauthorized action request	Block the action	No unauthorized tool execution

These are proposed test cases, not completed test results.

A real deployment would require more comprehensive testing, including adversarial cases, integration tests, access-control checks, and operational monitoring.

11. Today's Accomplishment

Build one Failure-Aware AI Assistant Blueprint.

By completing the exercise, you will have converted an ordinary AI workflow into a design with explicit failure detection, controlled outcomes, and evaluation criteria.

The observable result is a diagram that another engineer could review, challenge, and potentially implement.

12. The Bigger Insight

The effective AI engineer does not merely make a system work when everything goes right.

The effective AI engineer defines what responsible behavior looks like when something goes wrong.

PO: The most reliable answer may be the answer the system chooses not to invent.

Reliability is not the absence of failure.

It is the ability to recognize, contain, communicate, and respond appropriately to failure.

The engineer who designed for failure was really designing for trust.

13. References and Further Reading

1. Google SRE Book — Handling Overload
2. Google SRE Workbook — Implementing SLOs
3. Microsoft Azure Architecture Center — Circuit Breaker Pattern
4. AWS Well-Architected — Reliability Pillar
5. NIST — AI Risk Management Framework
6. OpenAI — Evaluation Guidance
7. Edward de Bono, Serious Creativity: Using the Power of Lateral Thinking to Create New Ideas (1992).

Editorial note: This is an independent educational case study. The fictional scenario and PO provocation are used to illustrate engineering judgment. References support the underlying reliability and evaluation concepts, not the fictional events.

AIYING

The Effective AI Engineer — Case Study #024

Choose Work That Matters.

From AI Awareness to Accomplishment

Ravindra Dastikop | AIYING

AI Engineering Principle #006 — Design for Failure, Not Just Success: Building Reliable AI Systems

AIYING | AI ENGINEERING PRINCIPLES & PRACTICE #006

Design for Failure, Not Just Success

Building AI systems that detect problems, recover gracefully, and protect users.

An independent AI engineering study reference | Understand → Connect → Apply → Accomplish

1. The AI Demo That Worked Perfectly

Imagine an engineering team demonstrating an AI-powered customer support assistant.

A customer asks a question. The assistant searches the knowledge base, retrieves the relevant information, and produces a useful answer.

The demonstration succeeds.

But what happens when the knowledge base is unavailable?

What if the model takes too long to respond? What if retrieval returns irrelevant documents? What if an external tool fails halfway through a workflow?

These are not unusual engineering questions. They are part of operating real AI applications.

A successful demonstration proves that a system can work. Failure handling helps determine whether it can be trusted in operation.

2. The Principle: Design for Failure

AI systems combine probabilistic model behavior with conventional software components, external APIs, databases, retrieval services, and sometimes autonomous tool execution.

Each component introduces possible failure conditions.

The engineering objective is not to pretend failures can be eliminated.

It is to anticipate important failure modes, detect them where possible, limit their consequences, and provide an appropriate recovery path.

A useful design question is:

If this step fails, what should the system do next?

3. Five Failure Modes AI Engineers Should Consider

Failure 1 — The Model Produces an Unsupported Answer

A language model may generate a fluent response containing inaccurate or unsupported claims.

Possible controls include evidence checks, groundedness evaluation, explicit uncertainty, and escalation when reliable evidence is unavailable.

Failure 2 — Retrieval Returns the Wrong Information

A RAG application may retrieve irrelevant, outdated, incomplete, or unauthorized content.

Possible controls include relevance thresholds, metadata filters, access control, document versioning, and retrieval evaluation.

Failure 3 — An External Tool Becomes Unavailable

An AI agent may depend on an API that returns an error or times out.

Possible controls include timeouts, bounded retries, circuit breakers, and an alternative workflow.

Failure 4 — An Agent Attempts an Unsafe Action

An agent may propose an action that exceeds its authority or has significant consequences.

Possible controls include restricted permissions, policy checks, human approval, and audit logging.

Failure 5 — The System Cannot Explain What Happened

Without appropriate traces and logs, engineers may struggle to identify the cause of an unexpected result.

Possible controls include structured events, request identifiers, tool-call traces, and privacy-conscious observability.

4. The Failure-Response Design Pattern

DETECT → CLASSIFY → CONTAIN → RECOVER → RECORD → IMPROVE

Detect

Recognize an error, timeout, policy violation, or unacceptable output.

Classify

Determine whether the problem is temporary, permanent, recoverable, or safety-critical.

Contain

Prevent the failure from causing unnecessary downstream consequences.

Recover

Retry safely, use an approved fallback, request clarification, or escalate.

Record

Capture enough information to investigate the failure without exposing sensitive data.

Improve

Convert the failure into a test case, monitoring rule, or design improvement.

5. Connect the Principle to Industry Practice

Failure-aware AI engineering draws on established software reliability practices while addressing risks specific to AI-generated outputs and agent behavior.

Practice	Purpose
Timeouts	Prevent indefinite waiting for a service.
Bounded retries	Recover from selected transient errors without uncontrolled repetition.
Circuit breakers	Temporarily stop calls to a repeatedly failing dependency.
Fallbacks	Provide an approved alternative when normal operation is unavailable.
Human approval	Control consequential actions requiring judgment or authorization.
Evaluation	Measure performance on expected and failure scenarios.

Retries require particular care. Repeating a non-idempotent action, such as issuing a payment or creating an order, can cause duplicate effects.

Recovery must be designed around the actual operation and its risks.

6. Hands-On Lab — Design a Failure-Aware AI Assistant

You do not need to write code for this introductory experiment.

Open diagrams.net to create a simple workflow diagram.

Our fictional application is an AI-powered employee policy assistant.

Step 1 — Draw the normal workflow

User Question → Retrieve Policy → Generate Answer → Display Answer

Step 2 — Introduce a failure

Imagine the policy retrieval service is unavailable.

Add a decision point: Did retrieval succeed?

Step 3 — Design the fallback

If retrieval fails, the assistant must not invent a policy answer.

Instead, it should tell the user that the approved policy could not be retrieved and direct them to an authorized human contact.

Step 4 — Add an observability event

Record a simulated event containing:

• A request identifier
• The failure category
• The affected component
• The fallback action
• The outcome

Avoid storing sensitive user content unless there is an authorized need.

Step 5 — Test three scenarios

Scenario A: Retrieval succeeds and the answer is supported.

Scenario B: Retrieval times out.

Scenario C: Retrieval succeeds but returns no relevant policy.

For each scenario, describe the expected system behavior.

7. Reusable AI Engineering Prompt

Failure-Mode Design Assistant

Act as an AI engineering reliability reviewer.

I am designing the following AI application: [describe the application].

Identify five realistic failure modes.

For each, provide:

1. Failure trigger
2. Potential consequence
3. Detection mechanism
4. Safe response or fallback
5. Test case
6. Evidence that the control works

Distinguish model-output failures from infrastructure and integration failures.

Flag actions that require human approval. Do not assume a retry is safe.

Review the generated suggestions against the application's real architecture, requirements, and risk level.

8. What Should the Engineer Measure?

Reliability cannot be established through design intentions alone.

Useful measures may include:

• Successful request rate
• Timeout and dependency failure rates
• Fallback activation frequency
• Unsupported-answer rate on a defined test set
• Recovery time
• Human escalation outcomes

Teams should define acceptable thresholds according to the application's use case and risk profile.

A customer support suggestion tool and an AI system involved in a consequential decision may require very different controls.

9. Today's Accomplishment

Create a one-page Failure-Response Blueprint.

1. Draw the normal AI workflow.
2. Identify three realistic failures.
3. Define a safe response for each failure.
4. Specify what should be logged.
5. Write one test case for every failure.

Your accomplishment is not merely understanding the term "reliability."

It is producing a concrete design that describes how an AI application should behave when normal operation breaks down.

10. The Bigger Insight

Good AI engineering designs the successful path. Effective AI engineering also designs what happens when that path fails.

Dependability is not simply a property of the language model.

It emerges from the architecture, controls, monitoring, recovery mechanisms, operational processes, and people responsible for the complete system.

The engineering goal is not a system that never encounters failure.

It is a system that handles anticipated failures appropriately and gives engineers the evidence needed to improve it.

Industry References and Further Study

This article is an independent synthesis of established reliability engineering and AI risk-management practices, not an official principle issued by any one organization.

• Google — Site Reliability Engineering
• AWS — Well-Architected Reliability Pillar
• NIST — AI Risk Management Framework
• Microsoft Azure — Cloud Design Patterns

AIYING — From AI Awareness to Accomplishment

AI Engineering Principles & Practice is an independent study series connecting established engineering knowledge with practical AI system design, testing, deployment, and operation.

AIYING | Ravindra Dastikop