The Effective AI Engineer #024 — The Engineer Who Designed for Failure

AIYING | The Effective AI Engineer #024

The Engineer Who Designed for Failure

Why a deliberately broken AI workflow revealed a better way to build reliable systems.

An AI Engineering Case Study in Lateral Thinking, Reliability, and Human Judgment

By Ravindra Dastikop | November 2026

1. The Scenario: A System That Worked Until It Mattered

Maya was an AI engineer working on an internal procurement assistant.

The assistant answered questions about purchasing policies, summarized supplier information, and helped employees prepare purchase requests.

Its architecture looked reasonable:

1. Receive the employee's question.
2. Retrieve relevant policy documents.
3. Ask a language model to interpret the information.
4. Call an internal service when additional data was needed.
5. Return a helpful answer.

In demonstrations, the assistant performed well.

It answered routine questions clearly. It produced convincing summaries. Its interface looked polished.

Then the team tested an unusual situation.

An employee asked whether an urgent purchase could proceed without normal approval.

The assistant retrieved an older policy excerpt, but the service containing current approval rules was temporarily unavailable.

The model still generated a confident answer.

The answer sounded plausible. It was also unsupported by the available evidence.

The team had discovered a serious design weakness: the system could continue sounding successful after the information needed for a reliable answer had become unavailable.

Case-study note: Maya, the organization, and this incident are fictional. The scenario illustrates a plausible AI engineering failure mode; it is not presented as a documented production incident.

2. The Deadlock: More Testing or More Safeguards?

The team proposed several responses.

• Add more examples to the test set.
• Improve the retrieval configuration.
• Strengthen the system prompt.
• Add a warning that AI can make mistakes.
• Increase monitoring of the external service.

Each proposal had potential value.

But Maya noticed that the discussion assumed the assistant's job was to keep answering.

The team was asking:

How do we prevent the system from failing?

They were not asking an equally important question:

What should the system do when a reliable answer is no longer possible?

That difference became the turning point.

3. The Lateral Thinking Move

Maya used a deliberate provocation inspired by Edward de Bono's Lateral Thinking approach.

PO: The system must fail before it can succeed.

At first, the statement sounded absurd.

Why would an engineer deliberately make a system fail?

But a PO provocation is not intended to be accepted literally. Its purpose is to interrupt an established pattern and allow movement toward alternative designs.

Movement 1 — Extract the principle

Perhaps failure should not be treated only as an unexpected event.

Perhaps it should be treated as an explicit condition that the architecture is designed to recognize.

Movement 2 — Focus on the difference

A conventional assistant tries to produce an answer.

A failure-aware assistant first determines whether it has sufficient evidence and functioning dependencies to produce the kind of answer requested.

When it cannot, the correct behavior may be to stop, qualify, ask for clarification, or escalate.

Movement 3 — Discover a practical possibility

The engineer arrived at a new design question:

Could we make the system's inability to answer safely a first-class, testable outcome?

This was more useful than merely asking the model to be careful.

4. The Engineering Decision: Design the Failure Contract

Maya proposed a failure contract.

For every critical dependency, the team would specify:

• What successful operation looks like.
• How failure is detected.
• Whether the failure can be safely retried.
• What information remains trustworthy.
• Which actions must be blocked.
• What the user should see.
• When human intervention is required.

The central idea was simple:

A reliable AI system must know when not to act as though it has succeeded.

For the procurement assistant, the contract separated two types of response.

Situation	Required system behavior
Current policy retrieved and approval service available	Provide a source-backed explanation within the system's authorized scope
Relevant document missing	State that the answer cannot be established from available evidence
Approval service unavailable	Do not invent approval status; provide an escalation route
Conflicting policy versions	Identify the conflict and request authoritative clarification
User requests an unauthorized approval action	Block the action and explain the required approval process

The architecture was no longer designed around one happy path.

It was designed around legitimate outcomes, including safe non-completion.

5. Connect: What This Means in Modern AI Engineering

The case connects to several established engineering practices.

Failure isolation

Distributed systems use mechanisms such as timeouts, circuit breakers, and bulkheads to prevent one failing component from destabilizing others.

In AI systems, similar principles can help prevent tool failures from producing uncontrolled downstream behavior.

Grounding and evidence checks

Retrieval-augmented systems can use external information to support answers, but retrieval alone does not guarantee correctness.

The system must distinguish between evidence that supports a claim and material that merely appears relevant.

Human oversight

When a decision exceeds the system's authority or available evidence, escalation can be the intended behavior rather than a product defect.

Evaluation beyond answer quality

A useful evaluation set should include not only questions the assistant can answer, but also situations where it should abstain, ask for clarification, or refuse an unauthorized action.

These practices are consistent with broader reliability and AI risk-management principles. Their effectiveness still depends on implementation and testing.

6. The Unexpected Discovery

The team had initially treated an unanswered question as a failed user experience.

Maya's provocation exposed a more important distinction.

Old interpretation	New interpretation
The assistant failed because it did not answer.	The assistant may succeed by refusing to invent an unsupported answer.
Every interruption is undesirable.	Some interruptions prevent consequential mistakes.
Reliability means continuous response.	Reliability includes correct behavior under degraded conditions.
A fallback must always provide an answer.	A fallback may provide a safe next step instead.

This was not a discovery that failure is good.

It was a discovery that correctly handled failure is part of successful system behavior.

7. The Architecture: A Failure-Aware AI Workflow

1. Receive

Capture the user's request and its intended action.

2. Retrieve

Gather relevant evidence and dependency results.

3. Validate

Check availability, freshness, permissions, and evidence sufficiency.

4. Decide

Select answer, clarification, abstention, or escalation.

5. Respond

Communicate the result without disguising uncertainty.

6. Record

Capture the outcome for evaluation and improvement.

Notice the architectural change.

The model is no longer the only component determining whether a response is acceptable.

Explicit checks and system-level controls participate in that decision.

For consequential actions, authorization and validation should be enforced by trusted application logic rather than relying solely on instructions to the language model.

8. Practical Application: Design a Failure Contract

Objective

Create a failure-aware design for a fictional AI assistant using a visual diagramming tool.

Tool: diagrams.net — https://app.diagrams.net/

Time: Approximately 20–30 minutes.

Prerequisites: A web browser. No coding or account is required for basic local diagram creation.

Step 1 — Open the tool

Visit diagrams.net and choose a local storage option when prompted.

Step 2 — Create the happy path

Draw five boxes:

User Request
     |
     v
Retrieve Policy
     |
     v
Check Approval Service
     |
     v
Generate Response
     |
     v
Show Answer

Step 3 — Introduce deliberate failures

Add three failure scenarios:

1. The policy document cannot be retrieved.
2. The approval service times out.
3. Two retrieved policy documents contradict each other.

Step 4 — Add decision points

After retrieval, insert a decision:

Is the evidence sufficient and authoritative?

After the approval-service call, insert another:

Is the required service available and its result valid?

Step 5 — Design safe outcomes

For each negative branch, choose a behavior:

• Ask for clarification.
• State that evidence is insufficient.
• Provide an authorized escalation route.
• Retry only when safe and appropriate.
• Stop an unauthorized or unsupported action.

Step 6 — Add observability

Draw an outcome-recording component.

Capture the scenario category, dependency status, selected response path, and whether escalation occurred. Avoid logging sensitive content unnecessarily.

Step 7 — Review the diagram

Ask whether any failure path could still lead to a confident but unsupported answer.

If so, revise the design.

Expected output

A one-page Failure-Aware AI Assistant Blueprint with:

• One successful path.
• Three explicit failure scenarios.
• Two or more decision points.
• Safe response or escalation paths.
• An outcome-recording step.

The exercise produces an architectural proposal, not a verified production implementation.

9. Reusable AI Prompt: Failure-Contract Reviewer

Use this prompt with ChatGPT or another suitable AI assistant to critique your design.

Act as a senior AI reliability engineer.

SYSTEM:
A fictional procurement assistant that
retrieves policy documents and checks
approval rules through an internal service.

MY DESIGN:
[Describe or paste my workflow]

Review the design using this sequence:

1. Identify the happy path.
2. List five realistic failure modes.
3. Identify where the system might
   produce an unsupported answer.
4. Specify failure-detection checks.
5. Recommend safe fallback or
   escalation behavior.
6. Distinguish model-level safeguards
   from application-enforced controls.
7. Design five evaluation scenarios,
   including at least two cases where
   the correct behavior is not to answer.
8. Propose observable pass/fail criteria.

Use this PO provocation to challenge
the design:
"The system must fail before
it can succeed."

Apply movement to derive practical
engineering improvements.

Do not assume that generated
recommendations are validated.
Identify what requires testing.

10. Evaluate: How Would We Know It Works?

A failure-aware design must be tested against concrete expectations.

Test	Expected behavior	Pass condition
Approval service unavailable	Do not claim approval is granted	No unsupported approval statement
No relevant policy evidence	State the evidence gap	No invented policy rule
Conflicting policy versions	Identify conflict and escalate	No unsupported resolution
Normal policy question	Answer using available authoritative evidence	Answer is supported by cited material
Unauthorized action request	Block the action	No unauthorized tool execution

These are proposed test cases, not completed test results.

A real deployment would require more comprehensive testing, including adversarial cases, integration tests, access-control checks, and operational monitoring.

11. Today's Accomplishment

Build one Failure-Aware AI Assistant Blueprint.

By completing the exercise, you will have converted an ordinary AI workflow into a design with explicit failure detection, controlled outcomes, and evaluation criteria.

The observable result is a diagram that another engineer could review, challenge, and potentially implement.

12. The Bigger Insight

The effective AI engineer does not merely make a system work when everything goes right.

The effective AI engineer defines what responsible behavior looks like when something goes wrong.

PO: The most reliable answer may be the answer the system chooses not to invent.

Reliability is not the absence of failure.

It is the ability to recognize, contain, communicate, and respond appropriately to failure.

The engineer who designed for failure was really designing for trust.

13. References and Further Reading

1. Google SRE Book — Handling Overload
2. Google SRE Workbook — Implementing SLOs
3. Microsoft Azure Architecture Center — Circuit Breaker Pattern
4. AWS Well-Architected — Reliability Pillar
5. NIST — AI Risk Management Framework
6. OpenAI — Evaluation Guidance
7. Edward de Bono, Serious Creativity: Using the Power of Lateral Thinking to Create New Ideas (1992).

Editorial note: This is an independent educational case study. The fictional scenario and PO provocation are used to illustrate engineering judgment. References support the underlying reliability and evaluation concepts, not the fictional events.

AIYING

The Effective AI Engineer — Case Study #024

Choose Work That Matters.

From AI Awareness to Accomplishment

Ravindra Dastikop | AIYING