AIYING | AI ENGINEERING PRINCIPLES & PRACTICE #006
Design for Failure, Not Just Success
Building AI systems that detect problems, recover gracefully, and protect users.
An independent AI engineering study reference | Understand → Connect → Apply → Accomplish
1. The AI Demo That Worked Perfectly
Imagine an engineering team demonstrating an AI-powered customer support assistant.
A customer asks a question. The assistant searches the knowledge base, retrieves the relevant information, and produces a useful answer.
The demonstration succeeds.
But what happens when the knowledge base is unavailable?
What if the model takes too long to respond? What if retrieval returns irrelevant documents? What if an external tool fails halfway through a workflow?
These are not unusual engineering questions. They are part of operating real AI applications.
A successful demonstration proves that a system can work. Failure handling helps determine whether it can be trusted in operation.
2. The Principle: Design for Failure
AI systems combine probabilistic model behavior with conventional software components, external APIs, databases, retrieval services, and sometimes autonomous tool execution.
Each component introduces possible failure conditions.
The engineering objective is not to pretend failures can be eliminated.
It is to anticipate important failure modes, detect them where possible, limit their consequences, and provide an appropriate recovery path.
A useful design question is:
If this step fails, what should the system do next?
3. Five Failure Modes AI Engineers Should Consider
Failure 1 — The Model Produces an Unsupported Answer
A language model may generate a fluent response containing inaccurate or unsupported claims.
Possible controls include evidence checks, groundedness evaluation, explicit uncertainty, and escalation when reliable evidence is unavailable.
Failure 2 — Retrieval Returns the Wrong Information
A RAG application may retrieve irrelevant, outdated, incomplete, or unauthorized content.
Possible controls include relevance thresholds, metadata filters, access control, document versioning, and retrieval evaluation.
Failure 3 — An External Tool Becomes Unavailable
An AI agent may depend on an API that returns an error or times out.
Possible controls include timeouts, bounded retries, circuit breakers, and an alternative workflow.
Failure 4 — An Agent Attempts an Unsafe Action
An agent may propose an action that exceeds its authority or has significant consequences.
Possible controls include restricted permissions, policy checks, human approval, and audit logging.
Failure 5 — The System Cannot Explain What Happened
Without appropriate traces and logs, engineers may struggle to identify the cause of an unexpected result.
Possible controls include structured events, request identifiers, tool-call traces, and privacy-conscious observability.
4. The Failure-Response Design Pattern
DETECT → CLASSIFY → CONTAIN → RECOVER → RECORD → IMPROVE
Detect
Recognize an error, timeout, policy violation, or unacceptable output.
Classify
Determine whether the problem is temporary, permanent, recoverable, or safety-critical.
Contain
Prevent the failure from causing unnecessary downstream consequences.
Recover
Retry safely, use an approved fallback, request clarification, or escalate.
Record
Capture enough information to investigate the failure without exposing sensitive data.
Improve
Convert the failure into a test case, monitoring rule, or design improvement.
5. Connect the Principle to Industry Practice
Failure-aware AI engineering draws on established software reliability practices while addressing risks specific to AI-generated outputs and agent behavior.
| Practice | Purpose |
|---|---|
| Timeouts | Prevent indefinite waiting for a service. |
| Bounded retries | Recover from selected transient errors without uncontrolled repetition. |
| Circuit breakers | Temporarily stop calls to a repeatedly failing dependency. |
| Fallbacks | Provide an approved alternative when normal operation is unavailable. |
| Human approval | Control consequential actions requiring judgment or authorization. |
| Evaluation | Measure performance on expected and failure scenarios. |
Retries require particular care. Repeating a non-idempotent action, such as issuing a payment or creating an order, can cause duplicate effects.
Recovery must be designed around the actual operation and its risks.
6. Hands-On Lab — Design a Failure-Aware AI Assistant
You do not need to write code for this introductory experiment.
Open diagrams.net to create a simple workflow diagram.
Our fictional application is an AI-powered employee policy assistant.
Step 1 — Draw the normal workflow
Step 2 — Introduce a failure
Imagine the policy retrieval service is unavailable.
Add a decision point: Did retrieval succeed?
Step 3 — Design the fallback
If retrieval fails, the assistant must not invent a policy answer.
Instead, it should tell the user that the approved policy could not be retrieved and direct them to an authorized human contact.
Step 4 — Add an observability event
Record a simulated event containing:
- A request identifier
- The failure category
- The affected component
- The fallback action
- The outcome
Avoid storing sensitive user content unless there is an authorized need.
Step 5 — Test three scenarios
Scenario A: Retrieval succeeds and the answer is supported.
Scenario B: Retrieval times out.
Scenario C: Retrieval succeeds but returns no relevant policy.
For each scenario, describe the expected system behavior.
7. Reusable AI Engineering Prompt
Failure-Mode Design Assistant
Act as an AI engineering reliability reviewer.
I am designing the following AI application: [describe the application].
Identify five realistic failure modes.
For each, provide:
- Failure trigger
- Potential consequence
- Detection mechanism
- Safe response or fallback
- Test case
- Evidence that the control works
Distinguish model-output failures from infrastructure and integration failures.
Flag actions that require human approval. Do not assume a retry is safe.
Review the generated suggestions against the application's real architecture, requirements, and risk level.
8. What Should the Engineer Measure?
Reliability cannot be established through design intentions alone.
Useful measures may include:
- Successful request rate
- Timeout and dependency failure rates
- Fallback activation frequency
- Unsupported-answer rate on a defined test set
- Recovery time
- Human escalation outcomes
Teams should define acceptable thresholds according to the application's use case and risk profile.
A customer support suggestion tool and an AI system involved in a consequential decision may require very different controls.
9. Today's Accomplishment
Create a one-page Failure-Response Blueprint.
- Draw the normal AI workflow.
- Identify three realistic failures.
- Define a safe response for each failure.
- Specify what should be logged.
- Write one test case for every failure.
Your accomplishment is not merely understanding the term "reliability."
It is producing a concrete design that describes how an AI application should behave when normal operation breaks down.
10. The Bigger Insight
Good AI engineering designs the successful path. Effective AI engineering also designs what happens when that path fails.
Dependability is not simply a property of the language model.
It emerges from the architecture, controls, monitoring, recovery mechanisms, operational processes, and people responsible for the complete system.
The engineering goal is not a system that never encounters failure.
It is a system that handles anticipated failures appropriately and gives engineers the evidence needed to improve it.
Industry References and Further Study
This article is an independent synthesis of established reliability engineering and AI risk-management practices, not an official principle issued by any one organization.
