AIYINGFrom AI Awareness to Accomplishment

Tuesday, November 17, 2026

Lesson #005 — Why AI Systems Fail Even When the Model Works

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #005

Why Can an AI System Fail Even When Its Model Works?



By Ravindra Dastikop

A correct model output is one part of a successful service

An AI system can fail because it supplies the wrong information to the model, mishandles the response, or fails to deliver a usable result. Model quality matters, but the surrounding application also determines the outcome.

“The model works” needs a precise meaning. It might mean that an API returned successfully, that an answer passed a particular check, or that the model performed well on a test set. None establishes that every real user journey succeeds.

Google’s production ML systems module presents the model as one component alongside data handling, serving infrastructure, configuration, and monitoring. We apply that system perspective to our document-help assistant.

Worked example: a correct answer that becomes an incorrect service

Our fictional training centre’s approved handbook states:

Participants must bring a laptop. Loan devices are not provided.

A participant asks: “Can you lend me a computer for the workshop?”

Imagine that the model receives this passage and returns an answer preserving both facts. But the application’s display code accidentally drops the second sentence. The user sees only the laptop requirement, without an explicit answer about borrowing.

The model output may satisfy our content check, while the displayed answer fails it. To diagnose this, compare what the model returned with what the interface rendered. Changing models would not directly repair this display defect.

This is a hypothetical failure, not an observed incident or measured result.

Follow the evidence across the system

A system boundary defines what our service is responsible for. A component boundary is a handoff within that service, such as passing source text to the model or passing its response to the interface.

For this case, inspect five stages. They are a teaching map, not a required software architecture.

Stage Possible failure Evidence to inspect
1. Receive the question The application sends a previous question instead of the current one. Compare submitted text with the request associated with this interaction.
2. Supply approved information A stale or incomplete handbook passage is supplied. Compare the approved version and passage with the actual model input.
3. Generate the answer The model invents a loan exception despite receiving the correct passage. Compare the raw response with its input and the requirements.
4. Deliver the result The interface omits the loan restriction or attaches an unrelated citation. Compare the response and source reference with the displayed result.
5. Handle failure A timeout leaves an endless spinner with no explanation or fallback. Inspect the error event, user-visible state, and recovery route.

These are candidate explanations. A symptom does not prove its cause. For example, an incorrect answer could originate in source selection, generation, or display. Preserve the evidence before selecting a remedy.

Three kinds of checks answer different questions

  • Component check: does a particular part behave as expected in a controlled situation?
  • Integration check: do connected parts exchange the right information and handle errors correctly?
  • End-to-end check: can a user complete the intended task through the actual interface?

A component check can help isolate a defect. An end-to-end check can expose a broken journey. Both are useful; passing one does not substitute for the other.

For our assistant, the end-to-end criterion includes preserving both laptop facts and showing supporting source text. It also includes respecting the existing boundary: the service explains policy; it does not grant exceptions or change enrolment records.

Create System Failure Map v0.1

  1. Choose one journey: the participant asks about borrowing a computer.
  2. Map the handoffs: question received, source supplied, answer generated, result displayed, and failure handled.
  3. List a plausible failure at each handoff. Include failures outside the model.
  4. Link requirements: use R-01 for preserving policy facts, R-02 for source support, and R-03 for missing-information handling. Carry forward the boundary against changing enrolment records.
  5. Name the evidence and owner: identify who will investigate each failure and which record would help.
  6. Propose a test and recovery: describe how to expose the failure safely and what the user should experience.
failure_id:
journey_and_stage:
linked_requirement:
possible_failure:
user_consequence:
evidence_needed:
proposed_test:
expected_user_visible_behaviour:
investigation_owner:
status: proposed / run / unresolved / resolved
actual_result:

Start with three proposed tests

  • Normal request: supply the approved passage. Check the displayed answer preserves both facts and its supporting text.
  • Missing source: deliberately omit the handbook in a controlled test. Check the service does not invent the centre’s policy and offers the agreed next step.
  • Unavailable response: simulate a model-service timeout in a test environment. Check the interface explains the failure and provides a usable fallback instead of claiming completion.

For a paper exercise, write the expected behaviour and mark each test not run. Once a prototype exists, record actual outputs, source and model identifiers, relevant settings, and the result. Agree timing targets with the service owner rather than inventing a universal timeout threshold.

Make failures diagnosable

Observability means having evidence that helps you understand what the running system did. Useful records for this case include an interaction identifier, source version, model identifier, stage status, elapsed time, and the displayed result or a suitable protected test record.

Collect only what is necessary under the project’s data-handling rules. Do not indiscriminately log personal information, credentials, or entire conversations.

Google’s productionization guidance emphasizes logging, monitoring, and alerts across pipelines, including latency and outages. Such signals help locate problems; a successful API status alone cannot establish answer correctness.

Verify your accomplishment

Ask another learner to use your map to investigate the missing-sentence example. Can they distinguish a source problem, a generation problem, and a display problem? Can they identify the evidence needed without guessing?

Practitioner extension: introduce one controlled defect in a non-production environment, such as dropping the source reference. Verify that your checks detect it. Restore the correct behaviour and retain the test result.

Case progress: System Brief → Baseline Record → Requirements Register → Model–Context Map → System Failure Map.

Next lesson: What Data Does Your System Need—and Can You Trust It?

Sources and scope: Google: Production ML systems; Google: Productionization. Checked 17 November 2026. The fictional case, failure map, and test proposals are AIYING teaching adaptations. No prototype or experiment was run for this lesson.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #005
See differently. Generate possibilities. Accomplish what matters.

No comments:

Post a Comment

Accomplish with AI

Powered by Blogger.

Featured post

How do I AI my first attempt? A Practical Guide to Getting Started

AIYING · HOW TO AI · EPISODE 001 How do I AI my first attempt? Turn an unfinished idea into one small version you can exa...

Search This Blog

List of blogs

Search This Blog

About

Translate

Instant Stories

Read in Your Language

Test

share this

ShareThis

Popular Posts

Continue exploring AIYING

Explore another case study, practical lab, or learning pathway.