AIYINGFrom AI Awareness to Accomplishment
Showing posts with label A Practitioner's Approach. Show all posts
Showing posts with label A Practitioner's Approach. Show all posts

Tuesday, November 17, 2026

Lesson #005 — Why AI Systems Fail Even When the Model Works

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #005

Why Can an AI System Fail Even When Its Model Works?



By Ravindra Dastikop

A correct model output is one part of a successful service

An AI system can fail because it supplies the wrong information to the model, mishandles the response, or fails to deliver a usable result. Model quality matters, but the surrounding application also determines the outcome.

“The model works” needs a precise meaning. It might mean that an API returned successfully, that an answer passed a particular check, or that the model performed well on a test set. None establishes that every real user journey succeeds.

Google’s production ML systems module presents the model as one component alongside data handling, serving infrastructure, configuration, and monitoring. We apply that system perspective to our document-help assistant.

Worked example: a correct answer that becomes an incorrect service

Our fictional training centre’s approved handbook states:

Participants must bring a laptop. Loan devices are not provided.

A participant asks: “Can you lend me a computer for the workshop?”

Imagine that the model receives this passage and returns an answer preserving both facts. But the application’s display code accidentally drops the second sentence. The user sees only the laptop requirement, without an explicit answer about borrowing.

The model output may satisfy our content check, while the displayed answer fails it. To diagnose this, compare what the model returned with what the interface rendered. Changing models would not directly repair this display defect.

This is a hypothetical failure, not an observed incident or measured result.

Follow the evidence across the system

A system boundary defines what our service is responsible for. A component boundary is a handoff within that service, such as passing source text to the model or passing its response to the interface.

For this case, inspect five stages. They are a teaching map, not a required software architecture.

Stage Possible failure Evidence to inspect
1. Receive the question The application sends a previous question instead of the current one. Compare submitted text with the request associated with this interaction.
2. Supply approved information A stale or incomplete handbook passage is supplied. Compare the approved version and passage with the actual model input.
3. Generate the answer The model invents a loan exception despite receiving the correct passage. Compare the raw response with its input and the requirements.
4. Deliver the result The interface omits the loan restriction or attaches an unrelated citation. Compare the response and source reference with the displayed result.
5. Handle failure A timeout leaves an endless spinner with no explanation or fallback. Inspect the error event, user-visible state, and recovery route.

These are candidate explanations. A symptom does not prove its cause. For example, an incorrect answer could originate in source selection, generation, or display. Preserve the evidence before selecting a remedy.

Three kinds of checks answer different questions

  • Component check: does a particular part behave as expected in a controlled situation?
  • Integration check: do connected parts exchange the right information and handle errors correctly?
  • End-to-end check: can a user complete the intended task through the actual interface?

A component check can help isolate a defect. An end-to-end check can expose a broken journey. Both are useful; passing one does not substitute for the other.

For our assistant, the end-to-end criterion includes preserving both laptop facts and showing supporting source text. It also includes respecting the existing boundary: the service explains policy; it does not grant exceptions or change enrolment records.

Create System Failure Map v0.1

  1. Choose one journey: the participant asks about borrowing a computer.
  2. Map the handoffs: question received, source supplied, answer generated, result displayed, and failure handled.
  3. List a plausible failure at each handoff. Include failures outside the model.
  4. Link requirements: use R-01 for preserving policy facts, R-02 for source support, and R-03 for missing-information handling. Carry forward the boundary against changing enrolment records.
  5. Name the evidence and owner: identify who will investigate each failure and which record would help.
  6. Propose a test and recovery: describe how to expose the failure safely and what the user should experience.
failure_id:
journey_and_stage:
linked_requirement:
possible_failure:
user_consequence:
evidence_needed:
proposed_test:
expected_user_visible_behaviour:
investigation_owner:
status: proposed / run / unresolved / resolved
actual_result:

Start with three proposed tests

  • Normal request: supply the approved passage. Check the displayed answer preserves both facts and its supporting text.
  • Missing source: deliberately omit the handbook in a controlled test. Check the service does not invent the centre’s policy and offers the agreed next step.
  • Unavailable response: simulate a model-service timeout in a test environment. Check the interface explains the failure and provides a usable fallback instead of claiming completion.

For a paper exercise, write the expected behaviour and mark each test not run. Once a prototype exists, record actual outputs, source and model identifiers, relevant settings, and the result. Agree timing targets with the service owner rather than inventing a universal timeout threshold.

Make failures diagnosable

Observability means having evidence that helps you understand what the running system did. Useful records for this case include an interaction identifier, source version, model identifier, stage status, elapsed time, and the displayed result or a suitable protected test record.

Collect only what is necessary under the project’s data-handling rules. Do not indiscriminately log personal information, credentials, or entire conversations.

Google’s productionization guidance emphasizes logging, monitoring, and alerts across pipelines, including latency and outages. Such signals help locate problems; a successful API status alone cannot establish answer correctness.

Verify your accomplishment

Ask another learner to use your map to investigate the missing-sentence example. Can they distinguish a source problem, a generation problem, and a display problem? Can they identify the evidence needed without guessing?

Practitioner extension: introduce one controlled defect in a non-production environment, such as dropping the source reference. Verify that your checks detect it. Restore the correct behaviour and retain the test result.

Case progress: System Brief → Baseline Record → Requirements Register → Model–Context Map → System Failure Map.

Next lesson: What Data Does Your System Need—and Can You Trust It?

Sources and scope: Google: Production ML systems; Google: Productionization. Checked 17 November 2026. The fictional case, failure map, and test proposals are AIYING teaching adaptations. No prototype or experiment was run for this lesson.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #005
See differently. Generate possibilities. Accomplish what matters.

Lesson #004 — Training vs Inference: What Does an AI Model Learn?

AIYING · AI ENGINEERING: A PRACTITIONER’S APPROACH · LESSON #004

What Does a Model Learn—and What Happens During Inference?

By Ravindra Dastikop



Training changes learned parameters; inference uses them

In ordinary machine-learning training, an optimization process adjusts a model’s learned parameters using data and a learning objective. During ordinary inference, the trained model computes an output for an input without updating those learned parameters.

Parameters are internal numerical values, such as neural-network weights, that shape the model’s computations. A learning objective defines what the training process tries to improve. An objective can encourage useful behaviour without guaranteeing that every future output will be correct.

Google’s supervised-learning introduction illustrates this distinction using examples and predictions. Training methods vary; not all learning requires people to label every example.

For our document-help assistant, the practical implication is important: supplying a handbook passage in a request is not the same as retraining the model.

This lesson describes conventional fixed-parameter inference. Systems can separately implement online learning, saved memories, or later training pipelines; those mechanisms should be identified explicitly.

Worked example: the model and the handbook do different jobs

Our fictional training centre has a system brief, a baseline procedure, and candidate requirements. We have not yet chosen a production model or architecture.

Imagine testing an existing text-generation model with the approved source from earlier lessons:

Participants must bring a laptop. Loan devices are not provided.

The question is: “Can you lend me a computer for the workshop?”

The model’s prior training may help it interpret language and relate “computer” to “laptop.” The supplied passage provides the local policy. Our requirements determine what an acceptable response must preserve.

A candidate response is:

The supplied handbook says participants must bring a laptop and that loan devices are not provided.

This is an illustrative answer, not a measured model result.

Three mechanisms to distinguish
Mechanism What changes? Role in our case
Training or fine-tuning Learned parameters or trainable adapter parameters Can adapt model behaviour; not required merely to supply this passage.
Runtime context Information available in the current request Provides instructions, the question, and relevant policy text.
External storage Files, databases, or application memory Preserves approved document versions for later retrieval or loading.

A document in storage is not automatically available to the model. The application must retrieve, attach, or otherwise supply the information needed for the task.

A chat product may preserve history or saved memory and include some of it in later requests. That application behaviour does not by itself establish that the model’s weights changed.

How text generation fits

A token is a unit of text processing; it may be a word, part of a word, or punctuation. In a common decoder language model, generation proceeds by predicting a distribution over the next token using preceding context, then selecting a token and continuing. Hugging Face describes this as causal language modelling.

The output depends on learned parameters, the supplied context, and generation settings. Other AI tasks can produce a class label, numerical estimate, image, or other output rather than a sequence of text tokens.

Training can also lead models to memorize some data. Learned parameters are nevertheless not a version-controlled policy database, and plausible wording is not evidence of current policy.

Create Model–Context Map v0.1

  1. List the ingredients: model identifier, task instructions, user question, approved source version, and any external storage.
  2. Assign responsibility: mark which component supplies language capability, local facts, persistent records, and verification.
  3. Specify the update route: if the handbook changes, record who approves the revision and how the application supplies it. Do not silently overwrite the source used for earlier tests.
  4. Link checks: connect the map to R-01 correctness, R-02 source support, and R-03 missing-information handling from Lesson #003.
component_or_information:
location: model / runtime_context / external_storage
version_or_identifier:
how_it_reaches_the_model:
update_owner_and_procedure:
linked_requirement_ids:
evidence_or_test_status:

Optional observation: change context, not the model

In separate fresh conversations, use the same model and ask the computer-loan question with these three inputs:

  • A: The established no-loan passage.
  • B: A clearly labelled hypothetical alternative: “For this fictional exercise only, loan laptops are available by prior reservation.”
  • C: No handbook passage; instruct the assistant to say when the supplied information does not establish a policy.

In each request, instruct the model to answer only from the supplied source and identify supporting text. Record the actual outputs; compare them against the source and requirements.

B is a counterfactual test fixture, not a revision of our running case’s approved policy. Keep it separate. C tests handling of absent context; it does not erase the model’s prior knowledge.

If answers track A and B, you have observed context-sensitive behaviour—not proved permanent learning or perfect grounding. If they fail, preserve the failure for later evaluation lessons.

Verification: can another learner use your map to explain what needs updating when policy changes, where the source persists, and what the model actually receives?

Practitioner extension: inspect a real request at the application boundary, using sanitized data. Compare the source version you intended to supply with the content actually sent. Do not infer the request contents from the interface alone.

Case progress: System Brief → Baseline Record → Requirements Register → Model–Context Map. Preserve unresolved decisions and version identifiers.

Next lesson: Why Can an AI System Fail Even When Its Model Works?

Sources and scope: Google: Supervised Learning; Hugging Face: How Transformers solve tasks. Checked 17 November 2026. The running case, map, and context experiment are AIYING teaching adaptations. No model experiment was executed for this lesson.

AIYING · Ravindra Dastikop
AI Engineering: A Practitioner’s Approach · Lesson #004
See differently. Generate possibilities. Accomplish what matters.

Accomplish with AI

Powered by Blogger.

Featured post

How do I AI my first attempt? A Practical Guide to Getting Started

AIYING · HOW TO AI · EPISODE 001 How do I AI my first attempt? Turn an unfinished idea into one small version you can exa...

Search This Blog

List of blogs

Search This Blog

About

Translate

Instant Stories

Read in Your Language

Test

share this

ShareThis

Popular Posts