AI Engineering Principle 003: Turn Real Failures into Better Tests

AI Engineering Principles & Practice · Post 003

Turn Real Failures into Better Tests

A failure is more than a bug to fix. It can become a test that helps catch the same problem next time.

In this post

• The principle
• A practical example
• Keep the test set balanced
• Independent study exercise
• Research basis

The principle

When an AI system gives an unsupported answer, misses a user’s intent, or uses a tool when it should not, save the example. Turn it into a repeatable test.

A test case should capture the situation that caused the failure, the behavior you expected, and how you will judge whether the system handled it correctly. After changing the prompt, model, tools, or workflow, run the test again. This helps you see whether the change fixed the problem and whether it affected other cases.

This creates a practical learning loop: observe a failure, understand what went wrong, add a test, improve the system, and check the result.

Back to top ↑

A practical example: a course handbook assistant

Imagine an assistant that answers questions using a course handbook. A learner asks about a deadline, but the assistant gives a date that does not appear in the handbook.

Turn that incident into a test case:

• Input: Ask the same question about the deadline.
• Expected behavior: The assistant should use a supported date or say that the handbook does not provide one.
• Evidence: Check whether the response is supported by the relevant handbook passage.
• Pass condition: The assistant does not invent a date.

The next time the system changes, rerun this case. If the assistant now answers correctly, the test guards against a known failure returning.

Back to top ↑

Keep the test set balanced

A test set made only from past failures can push the system too strongly toward avoiding those failures. Include a mix of examples:

• Common tasks: Questions the system should answer well.
• Edge cases: Ambiguous or unusually phrased requests.
• Should-not-do cases: Requests where the system should not guess, search, or take an action.
• Known failures: Real examples that previously produced a poor result.

For example, test both when a handbook assistant should search for a specific course rule and when it should answer directly from information already available. This checks that the system does not simply search every time—or skip searching when it is needed.

Anthropic suggests that 20–50 simple tasks drawn from real failures can be a useful starting point for early agent development. Treat this as a practical starting suggestion, not a fixed requirement: a mature system may need broader and more challenging coverage.

Back to top ↑

Independent study exercise

Choose an AI assistant or feature you use. Create four test cases:

1. A typical request the system should handle.
2. A difficult or ambiguous request.
3. A request that previously produced a poor response, if you have one.
4. A request where the system should ask for clarification, decline, or avoid guessing.

For each case, write down:

• The input you will use.
• The expected behavior.
• The evidence you will check.
• What would count as a pass or failure.

Keep the set and add new cases as you discover failures. Use examples you are authorized to work with, and remove personal or confidential details before using them in shared materials.

Engineering takeaway: Turn real failures into repeatable tests, and keep the test set balanced.

Back to top ↑

Research basis

• Anthropic, “Demystifying Evals for AI Agents” (January 9, 2026). The article recommends starting with a manageable set of tasks drawn from real failures, balancing cases where a behavior should and should not occur, and adding examples as coverage improves.
• OpenAI, “Evaluation Best Practices”. The guide recommends task-specific tests that reflect real-world distributions, using development logs to find evaluation cases, and growing the set over time.

This post is an independent study reference based on publicly available engineering material. It is not an official guide or endorsement from the cited organizations.

AIYING — Accomplishing with AI
AI Engineering Principles & Practice · 003