AI Engineering Principle 003: Turn Real Failures into Better Tests

AI Engineering Principles & Practice · Post 003

Turn Real Failures into Better Tests

A failure is more than a bug to fix. It can become a test that helps catch the same problem next time.

In this post

• The principle
• A practical example
• Keep the test set balanced
• Independent study exercise
• Research basis

The principle

When an AI system gives an unsupported answer, misses a user’s intent, or uses a tool when it should not, save the example. Turn it into a repeatable test.

A test case should capture the situation that caused the failure, the behavior you expected, and how you will judge whether the system handled it correctly. After changing the prompt, model, tools, or workflow, run the test again. This helps you see whether the change fixed the problem and whether it affected other cases.

This creates a practical learning loop: observe a failure, understand what went wrong, add a test, improve the system, and check the result.

Back to top ↑

A practical example: a course handbook assistant

Imagine an assistant that answers questions using a course handbook. A learner asks about a deadline, but the assistant gives a date that does not appear in the handbook.

Turn that incident into a test case:

• Input: Ask the same question about the deadline.
• Expected behavior: The assistant should use a supported date or say that the handbook does not provide one.
• Evidence: Check whether the response is supported by the relevant handbook passage.
• Pass condition: The assistant does not invent a date.

The next time the system changes, rerun this case. If the assistant now answers correctly, the test guards against a known failure returning.

Back to top ↑

Keep the test set balanced

A test set made only from past failures can push the system too strongly toward avoiding those failures. Include a mix of examples:

• Common tasks: Questions the system should answer well.
• Edge cases: Ambiguous or unusually phrased requests.
• Should-not-do cases: Requests where the system should not guess, search, or take an action.
• Known failures: Real examples that previously produced a poor result.

For example, test both when a handbook assistant should search for a specific course rule and when it should answer directly from information already available. This checks that the system does not simply search every time—or skip searching when it is needed.

Anthropic suggests that 20–50 simple tasks drawn from real failures can be a useful starting point for early agent development. Treat this as a practical starting suggestion, not a fixed requirement: a mature system may need broader and more challenging coverage.

Back to top ↑

Independent study exercise

Choose an AI assistant or feature you use. Create four test cases:

1. A typical request the system should handle.
2. A difficult or ambiguous request.
3. A request that previously produced a poor response, if you have one.
4. A request where the system should ask for clarification, decline, or avoid guessing.

For each case, write down:

• The input you will use.
• The expected behavior.
• The evidence you will check.
• What would count as a pass or failure.

Keep the set and add new cases as you discover failures. Use examples you are authorized to work with, and remove personal or confidential details before using them in shared materials.

Engineering takeaway: Turn real failures into repeatable tests, and keep the test set balanced.

Back to top ↑

Research basis

• Anthropic, “Demystifying Evals for AI Agents” (January 9, 2026). The article recommends starting with a manageable set of tasks drawn from real failures, balancing cases where a behavior should and should not occur, and adding examples as coverage improves.
• OpenAI, “Evaluation Best Practices”. The guide recommends task-specific tests that reflect real-world distributions, using development logs to find evaluation cases, and growing the set over time.

This post is an independent study reference based on publicly available engineering material. It is not an official guide or endorsement from the cited organizations.

AIYING — Accomplishing with AI
AI Engineering Principles & Practice · 003

The Effective AI Engineer #019: The Developer Who Tested the Tests

THE EFFECTIVE AI ENGINEER · A CASE-BASED APPROACH

Case #019: The Developer Who Tested the Tests

Choose Work That Matters

By Ravindra Dastikop
AIYING — Accomplishing with AI

The dashboard was green. Meera wanted to know what it had actually proved.

Case status and evidence boundary

This is an original fictional teaching case informed by published engineering material. Northstar Service Desk, its people, requirements, code, test counts and release situation are author-created. No fictional event or result is attributed to a real company.

Sources consulted: 8 November 2026. No software was executed for this post. The exercises are proposed learning activities, not reported experiments. Independent expert and classroom validation have not been completed.

Contents

1. The learning challenge
2. The case: twelve passing tests
3. The release decision
4. Exhibit A: the requirements contract
5. Exhibit B: misleading agreement
6. Exhibit C: acceptance examples
7. What published practice contributes
8. Your decision assignment
9. Worked analysis
10. Practical exercise: test the tests
11. Release evidence and stop conditions
12. Reusable AI prompts
13. Applications and student project
14. Teaching notes and assessment
15. Research record and references
16. Author and contact

1. The learning challenge

Case #017 explored understanding AI-generated code. Case #018 examined the work surrounding an AI answer. Case #019 asks a further question:

When AI helps write both an implementation and its tests, how do we establish that the tests represent the intended behaviour?

After completing this case, you should be able to:

• Translate a business requirement into independently reviewable examples.
• Distinguish implementation defects, test defects and ambiguous requirements.
• Identify tests that repeat the implementation’s assumptions.
• Use deliberate faults to examine whether tests detect meaningful errors.
• Prepare a release recommendation that states evidence and uncertainty.

Audience: students, freshers, freelancers, developers, AI engineers, technical leads and faculty.

Prerequisites: basic familiarity with functions and tests. The main decision exercise can be completed without coding.

↑ Back to contents

2. The case: twelve passing tests

A small change

Meera was a junior developer at the fictional Northstar Service Desk. Her team maintained a system that helped coordinators route equipment requests.

A change appeared straightforward: requests for equipment costing up to ₹50,000 could use the standard review route. More expensive requests required specialist review.

The change did not approve purchases. It selected the next review route.

Meera asked an AI coding assistant to implement the rule. It produced a short function, explanatory comments and twelve unit tests. In the fictional narrative, she inspected the code, ran the tests and saw twelve passes.

The project dashboard turned green.

Her delivery lead, Arun, wanted to include the change in the afternoon release. A coordinator had been manually correcting routing decisions, and the team had promised an improvement.

The boundary question

Before approving the pull request, senior engineer Farah asked: “Where did the expected results in these tests come from?”

“From the requirement,” Meera replied.

Farah pointed to the boundary test. It expected a request costing exactly ₹50,000 to go to specialist review.

Meera reopened the requirement:

Requests costing up to and including ₹50,000 use standard review.

The implementation used a strict comparison. The generated test expected the same strict comparison.

The test passed because the implementation and its expected result agreed. Both contradicted the stated boundary.

A second problem

Meera corrected the boundary and asked the assistant to update the tests. Farah stopped her before she accepted the revised suite.

“What happens when the amount is missing?”

The implementation treated a missing amount as zero. It was a convenient default. It also meant that an incomplete request could enter the standard review route.

The change request had not explicitly described missing amounts. However, an existing intake rule required an amount before routing. The coding prompt had included the new threshold sentence but omitted that existing rule.

These were different failures:

• The boundary error contradicted a supplied requirement.
• The missing-amount behaviour arose from incomplete context and an unsupported default.

Another round of generation might correct the visible defects while leaving the underlying process unchanged. Meera needed to establish what the tests should mean before asking for more tests.

The unresolved currency

The system served multiple business units. The change request used rupees, but the request record also contained a currency field.

Should another currency be converted, rejected or sent for clarification? No approved conversion service was available in this component.

Writing a test with an arbitrary expected result would turn an unanswered business question into an accidental product rule.

↑ Back to contents

3. The release decision

Arun was concerned about delay. “We have found two issues. Fix them and release. We cannot turn every small change into a research project.”

Farah agreed that review should remain proportionate. She wanted a small set of examples grounded in agreed requirements, including cases that a plausible but incorrect implementation would fail.

The team could:

1. Patch the identified defects and rely on the revised suite.
2. Clarify the contract and build a small independent acceptance set.
3. Delay the entire change for a broader redesign.
4. Retain manual routing while preparing a narrower release.

The release window was approaching. What should the team do?

↑ Back to contents

4. Exhibit A: the requirements contract

This author-created contract supports the exercise. Meera must obtain process-owner confirmation of the newly clarified rules before treating them as release requirements.

Requirement	Behaviour to confirm
Scope	Select a review route; never approve or submit a purchase.
Standard route	A valid INR amount from zero through ₹50,000, inclusive.
Specialist route	A valid INR amount greater than ₹50,000.
Missing amount	Return NEEDS_INFORMATION.
Negative amount	Return INVALID_AMOUNT.
Unsupported currency	Return NEEDS_INFORMATION; do not invent a conversion.
Monetary representation	Use integer paise in this exercise.
Invalid data type	Return INVALID_AMOUNT; Boolean values are invalid.

These rules are local teaching choices, not universal procurement policy. Currency and input-validation behaviour require an accountable owner. A test cannot resolve ambiguity merely by assigning an expected output.

↑ Back to contents

5. Exhibit B: misleading agreement

Deliberately incorrect illustrative pseudocode. This first version uses rupees; the acceptance exercise later uses integer paise.

route(amount):
    if amount is missing:
        amount = 0

    if amount < 50000:
        return STANDARD_REVIEW

    return SPECIALIST_REVIEW

A test that agrees with the defect

expect route(50000) == SPECIALIST_REVIEW

This assertion would pass against the illustrated implementation. It does not enforce the inclusive boundary in the requirement.

A human could make the same mistake. The issue is that the expected result was accepted without checking it against the agreed rule.

Asking another model to review the code can provide a useful challenge. It does not establish independent correctness by itself.

↑ Back to contents

6. Exhibit C: acceptance examples

For this exercise, ₹50,000 equals 5,000,000 paise. The table uses integer paise throughout.

ID	Input or condition	Expected outcome
T01	INR, 0 paise	STANDARD_REVIEW
T02	INR, 4,999,999 paise	STANDARD_REVIEW
T03	INR, 5,000,000 paise	STANDARD_REVIEW
T04	INR, 5,000,001 paise	SPECIALIST_REVIEW
T05	INR, missing amount	NEEDS_INFORMATION
T06	INR, −1 paise	INVALID_AMOUNT
T07	USD, otherwise valid amount	NEEDS_INFORMATION
T08	INR, text instead of integer paise	INVALID_AMOUNT
T09	INR, Boolean value	INVALID_AMOUNT
T10	Any valid routing result	No approval or submission side effect

This is a focused teaching set, not exhaustive production coverage. It intentionally isolates one input condition at a time. A production contract would also need to define behaviour when several inputs are invalid.

The process owner reviews expected business outcomes. The technical reviewer examines whether executable tests actually enforce them.

↑ Back to contents

7. What published practice contributes

GitHub: inspect intent as well as execution

GitHub’s guidance combines automated checks with requirement alignment, dependency inspection and collaborative review. It also highlights tests that have been deleted or skipped. This supports examining a change and its checks together. It does not establish that a checklist guarantees correctness. [1]

Anthropic: scrutinise evaluation machinery

Anthropic discusses checking task specifications, graders and execution records, alongside maintaining evaluation suites. Its account of Claude Code describes evaluations being added as the product developed. The relevant question is whether a result reflects the system’s behaviour or the evaluation’s design. Agent evaluations and unit tests differ, but both depend on meaningful success criteria. [2]

OpenAI: recognised benchmarks also have limits

OpenAI’s February 2026 discussion of SWE-bench Verified reports problems involving flawed tests and training contamination, and explains why it stopped using the benchmark for frontier coding capability reporting. A recognised evaluation therefore still needs scrutiny. This does not establish that Northstar’s local suite has the same problems. [3]

The author’s application: Examine the expected behaviour, the implementation and the mechanism that judges it. This is a synthesis for teaching, not a quoted company framework.

↑ Back to contents

8. Your decision assignment

Before reading the worked analysis, prepare a release recommendation of 500–800 words.

• Identify established defects and unresolved requirements.
• Propose the smallest useful acceptance set.
• Name the business-rule owner and technical reviewer.
• Recommend release, a narrower release or continued manual routing.
• Specify evidence that would change your recommendation.
• Describe a rollback or disablement plan.

Address this counterargument: If the change is small and reversible, when does additional testing become less useful than releasing it with appropriate monitoring?

Explain the consequences of an incorrect route. Do not treat every defect as equally serious.

Pause here and record your decision.

↑ Back to contents

9. Worked analysis

Open the author’s conditional recommendation

Hold the affected routing change briefly while the process owner confirms unresolved rules. Retain the existing routing process during that review. Then implement the narrow change against an independently reviewed acceptance set.

This is a conditional recommendation. The case supplies no measured delay cost, production incident rate or evidence that a broader redesign is necessary.

A focused sequence

1. Confirm the contract. Record the inclusive threshold, required amount, supported currency and invalid-input behaviour.
2. Establish expected examples. Derive outcomes from the approved contract before inspecting the revised implementation.
3. Correct the implementation. Keep unrelated refactoring out of the change unless necessary.
4. Review the tests. Inspect assertions, skipped cases, fixtures and the code paths exercised.
5. Run relevant regression and integration checks. Confirm that routing has not become approval and that the displayed state matches the returned result.
6. Prepare a bounded release. Name the owner, monitoring signals and disablement procedure.

A different recommendation can be defensible if the team demonstrates that the impact is sufficiently narrow and the responsible owner accepts the remaining uncertainty.

↑ Back to contents

10. Practical exercise: test the tests

Introduce a controlled implementation change that violates a requirement. Examine whether the tests detect its effect. This technique is commonly called mutation testing.

In this exercise, use a local teaching implementation. Do not introduce deliberate faults into a live service.

Deliberate fault	Check expected to detect it
Use a strict boundary instead of an inclusive boundary.	T03
Treat a missing amount as zero.	T05
Accept negative amounts.	T06
Ignore the currency field.	T07
Coerce text into a valid amount without permission.	T08
Accept Boolean values as integers.	T09
Add an approval action during routing.	T10, using an appropriate side-effect check

Procedure

1. Preserve a reference implementation reviewed against the contract.
2. Confirm that the reference passes the agreed checks.
3. Introduce one deliberate fault at a time.
4. Run the same tests and retain the output.
5. Record the assertion that detects the fault.
6. Restore the reference before trying the next fault.
7. Investigate faults that survive.

A surviving mutation requires interpretation. It may expose a missing check, or it may make no observable difference under the agreed contract. Not every mutation represents a meaningful defect.

Record your evidence

Fault ID	Requirement violated	Test result	Interpretation
M01	Inclusive boundary	Record after execution	Explain the result
M02	Missing amount	Record after execution	Explain the result
M03	Currency handling	Record after execution	Explain the result

Execution status: these are proposed exercises. No passing run, mutation score or production result is claimed in this post.

↑ Back to contents

11. Release evidence and stop conditions

A useful release note should identify:

• The approved requirement and its owner.
• The exact code revision and test environment.
• Checks executed, results and unresolved failures.
• Relevant deliberate faults detected or missed.
• Remaining limitations and excluded behaviours.
• The release owner, monitoring plan and rollback procedure.

Retain the existing routing process if the contract remains unresolved, the boundary defect persists, required checks are skipped, or routing can trigger an unauthorised approval or submission.

More passing tests do not automatically justify release. Equally, a proportionate review need not attempt to prove every possible property. The decision should connect the evidence to the consequences of failure.

↑ Back to contents

12. Reusable AI prompts

Requirement clarification

Read this change request and existing behaviour documentation.

Separate:
1. Explicit requirements
2. Unresolved questions
3. Implementation choices

For each unresolved question, identify the owner who should decide it.
Do not invent defaults.

Acceptance-example design

Using only this approved contract, propose boundary, missing-input,
invalid-input and regression examples.

Show the requirement supporting each expected outcome.
Flag any expectation that the contract does not establish.
Keep monetary units explicit and consistent.

Test review

Inspect this test suite against the approved acceptance examples.

Identify:
- Assertions that repeat implementation logic
- Omitted requirements
- Skipped checks
- Fixtures that may hide defects
- Expected results unsupported by the contract

Give concise reasons tied to the contract.

Fault challenge

Propose a small set of deliberate implementation faults that would
violate distinct requirements.

For each fault:
- Name the requirement violated
- Identify the test expected to fail
- Explain the observable difference

Do not modify production code.

AI can propose examples and challenges. The team still verifies whether those checks represent the intended behaviour.

↑ Back to contents

13. Applications and student project

Freshers and students

Explain how a passing assertion can encode an incorrect expectation. Produce the acceptance table without writing code.

Practising developers

Implement the approved contract and demonstrate whether selected meaningful faults are detected. Examine both valid behaviour and invalid-input handling.

Freelancers

Include requirement confirmation, acceptance examples, execution evidence and known limitations in the client handover.

AI and harness engineers

Review evaluation definitions, environment configuration and result reporting. Distinguish a failed task from a defective evaluator or execution environment.

Student project: build an evidence-backed release recommendation

Apply the method to a fictional booking rule, discount threshold, eligibility check or document-routing function.

1. Write a short requirement containing a boundary and one ambiguity.
2. Ask another group to identify the ambiguity before implementation.
3. Obtain an agreed contract.
4. Create acceptance examples and a small implementation.
5. Introduce at least three meaningful faults.
6. Record actual execution results, if coding is used.
7. Submit a release recommendation with remaining uncertainty.

Deliverables: contract, acceptance table, implementation or paper walkthrough, fault results and a one-page decision memo. Disclose any AI contribution and the checks performed.

↑ Back to contents

14. Teaching notes and assessment

Assign the narrative and exhibits before revealing the worked analysis. Different groups can own requirements, implementation and tests. Discuss disagreements before agreeing on expected outcomes.

Suggested 75-minute session

1. Read and identify the decision: 10 minutes.
2. Separate defects from ambiguities: 15 minutes.
3. Design acceptance examples: 20 minutes.
4. Challenge the tests with deliberate faults: 20 minutes.
5. Submit a release recommendation: 10 minutes.

Criterion	Weight
Requirement interpretation and ambiguity handling	25%
Acceptance examples and boundary reasoning	25%
Examination of test effectiveness	20%
Proportionate release judgment	20%
Clear evidence and limitation reporting	10%

Reward well-supported disagreement. A larger test count is not inherently a better submission.

Common misconceptions to discuss

• A green dashboard proves that the business requirement is satisfied.
• Using a second model automatically makes a review independent.
• Every failed test indicates an implementation defect.
• Every surviving mutation indicates a meaningful testing gap.
• More testing is always preferable, regardless of consequence or cost.

Exit question: What evidence would make you trust the test suite more—and what would still remain unproved?

↑ Back to contents

15. Research record and references

1. GitHub Docs. Review AI-generated code. Official guidance on functional checks, intent, code quality, dependencies and collaborative review.
2. Anthropic. Demystifying evals for AI agents. 9 January 2026. Engineering guidance and reported experience concerning task design, graders, execution records and evaluation maintenance.
3. OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities. 23 February 2026. A first-party analysis of benchmark limitations.

Claim and evidence boundaries

Claim or element	Evidence status
Review should examine requirements and code, alongside automated checks.	Supported by GitHub’s published guidance.
Task specifications and evaluation machinery require scrutiny.	Supported by Anthropic’s guidance and reported experience.
A recognised benchmark can develop or retain important limitations.	Illustrated by OpenAI’s published analysis.
Northstar’s implementation and tests share a boundary error.	Author-created fictional scenario.
The proposed review process reduces incidents or saves time.	Not established; no such benefit was measured.
The practical exercise detects the listed faults.	Proposed expectation; learners must execute and record results.

All sources were consulted on 8 November 2026. First-party engineering accounts are not independent replications. The sources do not establish how frequently AI-generated tests share implementation errors or quantify the benefit of the proposed Northstar process.

The narrative, contract and exercises are original teaching material. No benchmark was reproduced, no code was executed for this post and no independent expert or classroom review has been completed.

Revisit the case when source findings change, learners identify defects, or the material is prepared for formal book publication.

↑ Back to contents

16. Author and contact

Ravindra Dastikop
AIYING — Accomplishing with AI
The Effective AI Engineer: A Case-Based Approach

Email: ravindra.dastikop@gmail.com
Phone: +91 84532 12131

Choose Work That Matters

A passing test is evidence about an assertion. Check whether that assertion represents the work you intended.

← Read Case #018: The Builder Who Watched the User Work

↑ Back to contents