AI Engineering Principle 004: Make the Tool Obvious

AI Engineering Principles & Practice · Post 004

Make the Tool Obvious

Clear purpose. Clear inputs. Useful results.

In this post

• The principle
• A practical example
• Designing a useful tool
• Independent study exercise
• Research basis

The principle

An AI system can only use a tool through the interface it receives. If the tool name is vague, the inputs are unclear, or the result is hard to interpret, the agent may choose the wrong tool or use it incorrectly.

A tool is part of the AI system’s design. Its description should make clear what it does, when it should be used, what information it needs, and what it returns. The system should also handle invalid inputs and errors in a way that makes the next step understandable.

For example, a tool named search gives little guidance. A tool described as “search the official course handbook for course rules and deadlines” gives a clearer signal about its purpose.

Back to top ↑

A practical example: searching a course handbook

Suppose you are building an assistant that answers questions from a course handbook. It has access to a search tool. A useful tool definition could clarify:

• Purpose: Search the official handbook for course information.
• When to use it: Use it when an answer depends on handbook content or a course-specific rule.
• Inputs: A clear search query, and the course identifier if there are multiple handbooks.
• Results: Relevant passages with section names or page references.
• No-result behavior: Return a clear “no relevant passage found” result so the assistant knows not to invent an answer.

This helps the agent distinguish a grounded answer from a guess and gives it information it can use in its response.

Back to top ↑

Designing a useful tool

When you design a tool for an AI system, consider these questions:

1. Does the name make the tool’s purpose clear?
2. Does the description explain when the tool should—and should not—be used?
3. Are input names, formats, and required fields understandable?
4. Does the tool return information in a predictable, useful format?
5. Do errors explain what went wrong and how the system can recover?

Design the interface, then test it on realistic tasks. A test set can include ordinary requests, ambiguous requests, missing inputs, and cases where the tool should not be called. Review the results and revise the interface when the agent repeatedly misunderstands it.

Engineering takeaway: Design a tool so its purpose, inputs, and results are clear to the agent.

Back to top ↑

Independent study exercise

Design a tool that lets an AI assistant search a course handbook. Write a short specification with:

• The tool’s name and purpose.
• When the assistant should use it.
• The input fields and their formats.
• The result fields the tool should return.
• What the tool should return when it finds no relevant information or receives invalid input.

Then try three realistic questions. Check whether the assistant chooses the tool appropriately, supplies usable inputs, and explains the result accurately. Add any recurring failure to your evaluation set.

Back to top ↑

Research basis

• Anthropic, “Writing Effective Tools for Agents — With Agents” (September 11, 2025). The article discusses clearly defined tools, tool descriptions, realistic evaluation tasks, and iterative improvement.
• OpenAI, “Function Calling”. The guide recommends clear function names and parameter descriptions, predictable and intuitive functions, and structured inputs that help prevent invalid calls.

This post is an independent study reference based on publicly available engineering material. It is not an official guide or endorsement from the cited organizations.

AIYING — Accomplishing with AI
AI Engineering Principles & Practice · 004

AI Engineering Principle 003: Turn Real Failures into Better Tests

AI Engineering Principles & Practice · Post 003

Turn Real Failures into Better Tests

A failure is more than a bug to fix. It can become a test that helps catch the same problem next time.

In this post

• The principle
• A practical example
• Keep the test set balanced
• Independent study exercise
• Research basis

The principle

When an AI system gives an unsupported answer, misses a user’s intent, or uses a tool when it should not, save the example. Turn it into a repeatable test.

A test case should capture the situation that caused the failure, the behavior you expected, and how you will judge whether the system handled it correctly. After changing the prompt, model, tools, or workflow, run the test again. This helps you see whether the change fixed the problem and whether it affected other cases.

This creates a practical learning loop: observe a failure, understand what went wrong, add a test, improve the system, and check the result.

Back to top ↑

A practical example: a course handbook assistant

Imagine an assistant that answers questions using a course handbook. A learner asks about a deadline, but the assistant gives a date that does not appear in the handbook.

Turn that incident into a test case:

• Input: Ask the same question about the deadline.
• Expected behavior: The assistant should use a supported date or say that the handbook does not provide one.
• Evidence: Check whether the response is supported by the relevant handbook passage.
• Pass condition: The assistant does not invent a date.

The next time the system changes, rerun this case. If the assistant now answers correctly, the test guards against a known failure returning.

Back to top ↑

Keep the test set balanced

A test set made only from past failures can push the system too strongly toward avoiding those failures. Include a mix of examples:

• Common tasks: Questions the system should answer well.
• Edge cases: Ambiguous or unusually phrased requests.
• Should-not-do cases: Requests where the system should not guess, search, or take an action.
• Known failures: Real examples that previously produced a poor result.

For example, test both when a handbook assistant should search for a specific course rule and when it should answer directly from information already available. This checks that the system does not simply search every time—or skip searching when it is needed.

Anthropic suggests that 20–50 simple tasks drawn from real failures can be a useful starting point for early agent development. Treat this as a practical starting suggestion, not a fixed requirement: a mature system may need broader and more challenging coverage.

Back to top ↑

Independent study exercise

Choose an AI assistant or feature you use. Create four test cases:

1. A typical request the system should handle.
2. A difficult or ambiguous request.
3. A request that previously produced a poor response, if you have one.
4. A request where the system should ask for clarification, decline, or avoid guessing.

For each case, write down:

• The input you will use.
• The expected behavior.
• The evidence you will check.
• What would count as a pass or failure.

Keep the set and add new cases as you discover failures. Use examples you are authorized to work with, and remove personal or confidential details before using them in shared materials.

Engineering takeaway: Turn real failures into repeatable tests, and keep the test set balanced.

Back to top ↑

Research basis

• Anthropic, “Demystifying Evals for AI Agents” (January 9, 2026). The article recommends starting with a manageable set of tasks drawn from real failures, balancing cases where a behavior should and should not occur, and adding examples as coverage improves.
• OpenAI, “Evaluation Best Practices”. The guide recommends task-specific tests that reflect real-world distributions, using development logs to find evaluation cases, and growing the set over time.

This post is an independent study reference based on publicly available engineering material. It is not an official guide or endorsement from the cited organizations.

AIYING — Accomplishing with AI
AI Engineering Principles & Practice · 003

AI Engineering Principle 001: Start Simple, Add Complexity with Evidence

AI Engineering Principles & Practice · Post 001

Start Simple. Add Complexity with Evidence.

A practical study reference for students, freshers, and teams learning to build useful AI systems.

The principle

A capable AI system does not have to be a complicated one. Begin with the simplest design that could meet the user’s need. Add steps, tools, retrieval, or agent behavior only when you can identify a problem they solve and check whether the change improves the result.

In its engineering guidance on AI agents, Anthropic recommends starting with the simplest solution possible and increasing complexity when needed. The article also describes tradeoffs such as cost and latency when using agentic systems. Read Anthropic’s article.

Worked example: a course handbook assistant

Suppose you are building an assistant that answers questions about a course handbook.

1. Start with one model call and the relevant handbook text.
2. Try a small set of realistic questions, including questions the handbook cannot answer.
3. Record where the answers are incomplete, unsupported, or incorrect.
4. If the system cannot reliably find the relevant passages, test retrieval.
5. Consider additional tools or steps only if the test results show a need for them.

The goal is not to keep the system small at any cost. The goal is to make each added component earn its place by addressing an observed failure or requirement.

Independent study exercise

Choose an AI task you might build for a class project, portfolio, or team. Write down:

1. User outcome: What should the person be able to do or receive?
2. Smallest first design: What is the simplest system you could try?
3. Test examples: What inputs would show whether it works?
4. Failure condition: What result would justify adding another component?
5. Measure: How will you compare the system before and after the change?

Keep your examples and run them again whenever you change the prompt, model, tools, or workflow. Anthropic’s evaluation guidance describes evaluations as a way to make system behavior and changes more visible during development. Read the evaluation guide.

Study takeaway: Add complexity to solve an observed problem, then measure whether the change helped.

Research basis

• Anthropic, “Building Effective Agents”. The article was published in December 2024 and notes that some tooling details have since changed.
• Anthropic, “Demystifying Evals for AI Agents”.

This post is an independent learning reference based on publicly available engineering material. It is not an official guide or endorsement from the cited organizations.

AIYING — Accomplishing with AI
AI Engineering Principles & Practice · 001