Friday, November 13, 2026

AI Engineering Active Concepts #011 — Agent Evaluation: Measuring Whether AI Workflows Actually Succeed

AI ENGINEERING — ACTIVE CONCEPTS & PRACTICES | #011

Agent Evaluation: Measuring Whether AI Workflows Actually Succeed

Chapter 2: Engineering AI Applications and Agents

AIYING — Accomplishing with AI

The Big Idea: Agent evaluation measures whether an AI workflow completes its intended task correctly, reliably, safely, and efficiently.

Fluent responses and successful tool calls do not necessarily demonstrate successful task completion.

Engineering question: What evidence proves that an AI agent accomplished the intended business outcome?

Why Agent Evaluation Matters
UNDERSTAND — The Concept
CONNECT — AI Engineering Relationships
Evaluation Architecture
APPLY — Enterprise Procurement Agent
Worked Example: Evaluating 100 Workflows
Engineering Decisions and Trade-offs
Active Industry Practices
ACCOMPLISH — Practical Exercise
Student and Practitioner Project
Check Your Understanding
Active Connections
References and Further Reading

1. Why Agent Evaluation Matters

Imagine a procurement AI agent assigned to investigate delayed purchase orders and recommend recovery actions.

During testing, it successfully calls the required tools, produces professional reports, and appears to follow the intended workflow.

But closer inspection reveals that some recommendations are based on outdated inventory, some approvals are missing, and several completed tasks were reported incorrectly.

The agent may look effective without being operationally reliable.

The success of an AI agent must be evaluated against the intended outcome, not merely the quality of its generated response.

2. UNDERSTAND — The Concept

What is agent evaluation?

Agent evaluation is the systematic assessment of how well an AI agent or agentic workflow performs defined tasks under specified conditions.

It examines the final outcome, intermediate decisions, tool interactions, operational constraints, and failure behavior.

Five evaluation dimensions

Dimension	Key Question
Correctness	Did the agent produce the right result?
Reliability	Does it succeed consistently across relevant cases?
Efficiency	How much time and cost does execution require?
Safety	Did it respect permissions and operational boundaries?
Usefulness	Did the outcome help accomplish the real task?

Three levels of evaluation

Component evaluation: Tests individual operations such as retrieval, classification, and tool selection.

Trajectory evaluation: Examines the sequence of steps, tool calls, decisions, and state transitions.

Outcome evaluation: Determines whether the overall task was completed correctly.

Evaluation versus monitoring

Evaluation determines whether a system meets defined criteria.

Monitoring observes its behavior during operation.

Production traces and feedback can supply evidence for ongoing evaluation, but monitoring alone does not establish correctness.

3. CONNECT — AI Engineering Relationships

Concept	Connection to Evaluation
Context Engineering	Tests whether the model receives relevant information
Retrieval Engineering	Measures relevance, coverage, and evidence quality
Structured Outputs	Tests schema adherence and data accuracy
Tool Calling	Evaluates tool selection and argument correctness
MCP	Tests external capability discovery and invocation
Agent Orchestration	Evaluates workflow progression and recovery
Observability	Supplies execution traces and operational signals
AI Governance	Defines accountability, safety, and approval requirements
Token Economics	Measures execution costs and efficiency

Active connection: Evaluation turns the technical capabilities studied in earlier lessons into measurable engineering performance.

4. Evaluation Architecture

Define Intended Task and Success Criteria

↓

Build Representative Test Cases

↓

Execute Agent Workflow

↓

Capture Outputs and Execution Traces

↓

Apply Deterministic and Judgment-Based Checks

↓

Analyze Failures and Performance

↓

Improve and Re-Evaluate

Step 1 — Define success

Specify observable completion conditions, acceptable errors, and prohibited actions before testing.

Step 2 — Construct evaluation cases

Include normal cases, edge cases, ambiguous requests, incomplete data, tool failures, and authorization scenarios.

Step 3 — Run the workflow

Execute the system against controlled inputs using representative tools or simulated environments.

Step 4 — Capture evidence

Record final outputs, tool calls, execution state, latency, cost, and relevant decisions.

Step 5 — Score results

Use exact checks where objective answers exist and carefully designed human or model-based judgments where interpretation is required.

Step 6 — Diagnose failures

Determine whether failures originated in retrieval, planning, tool use, reasoning, authorization, or workflow execution.

Step 7 — Improve and retest

Repeat evaluations after changes and check for regressions in previously successful cases.

5. APPLY — Enterprise Procurement Agent

Scenario: Evaluating Purchase Order Recovery

A manufacturing organization uses an AI agent to identify delayed purchase orders and recommend recovery actions.

The system can retrieve order records, inspect alternative stock, examine procurement policies, and prepare approval requests.

What should be evaluated?

Evaluation Area	Example Test
Order retrieval	Did the agent identify the correct purchase order?
Inventory accuracy	Did it use current, authorized inventory data?
Recommendation	Is the proposed recovery action feasible?
Policy compliance	Did it follow procurement constraints?
Approval handling	Did it pause before restricted operations?
Execution reporting	Did it distinguish proposed and completed actions?
Failure handling	Did it recover or escalate appropriately?

Why end-to-end evaluation matters

A system may retrieve the correct order and select the correct tool yet recommend an invalid stock transfer.

Conversely, it may produce a correct recommendation but fail to obtain mandatory approval.

Evaluation must consider the complete workflow and its operational requirements.

6. Worked Example: Evaluating 100 Workflows

Consider the following fictional test results for a procurement agent.

Measure	Illustrative Result
Total test cases	100
Correct final outcomes	82
Correct tool selection	94
Correct tool arguments	90
Approval violations	2
Average execution time	18 seconds
Average estimated execution cost	$0.06

Assume each case contributes one binary result to each applicable measure, and all 100 cases are eligible for the listed accuracy checks.

Task success rate

Task Success Rate =
Successful Cases / Total Cases × 100

= 82 / 100 × 100
= 82%

Tool selection accuracy

Tool Selection Accuracy =
Correct Selections / Evaluated Cases × 100

= 94 / 100 × 100
= 94%

Approval violation rate

Approval Violation Rate =
Cases with Violations / Total Cases × 100

= 2 / 100 × 100
= 2%

For production evaluation, approval violations should also be measured against the subset of cases where approval requirements actually apply.

Interpretation

The agent selects tools correctly in most cases, but its final task-success rate is lower.

This indicates that tool selection alone does not explain overall performance.

More importantly, two approval violations indicate a control failure requiring investigation. A high average success rate cannot compensate for unauthorized consequential actions.

Engineering conclusion

The system requires root-cause analysis, stronger approval enforcement, and targeted regression tests before its suitability for production can be established.

7. Engineering Decisions and Trade-offs

Decision	Benefit	Trade-off
Exact-match scoring	Repeatable and objective	May reject valid alternatives
Human evaluation	Contextual judgment	Time and reviewer variability
LLM-as-judge	Scalable qualitative assessment	Bias and grading errors
Simulated environments	Safe and repeatable tests	May differ from production
Production feedback	Real-world relevance	Noise and incomplete ground truth
Trace-level scoring	Detailed diagnosis	Higher evaluation complexity
End-to-end scoring	Measures task completion	May obscure root causes

Important evaluation risks

Unrepresentative test datasets
Ambiguous or incorrect reference answers
Overreliance on subjective grading
Data leakage between development and test sets
Ignoring rare but severe failures
Confusing fluent explanations with correct outcomes
Failing to retest after changes
Using averages that conceal critical failure cases

Recommended practices

Define success criteria before implementation.
Use representative and adversarial test cases.
Combine deterministic checks with calibrated human judgment.
Capture execution traces.
Separate quality, safety, latency, and cost measures.
Track performance by scenario and failure category.
Maintain regression test suites.
Investigate critical failures independently of aggregate scores.

Evaluation quality depends not only on the metric but also on the relevance of the test cases, the validity of the reference answers, and the consequences of failure.

8. Active Industry Practices

Established practices

Task-based benchmark datasets
Unit and integration testing for tools
Human review of model outputs
Automated regression testing
Latency and cost measurement
Failure classification and analysis

Evolving agent evaluation practices

Trajectory and trace evaluation
LLM-assisted grading
Simulation-based agent testing
Multi-turn and multi-step task evaluation
Continuous evaluation from production feedback
Risk-weighted evaluation of agent actions

Relevant technologies

Engineering teams may use OpenAI Evals and agent evaluation tooling, LangSmith, MLflow, or custom test harnesses depending on their requirements.

Evaluation tools do not replace the need for clearly defined business success criteria and valid test data.

Industry relevance

Agent evaluation is important in enterprise automation, customer support, software development, financial operations, procurement, research assistants, and other systems where AI-generated decisions influence operational outcomes.

Practice status: Established software testing principles combined with rapidly developing agent-specific evaluation methods.

Reviewed: November 2026.

9. ACCOMPLISH — Practical Exercise

Exercise: Evaluate a Procurement Agent

Consider a fictional agent that investigates delayed purchase orders.

Task: Recommend a recovery action.

Purchase order: PO-2056

Required quantity: 500 units

Alternative stock: 350 units

Approval: Required for stock transfer

Your tasks

Define what successful completion means.
Identify the required evidence.
Write five normal test cases.
Write three failure or edge cases.
Define a scoring rubric.
Identify critical safety checks.
Specify the execution information to record.
Describe how failures should be classified.
Recommend a release decision based on the results.

Reusable AI-assisted prompt

Act as an AI engineering evaluation specialist.

Design an evaluation plan for a
fictional procurement AI agent.

The agent:
- Retrieves purchase-order information
- Checks alternative inventory
- Recommends recovery actions
- Requests approval when required
- Reports workflow outcomes

Provide:
1. Task success criteria.
2. Representative test scenarios.
3. Edge and failure cases.
4. Expected results.
5. Tool-use evaluation criteria.
6. Workflow trace checks.
7. Safety and approval tests.
8. Cost and latency metrics.
9. Failure classification.
10. Release-readiness recommendation.

Distinguish simulated test results
from real measured performance.

Accomplishment: Create an evaluation plan that measures whether an AI agent accomplishes its intended task safely and reliably.

10. Student and Practitioner Project

Mini Project: Build an Agent Evaluation Harness

Objective: Develop a small evaluation system for a fictional procurement agent.

Project requirements

Define 20 representative test cases.
Create expected outcomes for each case.
Implement or simulate an agent workflow.
Capture tool calls and execution results.
Score final task completion.
Check tool selection and argument correctness.
Test approval compliance.
Measure execution latency and estimated cost.
Classify failed cases.
Produce a performance report and improvement plan.

Suggested evaluation scorecard

Metric	Purpose
Task success rate	Measures correct final outcomes
Tool selection accuracy	Measures correct tool choice
Argument accuracy	Measures correctness of tool inputs
Policy compliance	Measures adherence to operational rules
Recovery success	Measures handling of failures
Average latency	Measures execution speed
Average cost	Measures resource efficiency
Critical failure count	Tracks severe errors independently

Project deliverable: A test dataset, evaluation harness or design, execution traces, scorecard, failure analysis, and engineering recommendations.

11. Check Your Understanding

What is agent evaluation?
Why is fluent output insufficient evidence of success?
How do component, trajectory, and outcome evaluations differ?
What is the difference between evaluation and monitoring?
When is deterministic scoring preferable?
What are the limitations of LLM-as-judge evaluation?
Why must critical failures be examined separately?
How would you evaluate an agent before production deployment?

Reflection: If an AI agent achieves 98% task success but occasionally performs unauthorized actions, should it be considered production-ready? What additional controls or evidence would be required?

12. Active Connections

Coming next — #012: Observability and Tracing: Understanding What AI Systems Actually Do.

13. References and Further Reading

Evaluation frameworks, metrics, and tooling evolve. Consult current documentation and validate scoring methods against the requirements of each application.

Final Takeaway

An AI agent is not successful because it completed a sequence of steps. It is successful when evidence demonstrates that it achieved the intended outcome correctly, safely, and reliably.

AI Engineering — Active Concepts & Practices
Active Concept #011 — Agent Evaluation