Showing posts with label Agentic AI. Show all posts

Friday, November 13, 2026

AI Engineering Active Concepts #011 — Agent Evaluation: Measuring Whether AI Workflows Actually Succeed

AI ENGINEERING — ACTIVE CONCEPTS & PRACTICES | #011

Agent Evaluation: Measuring Whether AI Workflows Actually Succeed

Chapter 2: Engineering AI Applications and Agents

AIYING — Accomplishing with AI

The Big Idea: Agent evaluation measures whether an AI workflow completes its intended task correctly, reliably, safely, and efficiently.

Fluent responses and successful tool calls do not necessarily demonstrate successful task completion.

Engineering question: What evidence proves that an AI agent accomplished the intended business outcome?

Why Agent Evaluation Matters
UNDERSTAND — The Concept
CONNECT — AI Engineering Relationships
Evaluation Architecture
APPLY — Enterprise Procurement Agent
Worked Example: Evaluating 100 Workflows
Engineering Decisions and Trade-offs
Active Industry Practices
ACCOMPLISH — Practical Exercise
Student and Practitioner Project
Check Your Understanding
Active Connections
References and Further Reading

1. Why Agent Evaluation Matters

Imagine a procurement AI agent assigned to investigate delayed purchase orders and recommend recovery actions.

During testing, it successfully calls the required tools, produces professional reports, and appears to follow the intended workflow.

But closer inspection reveals that some recommendations are based on outdated inventory, some approvals are missing, and several completed tasks were reported incorrectly.

The agent may look effective without being operationally reliable.

The success of an AI agent must be evaluated against the intended outcome, not merely the quality of its generated response.

2. UNDERSTAND — The Concept

What is agent evaluation?

Agent evaluation is the systematic assessment of how well an AI agent or agentic workflow performs defined tasks under specified conditions.

It examines the final outcome, intermediate decisions, tool interactions, operational constraints, and failure behavior.

Five evaluation dimensions

Dimension	Key Question
Correctness	Did the agent produce the right result?
Reliability	Does it succeed consistently across relevant cases?
Efficiency	How much time and cost does execution require?
Safety	Did it respect permissions and operational boundaries?
Usefulness	Did the outcome help accomplish the real task?

Three levels of evaluation

Component evaluation: Tests individual operations such as retrieval, classification, and tool selection.

Trajectory evaluation: Examines the sequence of steps, tool calls, decisions, and state transitions.

Outcome evaluation: Determines whether the overall task was completed correctly.

Evaluation versus monitoring

Evaluation determines whether a system meets defined criteria.

Monitoring observes its behavior during operation.

Production traces and feedback can supply evidence for ongoing evaluation, but monitoring alone does not establish correctness.

3. CONNECT — AI Engineering Relationships

Concept	Connection to Evaluation
Context Engineering	Tests whether the model receives relevant information
Retrieval Engineering	Measures relevance, coverage, and evidence quality
Structured Outputs	Tests schema adherence and data accuracy
Tool Calling	Evaluates tool selection and argument correctness
MCP	Tests external capability discovery and invocation
Agent Orchestration	Evaluates workflow progression and recovery
Observability	Supplies execution traces and operational signals
AI Governance	Defines accountability, safety, and approval requirements
Token Economics	Measures execution costs and efficiency

Active connection: Evaluation turns the technical capabilities studied in earlier lessons into measurable engineering performance.

4. Evaluation Architecture

Define Intended Task and Success Criteria

↓

Build Representative Test Cases

↓

Execute Agent Workflow

↓

Capture Outputs and Execution Traces

↓

Apply Deterministic and Judgment-Based Checks

↓

Analyze Failures and Performance

↓

Improve and Re-Evaluate

Step 1 — Define success

Specify observable completion conditions, acceptable errors, and prohibited actions before testing.

Step 2 — Construct evaluation cases

Include normal cases, edge cases, ambiguous requests, incomplete data, tool failures, and authorization scenarios.

Step 3 — Run the workflow

Execute the system against controlled inputs using representative tools or simulated environments.

Step 4 — Capture evidence

Record final outputs, tool calls, execution state, latency, cost, and relevant decisions.

Step 5 — Score results

Use exact checks where objective answers exist and carefully designed human or model-based judgments where interpretation is required.

Step 6 — Diagnose failures

Determine whether failures originated in retrieval, planning, tool use, reasoning, authorization, or workflow execution.

Step 7 — Improve and retest

Repeat evaluations after changes and check for regressions in previously successful cases.

5. APPLY — Enterprise Procurement Agent

Scenario: Evaluating Purchase Order Recovery

A manufacturing organization uses an AI agent to identify delayed purchase orders and recommend recovery actions.

The system can retrieve order records, inspect alternative stock, examine procurement policies, and prepare approval requests.

What should be evaluated?

Evaluation Area	Example Test
Order retrieval	Did the agent identify the correct purchase order?
Inventory accuracy	Did it use current, authorized inventory data?
Recommendation	Is the proposed recovery action feasible?
Policy compliance	Did it follow procurement constraints?
Approval handling	Did it pause before restricted operations?
Execution reporting	Did it distinguish proposed and completed actions?
Failure handling	Did it recover or escalate appropriately?

Why end-to-end evaluation matters

A system may retrieve the correct order and select the correct tool yet recommend an invalid stock transfer.

Conversely, it may produce a correct recommendation but fail to obtain mandatory approval.

Evaluation must consider the complete workflow and its operational requirements.

6. Worked Example: Evaluating 100 Workflows

Consider the following fictional test results for a procurement agent.

Measure	Illustrative Result
Total test cases	100
Correct final outcomes	82
Correct tool selection	94
Correct tool arguments	90
Approval violations	2
Average execution time	18 seconds
Average estimated execution cost	$0.06

Assume each case contributes one binary result to each applicable measure, and all 100 cases are eligible for the listed accuracy checks.

Task success rate

Task Success Rate =
Successful Cases / Total Cases × 100

= 82 / 100 × 100
= 82%

Tool selection accuracy

Tool Selection Accuracy =
Correct Selections / Evaluated Cases × 100

= 94 / 100 × 100
= 94%

Approval violation rate

Approval Violation Rate =
Cases with Violations / Total Cases × 100

= 2 / 100 × 100
= 2%

For production evaluation, approval violations should also be measured against the subset of cases where approval requirements actually apply.

Interpretation

The agent selects tools correctly in most cases, but its final task-success rate is lower.

This indicates that tool selection alone does not explain overall performance.

More importantly, two approval violations indicate a control failure requiring investigation. A high average success rate cannot compensate for unauthorized consequential actions.

Engineering conclusion

The system requires root-cause analysis, stronger approval enforcement, and targeted regression tests before its suitability for production can be established.

7. Engineering Decisions and Trade-offs

Decision	Benefit	Trade-off
Exact-match scoring	Repeatable and objective	May reject valid alternatives
Human evaluation	Contextual judgment	Time and reviewer variability
LLM-as-judge	Scalable qualitative assessment	Bias and grading errors
Simulated environments	Safe and repeatable tests	May differ from production
Production feedback	Real-world relevance	Noise and incomplete ground truth
Trace-level scoring	Detailed diagnosis	Higher evaluation complexity
End-to-end scoring	Measures task completion	May obscure root causes

Important evaluation risks

Unrepresentative test datasets
Ambiguous or incorrect reference answers
Overreliance on subjective grading
Data leakage between development and test sets
Ignoring rare but severe failures
Confusing fluent explanations with correct outcomes
Failing to retest after changes
Using averages that conceal critical failure cases

Recommended practices

Define success criteria before implementation.
Use representative and adversarial test cases.
Combine deterministic checks with calibrated human judgment.
Capture execution traces.
Separate quality, safety, latency, and cost measures.
Track performance by scenario and failure category.
Maintain regression test suites.
Investigate critical failures independently of aggregate scores.

Evaluation quality depends not only on the metric but also on the relevance of the test cases, the validity of the reference answers, and the consequences of failure.

8. Active Industry Practices

Established practices

Task-based benchmark datasets
Unit and integration testing for tools
Human review of model outputs
Automated regression testing
Latency and cost measurement
Failure classification and analysis

Evolving agent evaluation practices

Trajectory and trace evaluation
LLM-assisted grading
Simulation-based agent testing
Multi-turn and multi-step task evaluation
Continuous evaluation from production feedback
Risk-weighted evaluation of agent actions

Relevant technologies

Engineering teams may use OpenAI Evals and agent evaluation tooling, LangSmith, MLflow, or custom test harnesses depending on their requirements.

Evaluation tools do not replace the need for clearly defined business success criteria and valid test data.

Industry relevance

Agent evaluation is important in enterprise automation, customer support, software development, financial operations, procurement, research assistants, and other systems where AI-generated decisions influence operational outcomes.

Practice status: Established software testing principles combined with rapidly developing agent-specific evaluation methods.

Reviewed: November 2026.

9. ACCOMPLISH — Practical Exercise

Exercise: Evaluate a Procurement Agent

Consider a fictional agent that investigates delayed purchase orders.

Task: Recommend a recovery action.

Purchase order: PO-2056

Required quantity: 500 units

Alternative stock: 350 units

Approval: Required for stock transfer

Your tasks

Define what successful completion means.
Identify the required evidence.
Write five normal test cases.
Write three failure or edge cases.
Define a scoring rubric.
Identify critical safety checks.
Specify the execution information to record.
Describe how failures should be classified.
Recommend a release decision based on the results.

Reusable AI-assisted prompt

Act as an AI engineering evaluation specialist.

Design an evaluation plan for a
fictional procurement AI agent.

The agent:
- Retrieves purchase-order information
- Checks alternative inventory
- Recommends recovery actions
- Requests approval when required
- Reports workflow outcomes

Provide:
1. Task success criteria.
2. Representative test scenarios.
3. Edge and failure cases.
4. Expected results.
5. Tool-use evaluation criteria.
6. Workflow trace checks.
7. Safety and approval tests.
8. Cost and latency metrics.
9. Failure classification.
10. Release-readiness recommendation.

Distinguish simulated test results
from real measured performance.

Accomplishment: Create an evaluation plan that measures whether an AI agent accomplishes its intended task safely and reliably.

10. Student and Practitioner Project

Mini Project: Build an Agent Evaluation Harness

Objective: Develop a small evaluation system for a fictional procurement agent.

Project requirements

Define 20 representative test cases.
Create expected outcomes for each case.
Implement or simulate an agent workflow.
Capture tool calls and execution results.
Score final task completion.
Check tool selection and argument correctness.
Test approval compliance.
Measure execution latency and estimated cost.
Classify failed cases.
Produce a performance report and improvement plan.

Suggested evaluation scorecard

Metric	Purpose
Task success rate	Measures correct final outcomes
Tool selection accuracy	Measures correct tool choice
Argument accuracy	Measures correctness of tool inputs
Policy compliance	Measures adherence to operational rules
Recovery success	Measures handling of failures
Average latency	Measures execution speed
Average cost	Measures resource efficiency
Critical failure count	Tracks severe errors independently

Project deliverable: A test dataset, evaluation harness or design, execution traces, scorecard, failure analysis, and engineering recommendations.

11. Check Your Understanding

What is agent evaluation?
Why is fluent output insufficient evidence of success?
How do component, trajectory, and outcome evaluations differ?
What is the difference between evaluation and monitoring?
When is deterministic scoring preferable?
What are the limitations of LLM-as-judge evaluation?
Why must critical failures be examined separately?
How would you evaluate an agent before production deployment?

Reflection: If an AI agent achieves 98% task success but occasionally performs unauthorized actions, should it be considered production-ready? What additional controls or evidence would be required?

12. Active Connections

Coming next — #012: Observability and Tracing: Understanding What AI Systems Actually Do.

13. References and Further Reading

Evaluation frameworks, metrics, and tooling evolve. Consult current documentation and validate scoring methods against the requirements of each application.

Final Takeaway

An AI agent is not successful because it completed a sequence of steps. It is successful when evidence demonstrates that it achieved the intended outcome correctly, safely, and reliably.

AI Engineering — Active Concepts & Practices
Active Concept #011 — Agent Evaluation

AIYING — Accomplishing with AI

Understand → Connect → Apply → Accomplish

AI Engineering Active Concepts #010 — Agent Orchestration: Designing Reliable Multi-Step AI Workflows

AI ENGINEERING — ACTIVE CONCEPTS & PRACTICES | #010

Agent Orchestration: Designing Reliable Multi-Step AI Workflows

Chapter 2: Engineering AI Applications and Agents

AIYING — Accomplishing with AI

The Big Idea: Agent orchestration coordinates the sequence, dependencies, tools, decisions, state, and controls required to complete multi-step AI tasks.

An individual model response may be useful. An orchestrated workflow must deliver a verifiable outcome.

Engineering question: How do we make an AI workflow reliable when it involves multiple steps, external systems, failures, and human approvals?

Why Agent Orchestration Matters
UNDERSTAND — The Concept
CONNECT — AI Engineering Relationships
Agent Orchestration Architecture
APPLY — Enterprise Procurement Workflow
Worked Example: A Multi-Step Workflow
Engineering Decisions and Trade-offs
Active Industry Practices
ACCOMPLISH — Practical Exercise
Student and Practitioner Project
Check Your Understanding
Active Connections
References and Further Reading

1. Why Agent Orchestration Matters

Consider an enterprise procurement manager asking:

"Check purchase order PO-1048. Verify whether the required material is available elsewhere. If delivery is delayed, recommend a recovery action and prepare the necessary follow-up."

This is not a single question. It is a sequence of dependent tasks.

The AI application may need to retrieve purchase-order information, inspect inventory, compare delivery alternatives, apply procurement policies, prepare a recommendation, and obtain approval before making changes.

Each step depends on information or decisions from previous steps.

What happens if the inventory service is unavailable? What if the purchase-order record is inconsistent? What if a human reviewer rejects the proposed action?

These are orchestration problems.

Agent orchestration is not simply connecting several AI calls. It is engineering the control flow that turns multiple operations into a dependable business outcome.

2. UNDERSTAND — The Concept

What is agent orchestration?

Agent orchestration is the coordination of AI models, tools, workflow steps, decisions, application state, and control mechanisms to accomplish a task.

The orchestrator may be implemented through conventional software logic, a workflow engine, an agent framework, or a combination of these.

Five essential responsibilities

Responsibility	Engineering Question
Planning	Which steps are required?
Coordination	What should run next, and what depends on what?
State management	What has already happened?
Control	Which operations are permitted?
Recovery	What happens when a step fails?

Workflow versus agent

A workflow generally follows a predefined control structure. An agent may dynamically select actions and tools based on intermediate observations.

In practice, production systems often combine both approaches.

For example, an agent may analyze a supplier delay while a deterministic workflow controls approvals, record updates, and completion status.

Common orchestration patterns

Sequential: Execute dependent steps in order.
Parallel: Run independent operations concurrently.
Conditional routing: Choose a branch based on validated results.
Iterative: Repeat a bounded process until a condition is satisfied.
Supervisor-worker: Delegate specialized tasks to different components.
Human-in-the-loop: Pause for an authorized decision.
Event-driven: Resume when an external event occurs.

Not every problem needs multiple agents. A deterministic workflow with one model and a few tools may be simpler, cheaper, and easier to test.

3. CONNECT — AI Engineering Relationships

Concept	Relationship to Orchestration
Context Engineering	Supplies relevant information to each step
Token Economics	Controls the cost of repeated model calls
Retrieval Engineering	Provides external evidence
Context Management	Preserves state and continuity
Structured Outputs	Defines data contracts between steps
Tool Calling	Invokes external capabilities
MCP	Provides standardized access to external capabilities
Evaluation	Tests both individual steps and overall outcomes
Observability	Traces execution, latency, cost, and failures
Governance	Establishes accountability and approval controls

Active connection: Orchestration brings together the capabilities studied in earlier lessons into an operational AI application.

4. Agent Orchestration Architecture

User Request

↓

Workflow Controller / Orchestrator

↓

Task Planning and State Initialization

↓

Model and Tool Execution

↓

Result Validation

↓

Decision: Continue, Retry, Escalate, or Stop?

↓

Human Approval When Required

↓

Authorized Business Action

↓

Outcome Verification and Audit Record

State transitions

A workflow should have explicit states, such as:

CREATED
   |
   v
RUNNING
   |
   +--> WAITING_FOR_APPROVAL
   |           |
   |           v
   |        RUNNING
   |
   +--> FAILED
   |
   +--> COMPLETED

Real implementations may need additional states such as cancelled, timed out, rejected, or requiring manual intervention.

Why state matters

State enables a workflow to resume after interruptions, avoid repeating completed actions, and distinguish a pending operation from a successful one.

State should be stored in a durable application system when the workflow must survive process restarts or extended human approval delays.

5. APPLY — Enterprise Procurement Workflow

Scenario: Recovering from a Supplier Delay

A manufacturing company uses an AI assistant to investigate delayed material deliveries.

The assistant has access to fictional procurement, inventory, and supplier systems.

Step 1 — Retrieve purchase order

The system obtains the current order status through an authorized lookup tool.

Step 2 — Check inventory

The application retrieves stock availability from alternative locations.

Step 3 — Evaluate alternatives

The AI analyzes available evidence and prepares possible recovery options.

Step 4 — Apply business constraints

Deterministic checks verify inventory eligibility, policy requirements, and permitted actions.

Step 5 — Request approval

If a transfer or supplier change requires approval, the workflow pauses for the designated decision-maker.

Step 6 — Execute authorized action

The application invokes the approved operation and verifies its result.

Step 7 — Complete and record

The workflow reports the outcome and records the relevant execution history.

Important: The assistant should never describe an operation as completed merely because the model proposed it.

6. Worked Example: A Multi-Step Workflow

The following is a fictional design illustration, not a live enterprise integration.

Sample workflow state

{
  "workflow_id": "WF-1048",
  "po_id": "PO-1048",
  "status": "waiting_for_approval",
  "order_status": "delayed",
  "alternative_stock_available": true,
  "recommended_action": "internal_transfer",
  "approval_status": "pending",
  "execution_status": "not_started"
}

Illustrative orchestration pseudocode

async def process_order(po_id, user):
    order = await get_order(po_id, user)
    validate_order(order)

    if order["status"] != "delayed":
        return {"status": "completed",
                "action": "none"}

    inventory = await check_inventory(order, user)
    validate_inventory(inventory)

    proposal = await recommend_action(
        order, inventory
    )
    validate_proposal(proposal)

    approval_id = await request_approval(
        proposal, user
    )

    # Persist state and stop execution.
    # Resume only after a verified approval event.
    return {
        "status": "waiting_for_approval",
        "approval_id": approval_id,
        "execution_status": "not_started"
    }

This pseudocode illustrates a workflow boundary. A production implementation must authenticate the approver, persist state, resume through a controlled event, enforce authorization, and verify any eventual write operation.

What makes the workflow reliable?

Each step has defined inputs and outputs.
External results are validated.
State is recorded before suspension.
Approvals are explicit.
Retries are bounded and controlled.
Side-effecting actions use duplicate-prevention safeguards.
Completion is based on verified execution results.

7. Engineering Decisions and Trade-offs

Decision	Benefit	Trade-off
Fixed workflow	Predictability and testability	Less flexibility
Dynamic agent planning	Adapts to varied tasks	Harder to constrain and evaluate
Parallel execution	Reduced latency	Dependency and concurrency risks
Persistent state	Resumability	Storage and consistency complexity
Automatic retries	Failure recovery	Cost and duplicate-action risks
Human approval	Stronger oversight	Additional time
Multiple specialized agents	Division of responsibilities	Coordination overhead

Common failure modes

Infinite or excessive execution loops
Incorrect tool selection
Loss of workflow state
Contradictory intermediate results
Repeated actions after retries
Unauthorized execution
Unbounded token consumption
Incorrect completion reporting
Prompt injection through retrieved data or tool results

Recommended engineering controls

Maximum step and execution budgets
Timeouts and bounded retries
Checkpointing and durable state
Idempotency for operations with side effects
Structured contracts between components
Explicit approval boundaries
Least-privilege tool permissions
Execution tracing and audit records
Outcome-based evaluations

Reliable orchestration requires a controlled workflow, not merely a capable model.

8. Active Industry Practices

Established engineering approaches

State-machine and graph-based workflows
Task dependency management
Tool execution with validation
Human approval checkpoints
Persistent workflow state
Execution tracing and monitoring

Evolving AI engineering practices

Model-directed workflow routing
Supervisor and specialist-agent architectures
Agent evaluation using execution traces
Policy-aware tool orchestration
Adaptive planning with bounded autonomy
Durable execution for long-running agents

Relevant technologies

Developers may evaluate frameworks such as LangGraph, Microsoft Agent Framework, OpenAI Agents SDK, and conventional workflow engines.

These technologies have different architectures and operational characteristics. Framework selection should follow the requirements of the application rather than determine the architecture in advance.

Industry applications

Agent orchestration is relevant to customer support, software engineering, incident response, research assistants, enterprise operations, procurement, document processing, and multi-step business automation.

Practice status: Established workflow-engineering principles combined with evolving agentic AI architectures.

Reviewed: November 2026.

9. ACCOMPLISH — Practical Exercise

Exercise: Design a Procurement Recovery Workflow

Use this fictional scenario:

Purchase order: PO-2056

Status: Delayed

Required quantity: 500 units

Alternative location stock: 350 units

Proposed action: Partial internal transfer

Approval: Required

Your tasks

Define the workflow objective.
List the required tools and data sources.
Identify sequential and parallel steps.
Define workflow states and transitions.
Specify validation checkpoints.
Identify actions requiring approval.
Design failure and retry handling.
Define completion and escalation conditions.
Explain how the remaining 150-unit shortage will be handled.

Reusable AI-assisted prompt

Act as an enterprise AI workflow architect.

Design an agent orchestration workflow
for a delayed purchase order.

Scenario:
- Order: PO-2056
- Required quantity: 500 units
- Alternative stock: 350 units
- Supplier delivery: delayed
- Internal transfer: approval required

Provide:
1. Workflow objective.
2. Required tools and data.
3. Step-by-step execution plan.
4. State transitions.
5. Validation rules.
6. Approval checkpoints.
7. Failure recovery strategy.
8. Cost and execution limits.
9. Observability requirements.
10. Completion and escalation criteria.

Distinguish AI-generated recommendations
from verified system actions.

Do not claim any real action was executed.

Accomplishment: Produce an orchestration design that can be reviewed for reliability, authorization, recovery, and measurable completion.

10. Student and Practitioner Project

Mini Project: Build a Stateful Procurement Agent

Objective: Prototype a multi-step AI workflow using simulated enterprise systems.

Project requirements

Create fictional purchase-order and inventory datasets.
Define lookup and simulated action tools.
Implement a workflow controller.
Persist workflow state between steps.
Use structured inputs and outputs.
Introduce an approval checkpoint.
Simulate a tool timeout and a service failure.
Test workflow resumption after interruption.
Prevent duplicate side effects.
Record execution traces and evaluate outcomes.

Suggested evaluation metrics

Metric	What It Measures
Task completion rate	Successful completion of authorized workflows
Step accuracy	Correct execution of required stages
Recovery success	Resumption after failures
State consistency	Correctness of persisted workflow information
Approval compliance	Whether required approvals are enforced
Duplicate action rate	Repeated side effects after retries
Execution cost	Model and tool resource consumption
End-to-end latency	Total workflow processing time

Project deliverable: A workflow diagram, working prototype or design, sample execution traces, evaluation results, and an engineering recommendation.

11. Check Your Understanding

What is agent orchestration?
How is orchestration different from tool calling?
What is the difference between a fixed workflow and a dynamic agent?
Why is persistent state important?
When should workflow steps run in parallel?
How should approval checkpoints be implemented?
Why are retries dangerous for write operations?
How would you measure end-to-end workflow reliability?

Reflection: If every individual AI step succeeds but the overall workflow performs an unauthorized action, has the system succeeded? What does this reveal about orchestration quality?

12. Active Connections

Coming next — #011: Agent Evaluation: Measuring Whether AI Workflows Actually Succeed.

13. References and Further Reading

Framework capabilities and implementation guidance evolve. Verify the current documentation when building production systems.

Final Takeaway

A reliable AI workflow is not defined by how many agents or tools it uses. It is defined by whether it coordinates work correctly, handles failures, respects controls, and delivers a verified outcome.

AI Engineering — Active Concepts & Practices
Active Concept #010 — Agent Orchestration

AIYING — Accomplishing with AI

Understand → Connect → Apply → Accomplish

Thursday, November 12, 2026

AI Engineering Living Glossary #016 — Agent Orchestration: Coordinating Agent Execution and Handoffs

AI ENGINEERING | LIVING GLOSSARY | #016

AGENT ORCHESTRATION

Coordinating Agent Execution and Handoffs

Chapter 3: Connecting AI to Tools, Workflows, and Agents

Lesson 16 of 24

AIYING — Accomplishing with AI

At 10:15 AM, a software company's payment service begins failing.

Customers cannot complete transactions. The incident response team needs answers quickly.

Imagine an AI-enabled engineering system with three specialized agents:

A diagnostic agent that examines metrics and logs.
A code-analysis agent that inspects recent changes.
A verification agent that checks proposed fixes against available evidence and tests.

Each agent can contribute useful work. But who decides which agent acts first? What information should be passed between them? What happens if their conclusions conflict? Who authorizes a production change?

Simply connecting multiple agents does not answer these questions.

The system needs agent orchestration.

Central engineering insight: Agent orchestration coordinates responsibilities, execution, handoffs, state, and controls so that model-driven components contribute to one verifiable outcome.

1. What Is Agent Orchestration?

Agent orchestration is the coordination of one or more AI agents, tools, workflow operations, and human participants to accomplish a larger goal under defined execution and control rules.

It may manage:

Task assignment and delegation.
Execution order and dependencies.
Parallel and sequential operations.
Agent-to-agent handoffs.
Shared task state and context.
Validation of intermediate results.
Permissions and human approvals.
Retries, timeouts, and recovery.
Completion and outcome verification.

Orchestration does not necessarily require multiple agents. A single agent may be orchestrated alongside deterministic software and tools.

Likewise, a multi-agent system does not automatically need a separate AI-powered supervisor. Coordination can be implemented through conventional software.

Key distinction: Agents contribute decisions or specialized work. Orchestration determines how that work fits into the larger process.

Without explicit coordination	With engineered orchestration
Responsibilities may overlap.	Task ownership is defined.
Handoffs may be informal.	Handoffs use structured contracts.
Agents may use inconsistent context.	Relevant state is managed and versioned.
Execution may be difficult to trace.	Steps and dependencies are recorded.
Failures may leave uncertain outcomes.	Recovery and escalation paths are specified.
Local agent success may be mistaken for task completion.	The overall business outcome is verified.

Concept	Main role	Illustration
Tool Calling	Requests an external capability.	Read incident logs.
Workflow	Defines and executes a coordinated process.	Incident intake, triage, review, closure.
AI Agent	Selects actions toward a goal using model-guided decisions.	Investigates an unfamiliar error.
Agent Orchestration	Coordinates agent tasks, dependencies, handoffs, and execution controls.	Assigns investigation and verification to specialized agents.

Domain	Potential orchestration pattern
Software engineering	Diagnosis, code analysis, testing, human review.
Customer support	Issue classification, specialist investigation, escalation.
Business research	Source gathering, evidence analysis, synthesis, verification.
Supply chain	Exception detection, inventory analysis, replenishment recommendation.
Financial operations	Document review, reconciliation, exception handling, approval.
Enterprise knowledge	Retrieval, comparison, summarization, factual review.
Security operations	Alert triage, evidence collection, analysis, authorized response.
Content production	Research, drafting, editorial checking, publication approval.

Component	Responsibility	Permission boundary
Orchestrator	Assign tasks, manage dependencies and state.	Cannot bypass approvals.
Diagnostic agent	Inspect metrics, errors, and incident timelines.	Read-only telemetry.
Code-analysis agent	Inspect relevant commits and code paths.	Read-only repository access.
Verification agent	Assess evidence and run permitted tests.	Isolated test environment.
Human incident lead	Authorize consequential remediation.	Established organizational authority.

Metric	Evaluation question
Task completion rate	Did the system reach an appropriate verified outcome or escalation?
Handoff validity	Were required fields and evidence references present?
Coordination accuracy	Were tasks assigned to appropriate components?
Authorization violations	Did any prohibited operation execute?
Conflict handling	Were contradictory findings handled explicitly?
Recovery success	Could the system resolve or safely escalate failures?
Execution efficiency	How many model and tool calls were required?
End-to-end latency	How long did the complete process take?
Evidence fidelity	Were final conclusions supported by the available evidence?

Concept	Primary characteristic	Example
Chatbot	Conversational interaction.	Explains how VPN authentication works.
Tool Calling	Requests an external capability.	Queries VPN client version.
Workflow	Coordinates a designed process.	Runs a fixed set of VPN checks.
AI Agent	Selects actions dynamically toward a goal.	Chooses the next diagnostic based on observed results.

Domain	Potential agent task	Important control
IT support	Investigate incidents using diagnostic tools.	Restrict system changes and credential access.
Software engineering	Inspect code, propose patches, run tests.	Sandbox execution and require review before deployment.
Customer service	Investigate complex support requests.	Protect customer records and constrain transactions.
Business research	Collect and compare information.	Verify sources and preserve provenance.
Operations	Investigate inventory or process exceptions.	Control operational changes.
Security operations	Assist with alert triage and evidence gathering.	Require authorization for containment actions.
Data analysis	Explore datasets and generate findings.	Limit data access and verify calculations.

Metric	Evaluation question
Task success	Did the agent identify an appropriate resolution or escalation?
Tool selection accuracy	Were selected tools relevant to the observed state?
Authorization violations	Did any prohibited operation execute?
Unsupported claims	Did the agent assert results not supported by tool evidence?
Loop termination	Did execution stop within the defined limits?
Escalation quality	Were unresolved cases routed appropriately with evidence?
Tool efficiency	How many calls were needed for a useful outcome?
Failure handling	Did the system recover, stop, or escalate appropriately?

Friday, November 13, 2026

AI Engineering Active Concepts #011 — Agent Evaluation: Measuring Whether AI Workflows Actually Succeed

Agent Evaluation: Measuring Whether AI Workflows Actually Succeed

Contents

1. Why Agent Evaluation Matters

2. UNDERSTAND — The Concept

What is agent evaluation?

Five evaluation dimensions

Three levels of evaluation

Evaluation versus monitoring

3. CONNECT — AI Engineering Relationships

4. Evaluation Architecture

Step 1 — Define success

Step 2 — Construct evaluation cases

Step 3 — Run the workflow

Step 4 — Capture evidence

Step 5 — Score results

Step 6 — Diagnose failures

Step 7 — Improve and retest

5. APPLY — Enterprise Procurement Agent

Scenario: Evaluating Purchase Order Recovery

What should be evaluated?

Why end-to-end evaluation matters

6. Worked Example: Evaluating 100 Workflows

Task success rate

Tool selection accuracy

Approval violation rate

Interpretation

Engineering conclusion

7. Engineering Decisions and Trade-offs

Important evaluation risks

Recommended practices

8. Active Industry Practices

Established practices

Evolving agent evaluation practices

Relevant technologies

Industry relevance

9. ACCOMPLISH — Practical Exercise

Exercise: Evaluate a Procurement Agent

Your tasks

Reusable AI-assisted prompt

10. Student and Practitioner Project

Mini Project: Build an Agent Evaluation Harness

Project requirements

Suggested evaluation scorecard

11. Check Your Understanding

12. Active Connections

13. References and Further Reading

Final Takeaway

AI Engineering Active Concepts #010 — Agent Orchestration: Designing Reliable Multi-Step AI Workflows

Agent Orchestration: Designing Reliable Multi-Step AI Workflows

Contents

1. Why Agent Orchestration Matters

2. UNDERSTAND — The Concept

What is agent orchestration?

Five essential responsibilities

Workflow versus agent

Common orchestration patterns

3. CONNECT — AI Engineering Relationships

4. Agent Orchestration Architecture

State transitions

Why state matters

5. APPLY — Enterprise Procurement Workflow

Scenario: Recovering from a Supplier Delay

Step 1 — Retrieve purchase order

Step 2 — Check inventory

Step 3 — Evaluate alternatives

Step 4 — Apply business constraints

Step 5 — Request approval

Step 6 — Execute authorized action

Step 7 — Complete and record

6. Worked Example: A Multi-Step Workflow

Sample workflow state

Illustrative orchestration pseudocode

What makes the workflow reliable?

7. Engineering Decisions and Trade-offs

Common failure modes

Recommended engineering controls

8. Active Industry Practices

Established engineering approaches