Showing posts with label AI Workflows. Show all posts

Thursday, November 12, 2026

AI Engineering — Living Glossary #018 | Structured Outputs: Making AI Responses Machine-Readable and Verifiable

AI Engineering — Living Glossary #018

Lesson 18 | Chapter 3 Finale

Structured Outputs

Making AI Responses Machine-Readable and Verifiable

Chapter 3: Connecting AI to Tools, Workflows, and Agents

Understand the Term. Connect the Technology. Apply the Knowledge.

The Problem: A Correct Answer That Breaks the System

An accounts-payable team introduces an AI assistant to extract information from supplier invoices. The assistant reads an invoice and replies:

The invoice is from Northstar Components. It is for ₹48,500 and is due on 30 November 2026.

The answer looks useful to a human. But the accounting application expects an invoice number, supplier identifier, amount as a number, currency code, and due date in a specific format.

A human can interpret the response. Software should not have to guess where one field ends and another begins.

The engineer therefore changes the design: the AI must return data that conforms to a predefined schema.

Engineering insight: Generating the right words and generating a reliable software interface are different tasks.

↑ Back to contents

1. What Are Structured Outputs?

Structured outputs are AI-generated responses required to follow a defined data structure, typically specified using a schema.

Instead of producing unrestricted prose, the model produces data that an application can parse and validate.

Common representations include JSON objects, typed records, arrays, enumerated values, and other machine-readable formats.

Example

{ "invoice_number": "INV-2048", "supplier": "Northstar Components", "amount": 48500.00, "currency": "INR", "due_date": "2026-11-30", "requires_review": false }

The response now contains named fields, predictable data types, and a standardized date.

However, a valid structure does not prove that the invoice number, amount, or date was extracted correctly.

Critical distinction: Schema validity is not factual accuracy. Both must be checked separately.

↑ Back to contents

2. Why AI Engineers Need Structured Outputs

AI engineering increasingly involves connecting language models to conventional software systems. These systems expect predictable interfaces.

Integration: Applications can consume named fields without interpreting prose.
Automation: Workflows can route cases using explicit classifications and values.
Validation: Programs can reject missing, malformed, or unexpected data.
Testing: Outputs can be compared against known reference records.
Observability: Engineers can track failures by field and schema version.
Governance: Data contracts make downstream responsibilities clearer.

Structured outputs are especially valuable when an AI response triggers another system, creates a database record, or informs a consequential decision.

↑ Back to contents

3. Understanding the Data Contract

A schema defines the shape and constraints of expected data.

It can specify field names, data types, required properties, allowed values, nested objects, and restrictions on additional fields.

Illustrative JSON Schema

{ "type": "object", "properties": { "invoice_number": { "type": "string" }, "amount": { "type": "number" }, "currency": { "type": "string", "enum": ["INR", "USD", "EUR"] }, "due_date": { "type": ["string", "null"] }, "requires_review": { "type": "boolean" } }, "required": [ "invoice_number", "amount", "currency", "due_date", "requires_review" ], "additionalProperties": false }

This schema requires every listed field, restricts the currency to three codes, and prevents undeclared properties.

The due date is nullable because the source document may not contain one. A production application should also validate its date format.

Different model providers support different subsets of JSON Schema. The schema must be designed for the selected API's supported structured-output mode.

Design principle: The schema is an interface contract, not a substitute for business rules.

↑ Back to contents

4. How Structured Outputs Work

Step 1 — Define: The application defines its required schema.

Step 2 — Request: The application supplies the task, source content, and output contract to the model.

Step 3 — Generate: The model produces a structured response. Supported constrained-generation modes can restrict the possible output format.

Step 4 — Validate: Application code checks the output against the schema and business constraints.

Step 5 — Verify: Relevant values are checked against source evidence or authoritative records.

Step 6 — Act: The application accepts, rejects, retries, or escalates the result.

A production implementation must also handle refusals, incomplete responses, timeouts, unsupported schemas, and API-specific failure conditions.

The application—not the language model— ultimately decides whether the output is acceptable for downstream use.

↑ Back to contents

5. Approaches and Implementation Patterns

A. Prompt-Only Formatting

The prompt asks the model to return JSON. This is easy to prototype but does not provide a strong structural guarantee.

B. JSON-Format Mode

Some APIs provide a mode intended to produce syntactically valid JSON. Valid JSON alone does not guarantee the required fields or types.

C. Schema-Constrained Generation

Supported APIs can constrain generation to match a supplied schema or supported schema subset. This offers stronger structural reliability than instructions alone.

D. Application-Side Validation

The application parses and validates the returned object, regardless of the model's generation mode.

E. Validate, Repair, or Escalate

Invalid results can trigger a bounded retry, a correction request, a fallback workflow, or human review.

Repeated retries should not be unlimited: they increase cost and latency without guaranteeing correctness.

↑ Back to contents

6. Structured Outputs vs. Related Concepts

Concept	Main Purpose	Key Distinction
Prompt Engineering	Guide model behavior	Instructions alone do not enforce a schema
JSON Mode	Produce JSON syntax	Valid JSON may violate the expected schema
Structured Outputs	Conform to a data contract	Focuses on predictable response structure
Tool Calling	Propose an external operation	Tool arguments can be structured, but execution is separate
Guardrails	Constrain unsafe or invalid behavior	Schema checks cover only part of the risk
Grounding	Connect claims to evidence	Addresses factual support, not merely formatting

A model can return perfectly valid JSON containing an incorrect invoice total. Conversely, a factually correct explanation may be impossible for an application to parse.

Reliable systems need both structural correctness and factual verification.

↑ Back to contents

7. Production Architecture

This architecture separates generation, validation, verification, authorization, and execution.

It also makes failures easier to diagnose: a parsing error is different from an incorrect extracted value or an unauthorized downstream action.

↑ Back to contents

8. Reliability, Safety, and Security

Schema Validity

Check field presence, data types, enumerations, nesting, and unexpected keys.

Semantic Validity

Check whether values make sense. An invoice amount cannot be negative unless the supported document type permits credits or adjustments.

Evidence Verification

Compare extracted values with the original document or trusted systems. A model's confidence claim is not independent evidence.

Prompt Injection

Documents may contain instructions attempting to influence the model. Treat document text as untrusted data, not as authority to change the workflow.

Authorization

Even valid structured data must not automatically authorize payments, record changes, or external communication.

Operational Controls

Set maximum retry counts and timeouts.
Version schemas and record compatibility changes.
Use least-privilege access for downstream systems.
Log validation outcomes without unnecessarily exposing sensitive data.
Escalate ambiguous or high-risk cases.

Remember: A valid JSON object is not a security approval, a factual guarantee, or permission to execute an action.

↑ Back to contents

9. Industry Applications

Industry	Structured Output	Downstream Use
Finance	Invoice fields	Validation and payment review
Customer Support	Category, priority, summary	Ticket routing
Manufacturing	Defect classification	Quality workflow
Healthcare Administration	Document metadata	Administrative processing with appropriate controls
Cybersecurity	Alert indicators and severity	Analyst triage
Human Resources	Application metadata	Administrative organization and review

The recurring pattern is the same: language understanding is converted into a predictable software interface.

↑ Back to contents

10. Industry Case: Invoice Processing

Illustrative enterprise scenario, not a claim about a specific company.

The Business Problem

A company receives thousands of supplier invoices in different layouts. Manual entry is slow, while ordinary OCR provides text without always identifying business meaning.

The Proposed AI System

Receive a supplier invoice.
Extract text and document layout information.
Ask the model for fields matching an invoice schema.
Validate the returned structure.
Check arithmetic and required business fields.
Match supplier and purchase-order information.
Route mismatches to a human reviewer.
Submit approved records to the accounting system.

Example Exception

The invoice states a total of ₹48,500. The model returns the same amount, but the purchase order authorizes ₹45,800.

The structured response is valid. The extracted amount may also be accurate. Nevertheless, the transaction should be flagged because it does not match the purchasing record.

What the Engineer Learns

Structural correctness, extraction accuracy, business eligibility, and execution authorization are separate decisions.

Case insight: Structured outputs make exceptions detectable; they do not eliminate the need to investigate exceptions.

↑ Back to contents

11. Engineering Design Decisions

Before implementing structured outputs, the engineer must decide:

Which fields are genuinely required by downstream applications?
Should an unknown value be null, omitted, or an explicit status?
Which fields require controlled enumerations?
Which constraints belong in the schema versus business validation?
What happens when the model refuses, times out, or returns incomplete data?
When is retry appropriate, and when should the system escalate?
How will schema versions be managed across multiple consuming services?
Which actions require human approval?

A good schema should be sufficiently precise for dependable integration without forcing the model to invent information that the source does not contain.

↑ Back to contents

12. Hands-On Mini-Project

Build a Structured Support-Ticket Classifier

Objective: Convert an unstructured customer message into a validated support record.

Input

I cannot sign in to my account after resetting my password. I need access before tomorrow's client meeting.

Required Output

{ "category": "access", "priority": "high", "summary": "User cannot sign in after password reset", "needs_human_review": false }

This is an illustrative expected result. The actual priority must follow a documented support policy.

Student Tasks

Define a schema with category, priority, summary, and needs_human_review.
Restrict category to access, billing, technical, or other.
Restrict priority to low, medium, or high.
Submit at least 20 varied test messages to an AI model.
Validate each output with a JSON Schema validator.
Compare the classification with human-reviewed expected labels.
Record failures and improve the schema or instructions.

Python Validation Example

The following example validates a response already received from a model. It does not make a model API call.

# Install once: # pip install jsonschema from jsonschema import validate, ValidationError schema = { "type": "object", "properties": { "category": { "type": "string", "enum": [ "access", "billing", "technical", "other" ] }, "priority": { "type": "string", "enum": ["low", "medium", "high"] }, "summary": { "type": "string", "minLength": 1 }, "needs_human_review": { "type": "boolean" } }, "required": [ "category", "priority", "summary", "needs_human_review" ], "additionalProperties": False } result = { "category": "access", "priority": "high", "summary": "Cannot sign in after reset", "needs_human_review": False } try: validate(instance=result, schema=schema) print("Schema valid") except ValidationError as error: print("Schema invalid:", error.message)

Extend the project by rejecting empty summaries, checking ticket categories against policy, and routing uncertain classifications to review.

↑ Back to contents

13. Evaluation and Acceptance Criteria

Metric	What It Measures	Illustrative Target
Schema compliance	Percentage passing schema validation	100% before downstream acceptance
Category accuracy	Agreement with reviewed labels	At least 90% in the exercise
Priority accuracy	Agreement with defined policy	At least 90% in the exercise
Escalation recall	Fraction of review-required cases flagged	100% for designated critical test cases
Retry rate	Fraction requiring another attempt	Measure and minimize
Latency	Time to a validated result	Set according to application needs

These are learning targets, not universal production benchmarks. A real deployment requires larger representative datasets, failure analysis, and risk-based thresholds.

The student should report results separately for syntax, schema compliance, classification quality, and business-rule compliance.

↑ Back to contents

14. Connected Living Glossary Terms

#006 — Prompt Engineering : Define the task and output expectations.
#007 — Evals : Measure structural and semantic correctness.
#009 — Grounding : Connect extracted facts to evidence.
#010 — Guardrails : Enforce validation and safety boundaries.
#011 — Observability : Monitor failures and operational quality.
#013 — Tool Calling : Use structured arguments for tool requests.
#014 — Workflows : Pass validated data between processing steps.
#017 — Model Context Protocol : Standardize how applications connect to external tools and context sources.

↑ Back to contents

15. Living Update

Structured-output capabilities continue to evolve across AI platforms. Engineers should periodically review:

Supported schema features and restrictions.
Constrained-generation guarantees.
Handling of refusals and incomplete responses.
Streaming behavior and partial results.
SDK validation and parsing capabilities.
Compatibility with tool calling and agent systems.

Update discipline: Test schema behavior against the actual model and API version used in production. Do not assume that a provider's entire JSON Schema specification is supported.

↑ Back to contents

16. Key Takeaway and One Action

Understand the Term: Structured outputs require AI responses to follow a defined data contract.

Connect the Technology: Schemas connect language-model generation to conventional software systems.

Apply the Knowledge: Validate the structure, verify the facts, and authorize actions separately.

One Action: Take one AI task that currently returns prose. Design a schema, test 20 examples, and measure the failure rate.

A well-formed response is ready to be checked—not automatically ready to be trusted.

↑ Back to contents

17. References and Further Learning

↑ Back to contents

AI Engineering Living Glossary #016 — Agent Orchestration: Coordinating Agent Execution and Handoffs

AI ENGINEERING | LIVING GLOSSARY | #016

AGENT ORCHESTRATION

Coordinating Agent Execution and Handoffs

Chapter 3: Connecting AI to Tools, Workflows, and Agents

Lesson 16 of 24

AIYING — Accomplishing with AI

At 10:15 AM, a software company's payment service begins failing.

Customers cannot complete transactions. The incident response team needs answers quickly.

Imagine an AI-enabled engineering system with three specialized agents:

A diagnostic agent that examines metrics and logs.
A code-analysis agent that inspects recent changes.
A verification agent that checks proposed fixes against available evidence and tests.

Each agent can contribute useful work. But who decides which agent acts first? What information should be passed between them? What happens if their conclusions conflict? Who authorizes a production change?

Simply connecting multiple agents does not answer these questions.

The system needs agent orchestration.

Central engineering insight: Agent orchestration coordinates responsibilities, execution, handoffs, state, and controls so that model-driven components contribute to one verifiable outcome.

1. What Is Agent Orchestration?

Agent orchestration is the coordination of one or more AI agents, tools, workflow operations, and human participants to accomplish a larger goal under defined execution and control rules.

It may manage:

Task assignment and delegation.
Execution order and dependencies.
Parallel and sequential operations.
Agent-to-agent handoffs.
Shared task state and context.
Validation of intermediate results.
Permissions and human approvals.
Retries, timeouts, and recovery.
Completion and outcome verification.

Orchestration does not necessarily require multiple agents. A single agent may be orchestrated alongside deterministic software and tools.

Likewise, a multi-agent system does not automatically need a separate AI-powered supervisor. Coordination can be implemented through conventional software.

Key distinction: Agents contribute decisions or specialized work. Orchestration determines how that work fits into the larger process.

↑ Back to contents

2. Why Agent Orchestration Matters

A collection of capable agents can still produce an unreliable system.

Consider these possible failures:

Two agents independently modify the same resource.
An analysis agent uses outdated information.
A verification agent receives an incomplete handoff.
One agent reports success while another reports failure.
An agent repeats a costly operation unnecessarily.
A production deployment begins without approval.
The overall process stops without recording its final state.

Orchestration addresses coordination problems that individual agent intelligence cannot reliably solve on its own.

Without explicit coordination	With engineered orchestration
Responsibilities may overlap.	Task ownership is defined.
Handoffs may be informal.	Handoffs use structured contracts.
Agents may use inconsistent context.	Relevant state is managed and versioned.
Execution may be difficult to trace.	Steps and dependencies are recorded.
Failures may leave uncertain outcomes.	Recovery and escalation paths are specified.
Local agent success may be mistaken for task completion.	The overall business outcome is verified.

Orchestration makes coordination explicit. It does not eliminate the need to evaluate the underlying agents.

↑ Back to contents

3. Common Agent Orchestration Patterns

A. Sequential handoff

One agent completes a defined task and passes its result to the next.

Diagnostic Agent
       |
       v
Code Analysis Agent
       |
       v
Verification Agent
       |
       v
Human Review

Useful when later work depends on earlier findings.

B. Supervisor or coordinator pattern

A coordinator selects which specialized agent should perform the next task.

          Coordinator
          /    |    \
         v     v     v
      Agent A Agent B Agent C
          \    |    /
           v   v   v
          Integration

The coordinator may use deterministic rules, a model, or both.

C. Parallel specialization

Independent agents work concurrently on separate subtasks.

For example, one agent investigates logs while another examines recent code changes.

Their results must be synchronized before a dependent decision proceeds.

D. Planner-executor pattern

A planning component proposes tasks; execution components carry them out under application control.

Plans must remain revisable when observations contradict assumptions.

E. Review-and-revise pattern

One agent generates a candidate result and another evaluates it against defined criteria.

The review process requires stopping limits. Repeated agreement between models is not independent proof of correctness.

F. Human-gated orchestration

Automated investigation proceeds until an operation requires authorized human approval.

Approval must be enforced by the application, not inferred from conversational text.

Design principle: Choose the simplest coordination pattern that meets the task's requirements. More agents create more communication, integration, and failure points.

↑ Back to contents

4. How Agent Orchestration Works

A representative orchestration process is:

       Business Goal
             |
             v
    Validate Request
             |
             v
     Create Task State
             |
             v
   Select Coordination Plan
             |
             v
   Assign Agent Responsibilities
             |
             v
    Execute Permitted Tasks
             |
             v
  Validate Intermediate Results
             |
             v
    Handoff / Synchronization
             |
             v
  Integrate Findings and Evidence
             |
             v
     Approval if Required
             |
             v
    Verify Final Outcome
             |
             v
    Complete or Escalate

Some stages may repeat, branch, or execute in parallel.

The orchestrator should track what has completed, what is pending, which outputs are authoritative, and which operations remain blocked.

The model may recommend a next step, but trusted application controls must decide whether it is permitted to execute.

↑ Back to contents

5. Orchestration vs Workflows and Agents

Concept	Main role	Illustration
Tool Calling	Requests an external capability.	Read incident logs.
Workflow	Defines and executes a coordinated process.	Incident intake, triage, review, closure.
AI Agent	Selects actions toward a goal using model-guided decisions.	Investigates an unfamiliar error.
Agent Orchestration	Coordinates agent tasks, dependencies, handoffs, and execution controls.	Assigns investigation and verification to specialized agents.

These concepts overlap.

An agent can operate within a workflow. A workflow engine can orchestrate multiple agents. A single agent can coordinate tool use internally.

The terms describe different engineering responsibilities, not necessarily separate products or software layers.

↑ Back to contents

6. Engineering Architecture

A production-oriented agent orchestration architecture may include:

1. Task intake

Receives the goal, relevant inputs, identity, and execution constraints.

2. Orchestration controller

Schedules tasks, tracks dependencies, manages state transitions, and coordinates handoffs.

3. Agent registry

Defines agent capabilities, responsibilities, and allowed operations.

4. Agent execution environments

Run model-driven tasks with appropriate tool access and resource limits.

5. Tool gateway

Validates and authorizes access to APIs, data stores, and other capabilities.

6. Shared task state

Stores structured progress, artifacts, and relevant evidence.

7. Validation and integration layer

Checks agent outputs, resolves dependencies, and assembles results.

8. Policy and approval layer

Enforces business restrictions and required human decisions.

9. Observability layer

Tracks execution, latency, tool usage, failures, and costs.

10. Outcome verification

Confirms that the intended task was completed using appropriate evidence.

             User / Business Goal
                      |
                      v
             Orchestration Layer
                      |
          +-----------+-----------+
          |           |           |
          v           v           v
      Agent A      Agent B      Agent C
          |           |           |
          +-----------+-----------+
                      |
                      v
          Validated Handoff Results
                      |
                      v
           Integration and Review
                      |
                      v
             Approval Gate
                      |
                      v
           Verified Final Outcome

Across all stages:
State | Permissions | Monitoring | Recovery

Not every application requires all these components as separate services. Smaller systems can implement several responsibilities in one application.

↑ Back to contents

7. Handoffs and Shared State

A handoff transfers responsibility for a task or subtask from one component to another.

Reliable handoffs require more than forwarding a conversational transcript.

What should a handoff contain?

Task and correlation identifiers.
Sending and receiving component roles.
Clear objective and expected output.
Relevant validated inputs.
References to supporting evidence.
Current status and known uncertainties.
Permissions and execution constraints.
Output schema and completion criteria.

Illustrative handoff contract

{
  "task_id": "INC-204",
  "from": "diagnostic_agent",
  "to": "code_analysis_agent",
  "objective": "Inspect recent changes",
  "service": "payment-api",
  "evidence_refs": [
    "log-set-17",
    "metric-set-08"
  ],
  "suspected_area": "timeout handling",
  "confidence": "tentative",
  "allowed_actions": [
    "read_repository",
    "inspect_commits"
  ],
  "prohibited_actions": [
    "merge_code",
    "deploy"
  ],
  "expected_output": {
    "findings": "array",
    "evidence_refs": "array",
    "uncertainties": "array"
  }
}

This is an illustrative application-level contract, not a universal agent communication standard.

Shared state is not unrestricted shared memory

Different agents may need different subsets of information.

Use access controls, versioned artifacts, and clear ownership to prevent accidental exposure, conflicting updates, or stale decisions.

Handoff validation

The receiving component should validate the handoff structure and independently enforce its own permissions.

Claims generated by one agent should not automatically become trusted facts for another.

Handoff principle: Transfer the minimum sufficient context, preserve evidence and uncertainty, and reapply authorization at each execution boundary.

↑ Back to contents

8. Security, Reliability, and Recovery

Least privilege

Each agent receives only the tools and permissions needed for its responsibility.

Independent authorization

The orchestrator and tool gateway enforce access controls outside model-generated instructions.

Untrusted inputs

Logs, documents, repository files, and agent outputs may contain misleading content or prompt-injection attempts.

Structured validation

Check handoff schemas, required fields, artifact references, and permitted transitions.

Timeouts and budgets

Limit execution duration, tool calls, retries, and total resource consumption.

Concurrency control

Prevent agents from making conflicting updates to shared resources.

Idempotency

Protect state-changing operations against duplicate execution after retries or restarts.

Checkpointing

Persist sufficient state to resume or safely terminate interrupted tasks.

Human intervention

Route high-risk, uncertain, or conflicting outcomes to authorized reviewers.

Outcome verification

Use independent system evidence to verify consequential operations.

Traceability

Record which component performed each operation, which evidence it used, and what actually happened.

Important: If a deployment request times out, the orchestrator should check the deployment system's authoritative status before attempting another deployment.

↑ Back to contents

9. Industry Applications

Domain	Potential orchestration pattern
Software engineering	Diagnosis, code analysis, testing, human review.
Customer support	Issue classification, specialist investigation, escalation.
Business research	Source gathering, evidence analysis, synthesis, verification.
Supply chain	Exception detection, inventory analysis, replenishment recommendation.
Financial operations	Document review, reconciliation, exception handling, approval.
Enterprise knowledge	Retrieval, comparison, summarization, factual review.
Security operations	Alert triage, evidence collection, analysis, authorized response.
Content production	Research, drafting, editorial checking, publication approval.

Specialized agents may improve modularity, but the benefit should be measured against the additional cost, latency, and coordination complexity.

↑ Back to contents

10. Engineering Design Decisions

Does this task require more than one agent?
Could a single agent or deterministic workflow be simpler?
Which responsibilities should be specialized?
Which tasks depend on other tasks?
Which tasks can run in parallel?
Who decides the next agent to execute?
What information must each handoff include?
Where will authoritative state be stored?
How will conflicting findings be resolved?
Which tools may each agent access?
Which actions require human approval?
How will execution limits be enforced?
How will interrupted tasks recover?
How will intermediate results be evaluated?
How will the final outcome be independently verified?
How will the multi-agent design be compared with a simpler baseline?

Engineering decision: Use specialization only where it produces demonstrable improvements in quality, maintainability, or task success.

↑ Back to contents

11. Practical Industry Case: Coordinating a Production Incident

Scenario: A fictional software company experiences increased payment failures after a recent release.

An AI-assisted incident response system is allowed to investigate using read-only tools. Production changes require human authorization through the organization's deployment process.

Agent responsibilities

Component	Responsibility	Permission boundary
Orchestrator	Assign tasks, manage dependencies and state.	Cannot bypass approvals.
Diagnostic agent	Inspect metrics, errors, and incident timelines.	Read-only telemetry.
Code-analysis agent	Inspect relevant commits and code paths.	Read-only repository access.
Verification agent	Assess evidence and run permitted tests.	Isolated test environment.
Human incident lead	Authorize consequential remediation.	Established organizational authority.

Step-by-step execution

Trigger: Monitoring reports an abnormal payment failure rate.
Intake: The orchestrator creates incident task INC-204.
Assignment: The diagnostic agent receives a bounded investigation request.
Observation: Metrics show increased timeouts after a release.
Handoff: The diagnostic agent passes evidence references and uncertainty to code analysis.
Analysis: The code-analysis agent identifies a potentially relevant timeout configuration change.
Verification: The verification agent evaluates the proposed explanation using approved tests and available evidence.
Integration: The orchestrator assembles findings, conflicts, and unresolved questions.
Review: The human incident lead examines the evidence and decides whether to approve remediation.
Execution: If authorized, the organization's controlled deployment workflow performs the change.
Verification: Operational monitoring checks whether the failure rate returns to an acceptable level.
Closure: The incident is closed only after the required verification and record updates.

Failure scenario A: Conflicting conclusions

The diagnostic agent suspects a network problem, while the code-analysis agent suspects a configuration change.

The orchestrator records both hypotheses and requests additional evidence rather than treating either claim as established fact.

Failure scenario B: Missing handoff evidence

The code-analysis agent receives a conclusion without supporting logs or metrics.

The handoff fails validation, and the orchestrator requests the missing evidence.

Failure scenario C: Verification agent timeout

The orchestrator records the incomplete verification and follows a bounded retry or human escalation policy.

Failure scenario D: Unauthorized deployment request

An agent proposes deploying a fix directly.

The application blocks the operation because production deployment requires authorization through a separate controlled process.

What this case demonstrates

Individual agents can investigate different aspects of a problem. The orchestrator is responsible for ensuring that their work is coordinated, evidence is preserved, and operational boundaries remain intact.

Industry insight: Multi-agent coordination is successful only when the complete system produces a verified result. Three agents agreeing on an answer is not equivalent to proving that a production incident has been resolved.

↑ Back to contents

12. Hands-On Mini Project

Design a Simulated Multi-Agent Incident Response System

Objective: Build or document an orchestration system that coordinates specialized agents using fictional incident data.

No real production infrastructure or deployment permissions are required.

Step 1: Define the goal

Investigate a fictional payment service incident and produce an evidence-supported recommendation or escalation.

Step 2: Define three roles

Diagnostic agent.
Code-analysis agent.
Verification agent.

Step 3: Create fictional evidence

Prepare incident logs, performance metrics, and a small list of recent code changes.

Step 4: Define handoff schemas

Include task ID, objective, evidence references, uncertainties, permitted actions, and expected outputs.

Step 5: Implement orchestration

Coordinate diagnostic investigation, code analysis, verification, and final integration.

Step 6: Add policy boundaries

Permit read-only investigation and simulated testing. Prohibit actual production changes.

Step 7: Introduce failures

Missing evidence in a handoff.
Conflicting agent findings.
Tool timeout.
Duplicate task assignment.
Unauthorized deployment request.

Step 8: Evaluate

Metric	Evaluation question
Task completion rate	Did the system reach an appropriate verified outcome or escalation?
Handoff validity	Were required fields and evidence references present?
Coordination accuracy	Were tasks assigned to appropriate components?
Authorization violations	Did any prohibited operation execute?
Conflict handling	Were contradictory findings handled explicitly?
Recovery success	Could the system resolve or safely escalate failures?
Execution efficiency	How many model and tool calls were required?
End-to-end latency	How long did the complete process take?
Evidence fidelity	Were final conclusions supported by the available evidence?

Step 9: Compare with a simpler baseline

Design a single-agent or deterministic workflow for the same task.

Compare task success, cost, latency, failure handling, and implementation complexity.

Success criteria

Every agent has a defined responsibility.
Handoffs are validated.
Conflicting findings are not silently discarded.
No unauthorized operation executes.
All test runs terminate within defined limits.
The final report preserves evidence and uncertainty.
The design can be justified against a simpler alternative.

Reflection

Did specialization improve the result? Which coordination costs were introduced? When would a single-agent design be preferable?

↑ Back to contents

13. Connected Glossary Terms

Tool Calling (#013): Provides external capabilities to agents.
Workflows (#014): Define structured execution and process transitions.
AI Agents (#015): Perform model-guided tasks toward defined goals.
Prompt Engineering (#006): Helps specify agent responsibilities and instructions.
RAG (#004): Supplies relevant retrieved evidence.
Context Window (#005): Limits how much information can be passed to an agent in one model interaction.
Evals (#007): Tests component behavior and system-level outcomes.
Hallucination (#008): Can propagate unsupported claims through handoffs.
Grounding (#009): Connects findings to supporting evidence.
Guardrails (#010): Enforces permissions and execution restrictions.
Observability (#011): Makes multi-step execution traceable.
Human-in-the-Loop (#012): Provides approval and escalation mechanisms.

Previous lesson: #015 — AI Agents: Goal-Directed Systems That Use Tools and Make Decisions

Next lesson: #017 — Model Context Protocol (MCP): Standardizing Connections Between AI Applications and Tools.

↑ Back to contents

14. Living Update

Editorial baseline: November 2026

Agent orchestration continues to evolve through agent frameworks, durable execution systems, tool integration standards, and enterprise governance practices.

Areas to monitor include:

Structured agent-to-agent communication.
Agent interoperability protocols.
Model Context Protocol integration.
Durable orchestration and recovery.
Distributed tracing across agent boundaries.
Agent identity and delegated authorization.
Human approval and intervention patterns.
Multi-agent evaluation benchmarks.
Coordination cost and latency optimization.
Security risks arising from cross-agent instructions.

Living Glossary note: Revisit this entry as agent communication standards, orchestration frameworks, and evaluation practices mature.

↑ Back to contents

15. Key Takeaway

Agent orchestration turns individual agent capabilities into a coordinated process by managing responsibilities, handoffs, shared state, controls, and verified outcomes.

The aim is not to build the largest possible team of AI agents.

The aim is to design the simplest coordinated system that accomplishes the task reliably.

Think like an AI engineer: If three agents produce three different answers, what mechanism determines the next action—and what evidence is needed before the system can claim success?

↑ Back to contents

16. References and Further Reading

↑ Back to contents

AI Engineering Living Glossary #015 — AI Agents: Goal-Directed Systems That Use Tools and Make Decisions

AI ENGINEERING | LIVING GLOSSARY | #015

AI AGENTS

Goal-Directed Systems That Use Tools and Make Decisions

Chapter 3: Connecting AI to Tools, Workflows, and Agents

Lesson 15 of 24

AIYING — Accomplishing with AI

An employee contacts the IT help desk:

"My VPN stopped working after yesterday's software update. Please investigate and help me restore access."

A conventional chatbot can suggest troubleshooting steps. A tool-enabled assistant can retrieve a diagnostic result. A predefined workflow can execute a fixed troubleshooting sequence.

But what if the next diagnostic step depends on what the system discovers?

Perhaps the VPN service is running but authentication is failing. Perhaps the client version is outdated. Perhaps the problem affects multiple employees and requires escalation.

An AI agent can select among permitted diagnostic actions, examine observations, and adapt its next step toward the user's goal.

This is the promise of AI agents—and also the source of important engineering challenges.

Engineering insight: An AI agent combines goal-directed, model-guided decisions with tools and feedback. Its autonomy must be bounded by permissions, policies, budgets, and verifiable stopping conditions.

1. What Is an AI Agent?

An AI agent is a software system that uses an AI model to help select actions toward a goal, often interacting with tools and its environment over multiple steps.

Unlike a single model response, an agent may repeatedly:

Interpret the current goal.
Observe relevant information.
Select an appropriate next action.
Invoke an available tool.
Examine the tool result.
Update its working state.
Continue, stop, or escalate.

There is no single universally accepted technical definition of an AI agent. Implementations vary in autonomy, planning, memory, tool access, and execution control.

In this lesson, the defining engineering feature is model-guided selection of actions toward a goal, within a host application's control boundaries.

An agent need not be fully autonomous. It may be limited to read-only tools, a fixed number of steps, or a narrowly defined task.

It also need not maintain permanent memory or generate an explicit long-term plan.

↑ Back to contents

2. Why AI Agents Matter

Some tasks cannot be reduced easily to a single response or a fully predetermined sequence.

They involve uncertainty, changing information, and choices that depend on intermediate results.

Examples include:

Investigating an unfamiliar software error.
Researching a question across multiple sources.
Diagnosing a customer support issue.
Analyzing a repository before proposing code changes.
Investigating an operational exception.

Agents can provide flexibility when the correct next step cannot be specified completely in advance.

However, flexibility introduces new risks: unexpected tool selection, repeated actions, uncontrolled costs, unreliable conclusions, and unauthorized changes.

Practical principle: Use agentic behavior when adaptive decision-making provides meaningful value. Do not introduce autonomy merely because an agent framework is available.

↑ Back to contents

3. Types and Patterns of AI Agents

A. Tool-using agent

Selects from permitted tools to gather information or perform operations.

B. Research agent

Searches, retrieves, compares, and synthesizes information, potentially through several iterations.

C. Coding agent

Inspects code, proposes changes, runs permitted tests, and evaluates results within an engineering environment.

D. Diagnostic agent

Investigates symptoms, chooses diagnostic checks, interprets observations, and recommends or performs permitted remediation.

E. Human-supervised agent

Can investigate independently but requires human approval for specified actions.

F. Multi-agent system

Coordinates multiple model-driven components, potentially with specialized responsibilities.

These patterns overlap. A coding agent may also be a tool-using, human-supervised agent.

Multi-agent designs introduce additional coordination and evaluation complexity. They are not automatically better than a single agent.

↑ Back to contents

4. The Agent Execution Loop

A common agent design follows a repeated cycle:

       User Goal
           |
           v
   Interpret Current State
           |
           v
   Select Next Action
           |
           v
   Validate and Authorize
           |
           v
   Execute Permitted Tool
           |
           v
   Observe Tool Result
           |
           v
   Evaluate Progress
           |
           v
    Goal Achieved?
      /       \
    Yes        No
     |          |
   Verify     Continue,
   and Stop   Escalate,
              or Stop

The model may choose actions, but the application must control actual execution.

The loop must have explicit termination conditions, such as:

The task has been verified as complete.
The maximum number of steps has been reached.
A time or cost budget has been exhausted.
An authorization boundary has been reached.
The task requires human intervention.
The system detects insufficient evidence or repeated failure.

Important: An agent's claim that a task is complete is not necessarily proof that the task was completed.

↑ Back to contents

5. AI Agents vs Chatbots, Tools, and Workflows

Concept	Primary characteristic	Example
Chatbot	Conversational interaction.	Explains how VPN authentication works.
Tool Calling	Requests an external capability.	Queries VPN client version.
Workflow	Coordinates a designed process.	Runs a fixed set of VPN checks.
AI Agent	Selects actions dynamically toward a goal.	Chooses the next diagnostic based on observed results.

These are not mutually exclusive categories.

An agent may be accessed through a chatbot. It may invoke tools and operate inside a workflow.

The key distinction is the amount of discretion the model has in choosing subsequent actions.

When a process is predictable, a deterministic workflow is often simpler to test and operate.

When the process requires adaptive investigation, a bounded agent may be more appropriate.

↑ Back to contents

6. AI Agent Architecture

A practical agent system may contain the following components.

1. Goal and task specification

Defines what the agent is trying to accomplish and the conditions for success.

2. AI model

Interprets information and proposes next actions.

3. Tool registry

Defines available functions, interfaces, and permitted capabilities.

4. Execution controller

Validates tool requests, enforces permissions, and manages execution.

5. Working state

Tracks the current task, observations, decisions, and relevant intermediate results.

6. Optional memory

Retains selected information across steps or sessions where appropriate and permitted.

7. Policy and approval layer

Restricts sensitive operations and routes designated actions to human reviewers.

8. Evaluation and verification

Checks whether the agent's actions and final results meet defined criteria.

9. Observability

Records tool calls, failures, latency, costs, and relevant execution events.

User / Task
     |
     v
Agent Controller
     |
     +--> AI Model
     |      |
     |      v
     |   Proposed Action
     |      |
     +--> Validation and Policy
     |      |
     +--> Authorized Tool
     |      |
     +--> Observation
     |      |
     +--> State Update
     |
     +--> Continue / Stop / Escalate
     |
     v
Verified Result

Memory and state are related but different. An agent can maintain task state without storing long-term personal information.

↑ Back to contents

7. Security, Reliability, and Control

Agentic systems can make multiple decisions and tool requests, sometimes using information obtained from untrusted sources.

This makes defense-in-depth especially important.

Least-privilege tools

Give the agent only the capabilities needed for its task.

Independent authorization

Check user identity, permissions, and business policy outside the model.

Prompt injection resistance

Treat web pages, retrieved documents, tickets, and tool responses as untrusted data rather than instructions.

Tool argument validation

Validate schemas, identifiers, ranges, and operational constraints.

Human approval

Require approval before specified consequential actions such as changing access rights or deleting resources.

Execution budgets

Limit steps, runtime, tool usage, and expenditure.

Safe stopping behavior

Stop or escalate when evidence is insufficient, authorization fails, or repeated attempts do not make progress.

Idempotency and recovery

Prevent duplicate effects and handle interrupted operations safely.

Result verification

Confirm outcomes through authoritative system evidence rather than trusting a model's summary alone.

Audit and monitoring

Capture appropriate records of tool selection, execution, decisions, and failures while protecting sensitive information.

Security principle: The agent can recommend what to do next. Trusted application components must determine what it is allowed to do.

↑ Back to contents

8. Industry Applications

Domain	Potential agent task	Important control
IT support	Investigate incidents using diagnostic tools.	Restrict system changes and credential access.
Software engineering	Inspect code, propose patches, run tests.	Sandbox execution and require review before deployment.
Customer service	Investigate complex support requests.	Protect customer records and constrain transactions.
Business research	Collect and compare information.	Verify sources and preserve provenance.
Operations	Investigate inventory or process exceptions.	Control operational changes.
Security operations	Assist with alert triage and evidence gathering.	Require authorization for containment actions.
Data analysis	Explore datasets and generate findings.	Limit data access and verify calculations.

The appropriate autonomy level varies with the task's reversibility, uncertainty, potential harm, and regulatory requirements.

↑ Back to contents

9. Engineering Design Decisions

What exact goal should the agent pursue?
What evidence demonstrates success?
Does the task truly require adaptive action selection?
Would a deterministic workflow be sufficient?
Which tools should the agent access?
Which tools are read-only and which change state?
How are tool requests validated and authorized?
What information may the agent retain?
What are the maximum step, time, and cost budgets?
When must the agent ask for human approval?
How will the system detect loops and lack of progress?
How will prompt injection be mitigated?
How will tool failures and uncertain results be handled?
How will final outcomes be independently verified?
How will performance be evaluated before deployment?

Design principle: Choose the minimum autonomy necessary to accomplish the task effectively.

↑ Back to contents

10. Practical Industry Case: The VPN Support Agent

Scenario: An employee reports that VPN access stopped working after a software update.

The organization deploys a fictional AI support agent with access to approved read-only diagnostics and internal troubleshooting documentation.

Available tools

get_client_version — retrieves the installed VPN client version.
check_service_status — reports whether the local VPN service is running.
get_recent_errors — retrieves approved diagnostic error codes.
search_knowledge_base — searches authorized support documentation.
create_support_ticket — creates an escalation ticket after required checks.

These are illustrative interfaces, not claims about a particular commercial VPN product.

Illustrative investigation

The agent receives the user's goal: restore VPN connectivity.
It checks the installed VPN client version.
The result indicates an older client version.
The agent searches the approved knowledge base for compatibility information.
The documentation identifies a relevant compatibility issue.
The agent retrieves recent diagnostic errors.
The errors are consistent with the documented issue.
The agent recommends the organization's approved remediation steps.
If the issue remains unresolved, it prepares an escalation ticket with supporting evidence.
The application verifies ticket creation before confirming escalation.

What makes this agentic?

The next diagnostic operation depends on observations. The model selects among permitted tools instead of following one fixed sequence for every incident.

What must the agent not do?

Request or expose the employee's password.
Disable security controls.
Modify access permissions without authorization.
Install software through an unapproved mechanism.
Claim the VPN is fixed without evidence.
Continue indefinitely when diagnostics fail.

Possible outcomes

Resolved: The employee follows approved remediation, and a permitted connectivity check confirms restoration.

Escalated: The agent cannot safely resolve the issue and creates a verified support ticket.

Blocked: Required permissions or diagnostic information are unavailable, so the agent stops and explains the limitation.

Industry insight: A useful support agent does not need unrestricted system access. It needs well-defined goals, trustworthy observations, appropriate tools, and controlled escalation.

↑ Back to contents

11. Hands-On Mini Project

Design a Simulated IT Support Agent

Objective: Create a bounded agent that investigates fictional VPN incidents using simulated diagnostic tools.

No real corporate systems, credentials, or administrative permissions are required.

Step 1: Define the goal

Investigate the incident and either identify a supported next step or escalate appropriately.

Step 2: Create fictional incidents

Outdated VPN client.
VPN service stopped.
Authentication failure requiring human support.
Unknown error code.
Diagnostic service unavailable.

Step 3: Define tools

Create simulated versions of get_client_version, check_service_status, get_recent_errors, and search_knowledge_base.

Step 4: Define permissions

Allow read-only diagnostics. Prohibit credential access and administrative changes.

Step 5: Implement the agent loop

For each incident, let the model choose a diagnostic tool, inspect the result, and select the next permitted action.

Step 6: Add stopping conditions

Limit the simulation to five tool calls per incident. Stop on verified completion, prohibited actions, repeated failures, or exhausted budget.

Step 7: Test failures

Simulate an invalid tool request, unavailable service, misleading knowledge-base text, and repeated selection of the same unsuccessful action.

Step 8: Evaluate

Metric	Evaluation question
Task success	Did the agent identify an appropriate resolution or escalation?
Tool selection accuracy	Were selected tools relevant to the observed state?
Authorization violations	Did any prohibited operation execute?
Unsupported claims	Did the agent assert results not supported by tool evidence?
Loop termination	Did execution stop within the defined limits?
Escalation quality	Were unresolved cases routed appropriately with evidence?
Tool efficiency	How many calls were needed for a useful outcome?
Failure handling	Did the system recover, stop, or escalate appropriately?

Success criteria

No unauthorized tool operation executes.
Every test run terminates within the defined budget.
Final recommendations are supported by simulated evidence.
Unresolved incidents are escalated rather than falsely reported as fixed.
Execution traces explain which tools were selected and why.

Reflection

Where did dynamic action selection help? Which parts could be replaced with a simpler workflow? What additional controls would production deployment require?

↑ Back to contents

12. Connected Glossary Terms

Tool Calling (#013): Gives agents access to external capabilities.
Workflows (#014): Can coordinate and constrain agent execution.
Prompt Engineering (#006): Helps specify goals, instructions, and boundaries.
RAG (#004): Supplies retrieved evidence for agent decisions.
Context Window (#005): Constrains how much task history and tool data can be included in a model call.
Evals (#007): Measures task success, tool selection, and safety behavior.
Hallucination (#008): Can affect conclusions and tool selection.
Grounding (#009): Connects agent claims to evidence.
Guardrails (#010): Restricts prohibited actions and unsafe outputs.
Observability (#011): Makes agent execution traceable.
Human-in-the-Loop (#012): Supports approval and escalation.

Previous lesson: #014 — Workflows: Orchestrating Multi-Step AI Processes

Next lesson: #016 — Agent Orchestration: Coordinating Agent Execution and Handoffs.

↑ Back to contents

13. Living Update

Editorial baseline: November 2026

AI agent engineering is evolving across model capabilities, orchestration frameworks, tool integration, and enterprise control systems.

Important areas to monitor include:

Agent evaluation benchmarks and task-completion measurement.
Secure tool access and permission enforcement.
Model Context Protocol and tool interoperability.
Agent observability and execution tracing.
Human approval and intervention patterns.
Durable execution and safe recovery.
Memory design, privacy, and retention controls.
Multi-agent coordination and handoffs.
Cost and latency management.
Prompt injection and untrusted tool-result handling.

Living Glossary note: Update this entry as agent frameworks, evaluation methods, and security practices mature.

↑ Back to contents

14. Key Takeaway

An AI agent uses model-guided decisions and external capabilities to pursue a goal. Reliable agents require bounded autonomy, independent controls, and verified outcomes.

More autonomy is not automatically better engineering.

The aim is to give the system enough flexibility to accomplish useful work while preserving authorization, accountability, recoverability, and human control.

Think like an AI engineer: What is the minimum freedom this agent needs to complete its task—and how will you know when it should stop?

↑ Back to contents

15. References and Further Reading

↑ Back to contents

AI Engineering Living Glossary #014 — Workflows: Orchestrating Multi-Step AI Processes

AI ENGINEERING | LIVING GLOSSARY | #014

WORKFLOWS

Orchestrating Multi-Step AI Processes

Chapter 3: Connecting AI to Tools, Workflows, and Agents

Lesson 14 of 24

AIYING — Accomplishing with AI

A customer contacts an online retailer:

"I returned my headphones last week. Please check whether I am eligible for a refund and process it if everything is in order."

An AI assistant might understand the request. It might even have access to a refund-processing tool.

But should it immediately issue the refund?

Not necessarily.

The retailer may need to verify the customer's identity, retrieve the original order, confirm the returned item was received, check the applicable policy, determine eligibility, calculate the refundable amount, obtain approval when required, issue the refund through an authorized payment service, and communicate the outcome.

These are connected steps with dependencies, decision points, and potential failures.

The engineering challenge is no longer just calling a tool. It is coordinating the complete process.

That is the purpose of a workflow.

Engineering insight: A workflow turns individual model calls, tool operations, rules, and human decisions into a coordinated process with defined outcomes and controls.

1. What Is an AI Workflow?

An AI workflow is a designed sequence or network of operations that coordinates AI model interactions, software tools, business rules, data, and sometimes human decisions to accomplish a defined task or outcome.

Not every step must involve AI.

A workflow may combine:

Deterministic software operations.
Language model calls.
External API calls.
Data retrieval.
Conditional branching.
Parallel tasks.
Human review or approval.
State persistence.
Error handling and recovery.

For example, an invoice-processing workflow might extract information from a document using AI, validate totals using conventional software, check supplier records through an API, and route exceptions to a finance officer.

Important distinction: A workflow is not necessarily autonomous. Its sequence and decision boundaries can be explicitly designed and controlled by engineers.

↑ Back to contents

2. Why Workflows Matter

Many business tasks cannot be completed reliably through a single prompt or tool call.

They require several coordinated operations.

Single AI interaction	Engineered workflow
Produces one response.	Coordinates a complete process.
May operate on limited information.	Can retrieve and validate required information.
May not track progress across stages.	Can maintain explicit process state.
May leave execution details implicit.	Defines execution responsibilities and transitions.
May be difficult to recover after failure.	Can include checkpoints, retries, and recovery paths.
Often measures response quality.	Can measure end-to-end task outcomes.

Workflows make it possible to engineer repeatability, accountability, and operational control around AI capabilities.

They do not guarantee reliability automatically. Reliability depends on workflow design, implementation, testing, and monitoring.

↑ Back to contents

3. Common AI Workflow Patterns

A. Sequential workflow

Each step runs after the previous step completes.

Input → Retrieve → Analyze → Draft → Validate → Output

Useful when later operations depend on earlier results.

B. Conditional workflow

The process follows different paths based on explicit conditions.

Check Eligibility
       |
       v
   Eligible?
    /     \
  Yes      No
   |        |
Refund    Explain
Review    Rejection

C. Parallel workflow

Independent operations execute concurrently, followed by synchronization.

           Request
          /   |   \
         v    v    v
      Check  Check  Check
      Order  Policy Payment
         \    |    /
          v   v   v
         Combine Results

Parallelism can reduce latency but introduces coordination and partial-failure challenges.

D. Human approval workflow

A designated person reviews or authorizes a consequential step.

AI Assessment
      |
      v
Human Approval
   /       \
Approve   Reject
  |         |
Execute   Stop

E. Iterative workflow

A result is evaluated and revised until a defined stopping condition is met.

Generate → Evaluate → Revise
               |         |
               +---------+
           Stop when criteria
           or limits are met

Iterations require explicit limits on time, cost, and attempts.

F. Event-driven workflow

A business event triggers a process.

Examples include a returned shipment arriving, a new support ticket being created, or a document being uploaded.

↑ Back to contents

4. How an AI Workflow Works

A typical workflow moves through a set of states and operations.

Business Request / Trigger
           |
           v
Identify Workflow and Inputs
           |
           v
Validate Request and Identity
           |
           v
Retrieve Required Information
           |
           v
AI Analysis (where useful)
           |
           v
Business Rules and Checks
           |
           v
Decision / Approval Gate
           |
           v
Authorized Tool Execution
           |
           v
Verify Actual Outcome
           |
           v
Update State and Records
           |
           v
Notify User / Complete

The workflow controller determines which step runs next.

Some decisions can be made by deterministic rules. Others may involve model-generated classifications or recommendations.

Consequential actions should not rely solely on unverified model output.

In a production application, execution status should be tracked independently of the model's conversational response.

↑ Back to contents

5. Workflows vs Tool Calling and Agents

Concept	Main responsibility	Example
Tool Calling	Request an external capability.	Retrieve an order record.
Workflow	Coordinate multiple operations and transitions.	Verify a return, assess eligibility, approve and refund.
AI Agent	Use model-guided decisions to pursue a goal, often with tools.	Investigate a customer issue and select permitted next steps.
Orchestrator	Manage execution, dependencies, and state.	Schedule the next workflow step after approval.

These categories are related rather than mutually exclusive.

An agent can operate within a workflow. A workflow can invoke agents. Both can use tool calling.

Engineering distinction: In a tightly controlled workflow, the developer specifies the main process structure. In a more agentic design, a model may have greater discretion over how to pursue a goal.

The appropriate choice depends on task uncertainty, risk, auditability, and required flexibility.

↑ Back to contents

6. Workflow Architecture

A production workflow typically requires more than a sequence of prompts.

1. Trigger layer

Receives a request, event, or scheduled instruction.

2. Orchestration layer

Determines which operations run, in what order, and under what conditions.

3. AI processing layer

Performs tasks such as classification, extraction, summarization, or language generation.

4. Tool integration layer

Connects to external APIs, databases, and enterprise services.

5. Business rules layer

Applies deterministic constraints, eligibility rules, and operational policies.

6. Human oversight layer

Routes designated cases to authorized reviewers.

7. State and persistence layer

Records progress, intermediate results, approvals, and execution outcomes.

8. Observability layer

Captures traces, errors, timing, and relevant quality indicators.

Architecture principle: Keep the workflow's authoritative state and business controls outside the language model. The model can contribute intelligence without becoming the sole source of operational truth.

↑ Back to contents

7. Reliability, State, and Recovery

Real workflows encounter failures.

External APIs time out. Model outputs fail validation. Human reviewers become unavailable. Systems may restart midway through a process.

Reliable workflow design must anticipate these situations.

State management

Track states such as RECEIVED, VALIDATING, AWAITING_APPROVAL, PROCESSING, COMPLETED, and FAILED.

Retries

Retry transient failures only when safe. Use bounded attempts and appropriate delays.

Idempotency

Prevent duplicate effects when an operation is repeated, especially for refunds, payments, and ticket creation.

Timeouts

Define how long a step may run and what happens when it exceeds that limit.

Compensation

Where possible, design compensating actions for completed steps that cannot simply be rolled back.

Human escalation

Route uncertain, high-risk, or repeatedly failing cases to an authorized person.

Auditability

Record important decisions, approvals, and outcomes using appropriate access controls and retention policies.

End-to-end verification

Verify that the business outcome occurred, not merely that a model or API reported an intermediate success.

Important: A refund API timeout does not prove that the refund failed. The application may need to query transaction status before attempting the operation again.

↑ Back to contents

8. Industry Applications

Industry	Example AI workflow
Retail	Return verification, refund eligibility, approval, payment, notification.
Banking	Document intake, data extraction, validation, risk checks, human decision.
Healthcare administration	Document intake, classification, verification, and authorized routing.
Human resources	Employee request, policy lookup, approval, record update.
Software engineering	Requirement analysis, code generation, tests, review, deployment gate.
Supply chain	Inventory monitoring, exception detection, replenishment recommendation.
Customer support	Issue classification, account lookup, resolution, escalation, closure.
Knowledge management	Document ingestion, extraction, indexing, review, publication.

Workflows are especially useful when tasks have dependencies, approvals, measurable outcomes, or operational consequences.

↑ Back to contents

9. Engineering Design Decisions

Before building an AI workflow, an engineer should answer:

What is the precise business outcome?
What triggers the workflow?
Which steps are deterministic?
Which steps genuinely benefit from AI?
What data and tools are required?
What are the dependencies between steps?
Which operations can run in parallel?
What states must be persisted?
Which actions require authorization or human approval?
How will errors, retries, and timeouts be handled?
How will duplicate operations be prevented?
What is the maximum acceptable cost and latency?
How will sensitive data be protected?
How will completion be verified?
What evaluation evidence demonstrates reliability?

The goal is not to insert AI into every step. It is to design the simplest process that reliably accomplishes the intended task.

↑ Back to contents

10. Practical Industry Case: The Refund That Needed a Workflow

Scenario: A customer requests a refund for returned headphones.

Assume a fictional retail company with a policy requiring confirmation of receipt, eligibility checks, and approval for specified exceptions.

Workflow design

Receive request: Record the refund request and create a workflow identifier.
Authenticate: Verify the customer and access permissions.
Retrieve order: Query the order management system.
Check return: Confirm the item was received and inspected.
Retrieve policy: Obtain the applicable refund rules.
Assess eligibility: Apply business rules; use AI for interpretation only where appropriate.
Handle exceptions: Route uncertain cases to an authorized reviewer.
Calculate amount: Use validated transaction and policy data.
Request approval: Apply any required authorization gate.
Initiate refund: Invoke the payment service with an idempotency key.
Verify transaction: Confirm the payment service's authoritative status.
Update records: Record the verified outcome.
Notify customer: Communicate the actual refund status.
Close workflow: Mark completion only after required checks succeed.

Failure scenario A: Missing return confirmation

The workflow pauses or requests additional information rather than issuing an unsupported refund.

Failure scenario B: Policy exception

The workflow routes the case to a human reviewer with the relevant evidence.

Failure scenario C: Payment service timeout

The workflow checks transaction status before deciding whether a retry is safe.

Failure scenario D: Customer notification fails

The workflow records the refund outcome separately from notification delivery and retries or escalates the notification step.

What this case teaches

A successful AI workflow is not defined by the number of model calls it makes.

It is defined by whether it coordinates the right operations, enforces the necessary controls, and produces a verified business outcome.

Industry insight: A workflow must distinguish between "AI recommended a refund," "the refund was authorized," "the refund was submitted," and "the refund was confirmed." These are different states.

↑ Back to contents

11. Hands-On Mini Project

Build a Simulated AI Refund Workflow

Objective: Design and test a multi-step AI workflow using fictional orders and refunds. No real payments or customer accounts are required.

Step 1: Define the workflow

Create a flowchart showing intake, validation, retrieval, eligibility, approval, execution, verification, and notification.

Step 2: Create sample data

Prepare five fictional refund requests:

Eligible return with confirmed receipt.
Return outside the permitted window.
Missing return confirmation.
Refund requiring manager approval.
Previously refunded order.

Step 3: Assign responsibilities

Mark each operation as one of:

AI model task.
Deterministic rule.
External tool or API.
Human decision.

Step 4: Define workflow states

Use states such as RECEIVED, VALIDATED, AWAITING_APPROVAL, REFUND_PENDING, COMPLETED, and FAILED.

Step 5: Simulate execution

Run all five requests through the workflow and record each transition.

Step 6: Inject failures

Simulate a timeout, invalid AI output, missing data, and duplicate refund request.

Step 7: Evaluate results

Metric	What to measure
Completion rate	Percentage of valid requests reaching a verified final state.
Routing accuracy	Correct selection of eligibility, exception, and approval paths.
Unauthorized execution count	Actions executed without required permissions or approvals.
Duplicate refund count	Repeated payment effects for the same request.
Recovery success	Ability to resume or safely resolve interrupted workflows.
End-to-end latency	Time from request to verified outcome.
State accuracy	Whether recorded states match actual operations.

Success criteria

Every test request follows an appropriate path.
No unauthorized refund is executed.
No duplicate refund is issued.
Missing information leads to a controlled pause or escalation.
Failed steps do not falsely mark the workflow complete.
Final status reflects the verified outcome.

Reflection

Which steps genuinely required AI? Which were better handled by conventional software? What would you change before production deployment?

↑ Back to contents

12. Connected Glossary Terms

Tool Calling (#013): Provides individual external capabilities used by workflows.
Prompt Engineering (#006): Shapes model behavior within workflow steps.
RAG (#004): Supplies retrieved information for particular workflow operations.
Context Window (#005): Limits the information available to each model call.
Evals (#007): Tests both individual steps and end-to-end outcomes.
Hallucination (#008): Must be detected and controlled when model outputs influence decisions.
Grounding (#009): Connects model responses to verified source information.
Guardrails (#010): Enforces restrictions at workflow boundaries.
Observability (#011): Supports tracing, monitoring, and diagnosis.
Human-in-the-Loop (#012): Introduces review and approval gates.

Previous lesson: #013 — Tool Calling: Connecting AI Models to External Capabilities

Next lesson: #015 — AI Agents: Goal-Directed Systems That Use Tools and Make Decisions.

↑ Back to contents

13. Living Update

Editorial baseline: November 2026

AI workflow engineering continues to evolve as organizations connect models with operational systems.

Areas to monitor include:

Durable execution and workflow resumption.
Integration of model calls with deterministic orchestration.
Structured outputs and schema validation.
Human approval and intervention mechanisms.
Multi-agent workflow coordination.
Workflow tracing and evaluation.
Security boundaries for tools and data.
Cost, latency, and reliability optimization.
Standards and interoperability for tool integration.

Living Glossary note: Revisit this entry as orchestration technologies, agent frameworks, and enterprise operational practices evolve.

↑ Back to contents

14. Key Takeaway

A tool performs an operation. A workflow coordinates operations into a controlled, measurable, and recoverable process.

The value of an AI workflow lies not in its complexity, but in its ability to reliably achieve a meaningful business outcome.

Think like an AI engineer: If the process fails halfway through, can your system determine what already happened, what remains to be done, and how to recover safely?

↑ Back to contents

15. References and Further Reading

↑ Back to contents