AI ENGINEERING — ACTIVE CONCEPTS & PRACTICES | #012
Observability and Tracing: Understanding What AI Systems Actually Do
Chapter 2: Engineering AI Applications and Agents
AIYING — Accomplishing with AI
The Big Idea: Observability makes the internal behavior of AI applications understandable through telemetry such as traces, metrics, logs, and execution events.
Tracing reveals how individual steps contribute to an end-to-end AI workflow.
Engineering question: How can engineers diagnose failures, understand costs, and improve reliability when AI systems involve multiple model calls, retrieval operations, tools, and decisions?
Contents
- Why Observability Matters
- UNDERSTAND — The Concept
- CONNECT — AI Engineering Relationships
- Observability Architecture
- APPLY — Enterprise Procurement Agent
- Worked Example: Diagnosing Slow Execution
- Engineering Decisions and Trade-offs
- Active Industry Practices
- ACCOMPLISH — Practical Exercise
- Student and Practitioner Project
- Check Your Understanding
- Active Connections
- References and Further Reading
1. Why Observability Matters
Imagine an enterprise AI agent that retrieves purchase orders, checks inventory, recommends recovery actions, and prepares approval requests.
Yesterday, the workflow completed in 12 seconds. Today, the same type of request takes 45 seconds.
The final answer still looks correct.
Without execution telemetry, the engineering team cannot determine whether the delay came from retrieval, model inference, tool execution, retries, or external systems.
Observability makes these internal operations visible.
2. UNDERSTAND — The Concept
What is observability?
Observability is the ability to understand the internal state and behavior of a system using information it emits during operation.
In AI engineering, this includes model calls, prompts and responses where permitted, retrieval operations, tool invocations, workflow transitions, latency, errors, and resource consumption.
Three foundational telemetry signals
| Signal | What It Reveals | Example |
|---|---|---|
| Traces | How a request progresses across operations | Model → Retrieval → Tool → Response |
| Metrics | Aggregated quantitative behavior | Latency, error rate, token usage |
| Logs | Recorded events and diagnostic details | Tool timeout, validation failure |
What is tracing?
A trace represents the execution path of an operation across components.
Individual operations are often represented as spans.
Each span may record its name, start time, duration, status, attributes, and relationship to other spans.
Example trace
Trace: Procurement Request | +-- Span 1: Parse Request | +-- Span 2: Retrieve Purchase Order | +-- Span 3: Check Inventory | +-- Span 4: Generate Recommendation | +-- Span 5: Validate Recommendation | +-- Span 6: Prepare Approval Request
Tracing helps engineers understand which operations occurred, their order, and where time or failures accumulated.
Observability versus evaluation
Observability: What happened during execution?
Evaluation: Did the system perform correctly against defined criteria?
Observability supplies evidence that evaluation can use, but traces alone do not establish whether a result was correct.
3. CONNECT — AI Engineering Relationships
| Concept | Connection to Observability |
|---|---|
| Context Engineering | Tracks context assembly and token usage |
| Retrieval Engineering | Measures retrieval latency and failures |
| Structured Outputs | Records validation failures |
| Tool Calling | Tracks tool requests, results, and errors |
| MCP | Monitors interactions with external servers |
| Agent Orchestration | Tracks workflow steps and state transitions |
| Agent Evaluation | Uses traces as evidence for quality assessment |
| Token Economics | Measures token consumption and estimated cost |
| AI Governance | Supports auditability and accountability |
Active connection: Observability connects AI development, testing, operations, and governance through recorded evidence of system behavior.
4. Observability Architecture
↓
AI Application / Agent↓
Instrumented Model, Retrieval, and Tool Operations↓
Telemetry Collection↓
Traces + Metrics + Logs↓
Monitoring, Analysis, and Alerts↓
Diagnosis, Evaluation, and ImprovementInstrumentation
Instrumentation adds the mechanisms that generate telemetry from application operations.
Trace correlation
Related spans share trace identifiers so that operations across services can be analyzed together.
Metrics collection
Systems aggregate measurements such as request counts, latency distributions, error rates, and token consumption.
Logging
Structured logs capture relevant events, errors, and operational details.
Dashboards and alerts
Dashboards display trends, while alerts notify operators when defined thresholds or conditions are met.
Security and privacy
Observability systems must avoid unnecessary exposure of sensitive prompts, personal information, credentials, and confidential tool results.
5. APPLY — Enterprise Procurement Agent
Scenario: A Slow Procurement Workflow
A procurement AI agent performs the following operations:
- Interprets the user request.
- Retrieves purchase-order information.
- Checks inventory.
- Generates a recovery recommendation.
- Validates the recommendation.
- Prepares an approval request.
The workflow is taking substantially longer than expected.
What should engineers inspect?
| Component | Observable Signal |
|---|---|
| Model inference | Latency, token usage, errors |
| Retrieval | Query duration, result count, failures |
| Inventory API | Response time, timeout rate, retries |
| Workflow controller | Step count, state transitions, loops |
| Validation | Schema and business-rule failures |
| Approval service | Request creation and delivery status |
With distributed tracing, engineers can identify which operation is contributing most to latency.
With metrics, they can determine whether the problem affects one request or a larger population.
With logs, they can investigate specific error events.
6. Worked Example: Diagnosing Slow Execution
Consider the following fictional trace of a sequential AI workflow.
| Operation | Duration |
|---|---|
| Request interpretation | 2 seconds |
| Purchase-order retrieval | 3 seconds |
| Inventory API call | 25 seconds |
| Recommendation generation | 10 seconds |
| Validation and response | 5 seconds |
| Total | 45 seconds |
Latency contribution
Inventory API contribution = 25 / 45 × 100 = 55.6%
In this example, the inventory API accounts for approximately 56% of total execution time.
What should the engineer investigate?
- Is the inventory service responding slowly?
- Did the application retry the request?
- Was there a network timeout?
- Was the query unusually expensive?
- Is the latency problem persistent or isolated?
The trace identifies where time was spent, but additional evidence is needed to establish the underlying cause.
Example structured trace record
{
"trace_id": "trace-1048",
"span_name": "inventory_lookup",
"duration_ms": 25000,
"status": "ok",
"retry_count": 2,
"service": "inventory-api",
"workflow_id": "WF-1048"
}
This is an illustrative application record, not a complete OpenTelemetry span representation.
Engineering interpretation
The engineer now has evidence that the inventory lookup experienced retries and substantial latency.
The next step is to inspect the underlying API and retry policy rather than blindly changing the model or prompt.
7. Engineering Decisions and Trade-offs
| Decision | Benefit | Trade-off |
|---|---|---|
| Detailed tracing | Better diagnosis | Storage and processing overhead |
| Full prompt logging | Detailed debugging context | Privacy and confidentiality risks |
| Sampling | Reduced telemetry cost | Some traces may be unavailable |
| Real-time alerts | Faster incident response | False positives and alert fatigue |
| Token-level metrics | Better cost analysis | Instrumentation complexity |
| Long retention | Historical analysis | Storage and data governance burden |
Common observability mistakes
- Recording only final responses
- Failing to correlate tool calls with workflow traces
- Ignoring retry behavior
- Logging sensitive information unnecessarily
- Measuring averages without latency distributions
- Collecting telemetry without actionable dashboards
- Confusing operational success with answer correctness
Recommended engineering practices
- Instrument important workflow boundaries.
- Use consistent trace and span identifiers.
- Record model, retrieval, and tool latency.
- Capture token usage and estimated costs.
- Record relevant workflow state transitions.
- Use structured logs.
- Protect sensitive telemetry.
- Monitor error rates and latency percentiles.
- Connect traces with evaluation results.
- Define operational alerts and incident procedures.
8. Active Industry Practices
Established foundations
- Distributed tracing
- Structured logging
- Metrics and dashboards
- Latency and error monitoring
- Alerting and incident investigation
- Trace correlation across services
AI-specific practices
- Model-call tracing
- Token and cost tracking
- Retrieval and tool-call instrumentation
- Agent execution trajectory tracing
- Prompt and response metadata tracking
- Evaluation-linked traces
- Privacy-aware telemetry collection
Relevant technologies
Engineering teams may use OpenTelemetry, Jaeger, Grafana, LangSmith, MLflow, or other observability platforms according to application requirements.
OpenTelemetry provides vendor-neutral instrumentation concepts and conventions, while specialized AI observability platforms add workflow-oriented capabilities.
Specific generative AI semantic conventions and tooling support can vary by implementation and version.
Industry applications
Observability is important for production AI agents, RAG applications, enterprise assistants, AI-enabled software systems, workflow automation, and operational AI platforms.
Practice status: Established software observability discipline extended to generative AI and agentic systems.
Reviewed: November 2026.
9. ACCOMPLISH — Practical Exercise
Exercise: Diagnose an AI Workflow
Consider the following fictional telemetry:
Workflow duration: 45 seconds
Model calls: 4
Tool calls: 3
Inventory API duration: 25 seconds
Inventory retries: 2
Estimated execution cost: $0.18
Your tasks
- Identify the likely latency bottleneck.
- Calculate the inventory API's share of total execution time.
- Identify additional telemetry required for root-cause analysis.
- Recommend improvements to retry handling.
- Define three operational metrics.
- Propose two alerts.
- Explain what information should not be logged.
- Describe how evaluation results could be linked to traces.
Reusable AI-assisted prompt
Act as an AI systems observability engineer. Analyze this fictional agent workflow: Total duration: 45 seconds Model calls: 4 Tool calls: 3 Inventory API duration: 25 seconds Inventory retries: 2 Estimated cost: $0.18 Provide: 1. Initial performance diagnosis. 2. Latency contribution analysis. 3. Possible root causes. 4. Additional telemetry required. 5. Recommended metrics. 6. Trace and log requirements. 7. Alerting recommendations. 8. Privacy and security safeguards. 9. Improvement experiments. 10. Verification plan. Distinguish observed facts from hypotheses. Do not assume the root cause is proven.
Accomplishment: Produce an evidence-based diagnosis and observability improvement plan for an AI workflow.
10. Student and Practitioner Project
Mini Project: Build an AI Agent Observability Dashboard
Objective: Instrument a fictional AI workflow and create a basic monitoring dashboard.
Project requirements
- Define a multi-step fictional AI workflow.
- Instrument model and tool operations.
- Assign trace and span identifiers.
- Capture execution duration and status.
- Track token usage and estimated cost.
- Record tool errors and retries.
- Create a simple latency and error dashboard.
- Simulate a slow external API.
- Investigate the resulting trace.
- Recommend and test an improvement.
Suggested dashboard measures
| Metric | Purpose |
|---|---|
| Request count | Tracks workload volume |
| End-to-end latency | Measures user-visible processing time |
| Tool latency | Identifies slow external operations |
| Model latency | Measures inference time |
| Error rate | Tracks operational failures |
| Retry count | Identifies repeated operations |
| Token usage | Measures model consumption |
| Estimated cost | Tracks financial efficiency |
Project deliverable: A trace diagram, telemetry dataset, dashboard or dashboard design, incident analysis, and engineering improvement recommendation.
11. Check Your Understanding
- What is observability in AI engineering?
- How do traces, metrics, and logs differ?
- What is a span?
- Why are trace identifiers important?
- How does observability support agent evaluation?
- Why can logging complete prompts create privacy risks?
- How would you investigate an unexpectedly slow AI workflow?
- What distinguishes operational success from task correctness?
Reflection: If an AI workflow completes successfully but consumes three times its normal cost, should the system be considered healthy? Which additional measures would help you decide?
12. Active Connections
- #001 — Context Engineering
- #002 — Token Economics
- #003 — Embeddings in Practice
- #004 — Retrieval Engineering
- #005 — Chunking and Reranking
- #006 — Context Management
- #007 — Structured Outputs
- #008 — Tool Calling
- #009 — Model Context Protocol (MCP)
- #010 — Agent Orchestration
- #011 — Agent Evaluation
Coming next — #013: Guardrails and Policy Enforcement: Keeping AI Applications Within Operational Boundaries.
13. References and Further Reading
- OpenTelemetry — Official Documentation
- OpenTelemetry — Generative AI Semantic Conventions
- LangSmith — Observability Documentation
- Grafana — Documentation
- Jaeger — Distributed Tracing Documentation
Instrumentation libraries, semantic conventions, and AI observability tools evolve. Verify the latest documentation before production implementation.
Final Takeaway
AI Engineering — Active Concepts & Practices
Active Concept #012 — Observability and Tracing
AIYING — Accomplishing with AI
Understand → Connect → Apply → Accomplish
