AI Engineering Living Glossary #007 — Evals: How AI Engineers Know Their Systems Actually Work

AI ENGINEERING | LIVING GLOSSARY | #007

EVALS — AI EVALUATIONS

How AI Engineers Know Whether Their Systems Actually Work

Chapter 2: Engineering AI for Reliability

AIYING — Accomplishing with AI

An AI assistant performs brilliantly in a demonstration.

It answers questions confidently, retrieves documents, and generates professional responses.

But when employees begin using it, some answers are incorrect, some citations are misleading, and some requests take too long.

The demonstration showed what the system could do. It did not establish how reliably the system performs.

How can engineers distinguish an impressive demonstration from a dependable AI application?

The answer begins with Evals.

Engineering insight: Evals transform expectations about AI performance into measurable evidence. They help engineers identify failures, compare alternatives, and make informed deployment decisions.

Contents

1. What Are Evals?
2. Why Evals Matter
3. Traditional Testing vs AI Evals
4. Types of Evals
5. Important Evaluation Metrics
6. The Evaluation Workflow
7. Industry Applications
8. Engineering Decisions
9. Practical Industry Case
10. Try It Yourself
11. Connected Glossary Terms
12. Living Update
13. Key Takeaway
14. References

1. What Are Evals?

Evals is the commonly used abbreviation for evaluations in AI engineering.

AI evaluations are systematic processes for measuring the behavior, quality, and performance of AI models and applications against defined criteria.

An evaluation can examine:

• Whether an answer is factually correct
• Whether it addresses the user's request
• Whether it follows specified instructions
• Whether it is supported by available evidence
• Whether it respects safety and business constraints
• Whether the application completes the intended task
• How much time and money the operation requires

Evals may use automated checks, reference answers, human reviewers, model-based graders, or combinations of these methods.

They can measure a model, an individual component, or an entire application workflow.

Key distinction: A benchmark reports performance on a defined test. An evaluation program determines whether the system is suitable for the tasks and conditions that matter to its users.

↑ Back to contents

2. Why Evals Matter

Generative AI systems can produce fluent outputs even when the underlying information is incomplete or incorrect.

They may also behave differently across inputs, model versions, prompt revisions, and changing external data.

Consequently, engineers need more than successful examples.

Without systematic evals	With systematic evals
Performance is judged through demonstrations.	Performance is measured on representative cases.
Failures are discovered unpredictably.	Known failure categories can be tested.
Prompt changes rely on intuition.	Prompt versions can be compared.
Deployment decisions are subjective.	Decisions can use explicit acceptance criteria.
Quality may deteriorate unnoticed.	Monitoring can detect changes in measured behavior.

Evals do not eliminate failures. They provide evidence about known and tested risks, and help engineers discover new ones.

↑ Back to contents

3. Traditional Testing vs AI Evals

Traditional software testing often checks whether a system produces an expected result for a defined input.

For example:

Input: 2 + 3
Expected output: 5
Actual output: 5
Result: PASS

Generative AI outputs are frequently open-ended. Several different responses may be acceptable.

Consider a customer asking:

"Can I claim hotel expenses during a business trip?"

A useful response might take different forms while still meeting requirements for factual correctness, relevance, tone, and policy compliance.

Therefore, AI evaluations may combine:

• Exact-match checks for deterministic outputs
• Schema and format validation
• Semantic or rubric-based grading
• Evidence-grounding checks
• Human assessment
• Task-completion measurements

Traditional tests remain important. AI systems often need both conventional software tests and probabilistic or judgment-based evaluations.

↑ Back to contents

4. Types of Evals

A. Functional evaluations

Measure whether the application performs its intended task correctly.

B. Grounding and factuality evaluations

Check whether generated claims are supported by appropriate evidence and whether factual answers are correct.

C. Instruction-following evaluations

Check whether the model follows required formats, constraints, and task instructions.

D. Safety and security evaluations

Test behavior involving unsafe requests, sensitive information, prompt injection, and policy boundaries.

E. Retrieval evaluations

Measure whether a retrieval system finds relevant information, independently of whether the language model produces a good final answer.

F. Human evaluations

Use qualified reviewers to assess usefulness, accuracy, clarity, and other criteria that automated checks may miss.

G. Operational evaluations

Measure latency, cost, reliability, resource usage, and production performance.

H. End-to-end evaluations

Measure whether the complete application achieves the intended user outcome.

Engineering principle: Evaluate both the individual components and the end-to-end experience. A component can perform well while the overall workflow fails.

↑ Back to contents

5. Important Evaluation Metrics

Metric	What it measures
Accuracy	Proportion of evaluated cases answered correctly under defined criteria.
Precision	Proportion of predicted positive cases that are correct.
Recall	Proportion of actual positive cases successfully identified.
Faithfulness	Whether generated claims are supported by supplied evidence.
Relevance	How well an output addresses the requested task.
Task success rate	Proportion of tasks completed according to defined success criteria.
Format validity	Proportion of outputs meeting required schemas or structures.
Latency	Time required to produce a response or complete a task.
Cost per task	Resources or monetary expense associated with completing a task.

Metrics must match the application. A classification system, RAG assistant, and coding agent require different evaluation designs.

Scores from automated or model-based graders should be checked against human judgments and known examples. A numerical score is not automatically an objective measure of quality.

↑ Back to contents

6. The Evaluation Workflow

Define the Task
       |
       v
Specify Success Criteria
       |
       v
Prepare Representative Test Cases
       |
       v
Run the AI System
       |
       v
Grade and Measure Results
       |
       v
Analyze Failure Patterns
       |
       v
Improve the System
       |
       v
Retest and Monitor

A useful evaluation dataset should include more than easy examples.

• Typical user requests
• Ambiguous inputs
• Missing information
• Conflicting evidence
• Outdated documents
• Boundary and exception cases
• Adversarial or malicious inputs where relevant

Engineers should also maintain separate development and held-out evaluation cases to reduce the risk of optimizing only for familiar examples.

Whenever possible, record model versions, prompt versions, test data, scoring methods, and relevant configuration settings so that comparisons are reproducible.

↑ Back to contents

7. Industry Applications

AI application	What engineers evaluate
Customer support assistant	Correct answers, policy compliance, escalation, and satisfaction.
RAG knowledge assistant	Retrieval quality, answer support, citations, and completeness.
AI coding assistant	Test passing, functional correctness, maintainability, and security.
Document extraction	Field accuracy, omissions, and output validity.
AI agents	Task completion, tool correctness, permissions, and recovery from errors.
Business analytics	Calculation accuracy, evidence quality, and decision usefulness.

For high-stakes applications, evaluation should include domain expertise and appropriate human oversight.

↑ Back to contents

8. What Must an AI Engineer Decide?

A. What does success mean?

Translate business requirements into measurable acceptance criteria.

B. Which cases represent real use?

Build datasets that reflect users, tasks, difficult cases, and relevant operating conditions.

C. What should be graded automatically?

Use deterministic checks for objectively verifiable properties such as schema validity or test execution.

D. When are human reviewers necessary?

Include human assessment when judgments require domain knowledge, nuance, or contextual understanding.

E. Can a model grade another model?

Model-based graders can help scale evaluation, but their judgments may contain bias, inconsistency, and errors. Validate them against trusted examples.

F. How will improvements be compared?

Use consistent datasets, scoring criteria, and configurations. Investigate both improvements and regressions.

G. What are the deployment thresholds?

Set application-specific acceptance criteria, including critical failure conditions that cannot be offset by high average scores.

H. What happens after deployment?

Monitor quality and operational indicators, collect permitted feedback, investigate incidents, and update evaluation datasets.

Engineering caution: An evaluation score is evidence, not proof of universal reliability. Untested scenarios and changing real-world conditions remain sources of risk.

↑ Back to contents

9. Practical Industry Case: Enterprise Policy Assistant

A company has built an AI assistant that answers employee questions about travel and reimbursement policies.

During demonstrations, the assistant appears accurate and helpful. Before wider deployment, the engineering team needs evidence of reliability.

Engineering approach

1. Define the tasks the assistant is expected to perform.
2. Prepare a set of questions based on approved policies.
3. Include current, outdated, conflicting, and missing-policy scenarios.
4. Establish reference answers and supporting document passages.
5. Test whether the retrieval system finds the correct evidence.
6. Check the factual correctness of generated responses.
7. Verify whether citations support the stated claims.
8. Test whether unsupported questions trigger appropriate uncertainty or escalation.
9. Measure response latency and cost.
10. Review failures and identify their causes.
11. Revise retrieval, prompts, or application controls.
12. Rerun evaluations before deployment.

Illustrative evaluation results

The following figures are fictional teaching examples, not measured industry results.

Measure	Baseline	After revision
Correct answers	78%	91%
Claims supported by evidence	72%	94%
Appropriate escalation	65%	89%

These figures suggest improvement on the evaluated cases, but they do not establish production readiness. The team must also examine sample size, uncertainty, critical failures, security, and real-world representativeness.

Industry insight: Evals help teams move from asking "Does the assistant look good?" to asking "Under which conditions does it succeed, fail, or require human intervention?"

↑ Back to contents

10. Try It Yourself

Mini Project: Build Your First AI Evaluation

Objective: Design a simple evaluation that measures an AI assistant's ability to answer questions from an approved document.

1. Create a fictional one-page company travel policy.
2. Prepare ten questions about the policy.
3. Include straightforward, ambiguous, and unanswerable questions.
4. Write reference answers and identify supporting passages.
5. Ask an AI model to answer using the policy.
6. Record the outputs.
7. Score each response for factual correctness, evidence support, and appropriate uncertainty.
8. Calculate the percentage of cases passing each criterion.
9. Identify recurring failure patterns.
10. Improve the prompt or retrieval process.
11. Repeat the same evaluation and compare results.
12. Test the revised system on several new questions not used during improvement.

Suggested scoring rubric

Score	Meaning
2	Correct and adequately supported.
1	Partially correct or incomplete.
0	Incorrect, unsupported, or inappropriate.

For questions that cannot be answered from the document, an appropriate expression of uncertainty can receive a score of 2.

Reflection questions

• Which questions caused failures?
• Did the model invent unsupported information?
• Were citations accurate?
• Did prompt changes improve performance on new cases?
• Which failures would be unacceptable in production?
• What additional evidence would you need before deployment?

Success criterion: Produce a repeatable evaluation report showing test cases, scoring criteria, results, and observed limitations.

↑ Back to contents

11. Connected Glossary Terms

• Token: Influences evaluation costs and response budgets.
• Embedding: Can be evaluated for retrieval relevance.
• Vector Database: Retrieval performance can be measured independently.
• RAG: Requires retrieval, grounding, and answer-quality evaluations.
• Context Window: Long-context performance can be tested systematically.
• Prompt Engineering: Prompt versions can be compared through evals.
• Hallucination: Unsupported outputs are an important evaluation target.
• Observability: Production signals help identify new evaluation needs.

Previous lessons:

• #001 — Token
• #002 — Embedding
• #003 — Vector Database
• #004 — RAG
• #005 — Context Window
• #006 — Prompt Engineering

Next lesson: #008 — Hallucination: When AI Generates Unsupported Answers.

↑ Back to contents

12. Living Update

Editorial baseline: November 2026

Evaluation practices are evolving alongside more capable models, AI agents, tool-using systems, and increasingly complex workflows.

Important directions to monitor include:

• Automated evaluation pipelines
• Model-based grading with calibrated rubrics
• Agent trajectory and tool-use evaluations
• Realistic task-completion benchmarks
• Adversarial and security evaluations
• Multimodal evaluations
• Production feedback incorporated into test suites
• Continuous evaluation during model and prompt updates

The fundamental engineering principle remains stable: evaluation methods must be appropriate to the intended application and supported by trustworthy evidence.

Living Glossary note: Update evaluation methods as new system capabilities and failure patterns emerge.

↑ Back to contents

13. Key Takeaway

Evals convert AI performance from an impression into something engineers can measure, investigate, compare, and improve.

A successful demonstration is the beginning of engineering confidence, not its conclusion.

Think like an AI engineer: What evidence would convince us that this system is reliable enough for its intended users and operating conditions?

↑ Back to contents

14. References and Further Reading

• OpenAI — Evaluations Guide
• LangChain — Documentation and Evaluation Resources
• Ragas — Evaluation Documentation
• NIST — AI Risk Management Framework
• OWASP — Generative AI Security Project

↑ Back to contents

AI Engineering — Living Glossary

One Term. One Industry Connection. One Practical Insight. One Action.

AIYING — Accomplishing with AI

Chapter 2 | Episode #007 | EVALS

↑ Return to top