AI Engineering — Living Glossary #024
Lesson 24 of 24 Chapter 4 Final LessonLifecycle Operations (LLMOps): Managing AI Systems from Development to Continuous Improvement
Chapter 4: Deploying, Scaling, and Operating AI Systems
AIYING — Accomplishing with AI
Understand the Term. Connect the Technology. Apply the Knowledge.
1. The Industry Problem: The AI Assistant That Worked Yesterday
A global enterprise launches an AI knowledge assistant for employee questions about HR policies, travel expenses, procurement, and internal procedures.
The initial deployment performs well. Employees receive useful answers, and the system passes its acceptance tests.
Three months later, problems begin to appear:
- An updated travel policy is not reflected in the retrieval index.
- A model update changes how some answers are formatted.
- A revised prompt causes longer responses and increased cost.
- Some employees receive answers based on outdated documents.
- Different teams cannot easily identify which configuration produced a particular response.
No single component is necessarily completely broken. Yet the application is becoming less dependable.
The answer is to establish a disciplined operational lifecycle for the entire AI application.
↑ Back to contents2. What Is LLMOps?
LLMOps stands for Large Language Model Operations.
It refers to the engineering practices, processes, automation, and controls used to develop, evaluate, deploy, monitor, maintain, and improve applications built with large language models.
LLMOps typically manages more than the model itself. It may include:
- Prompt templates and system instructions.
- Model selection and versioning.
- Retrieval datasets and vector indexes.
- Tool integrations and workflow definitions.
- Evaluation datasets and test results.
- Application deployment and infrastructure.
- Observability, reliability, and cost.
- Security, safety, and governance controls.
- User feedback and continuous improvement.
The term is used broadly in industry, and its precise scope varies by organization. It overlaps with MLOps, DevOps, platform engineering, and AI governance.
↑ Back to contents3. Why Lifecycle Operations Matter
Models change
Model versions, capabilities, provider behavior, and availability may evolve over time.
Knowledge changes
Business policies, regulations, product documentation, and operational procedures are updated.
Prompts change
Even a small prompt modification can affect accuracy, response length, formatting, or tool usage.
Workflows change
New tools and integrations create additional dependencies and potential failure modes.
User behavior changes
Real users may ask questions that were not represented in the original test dataset.
Risk changes
New attack techniques, regulatory requirements, and operational conditions may require new controls.
4. The Complete LLMOps Lifecycle
LLMOps is best understood as a continuous loop, not a one-way development pipeline.
PLAN | v DESIGN | v BUILD | v EVALUATE | v RELEASE | v OPERATE | v MONITOR | v IMPROVE | +----------> PLAN
Plan
Define intended users, business objectives, acceptable risks, performance requirements, and measurable success criteria.
Design
Select models, retrieval architecture, tool boundaries, workflows, and safety controls.
Build
Implement prompts, integrations, data pipelines, evaluation harnesses, and application logic.
Evaluate
Test quality, safety, reliability, latency, cost, and task completion against representative cases.
Release
Deploy an approved configuration with traceable artifacts and rollback options.
Operate
Manage infrastructure, access, availability, incidents, and operational changes.
Monitor
Observe technical health and AI-specific behavior under real workloads.
Improve
Convert incidents, user feedback, and evaluation findings into controlled changes.
Every improvement should enter the evaluation and release process again.
↑ Back to contents5. Core LLMOps Capabilities
| Capability | Engineering purpose |
|---|---|
| Experiment tracking | Compare configurations and record results |
| Version control | Track models, prompts, datasets, tools, and code |
| Evaluation management | Maintain test sets, metrics, and quality gates |
| Release automation | Promote approved changes through environments |
| Deployment management | Control versions, traffic, and rollback |
| Observability | Monitor traces, metrics, logs, and quality signals |
| Incident response | Detect, contain, investigate, and recover |
| Cost management | Track spending and cost per successful task |
| Security and safety | Enforce permissions, policies, and risk controls |
| Feedback management | Turn production evidence into improvement work |
Organizations may implement these capabilities using different combinations of existing software engineering tools and specialized AI engineering platforms.
↑ Back to contents6. Reference LLMOps Architecture
A production LLMOps design connects development controls with runtime evidence.
DEVELOPMENT ENVIRONMENT
-----------------------
Source code and prompts
|
Model and tool configuration
|
Data / retrieval index versions
|
Automated evaluation suite
|
Security and quality gates
|
v
RELEASE APPROVAL
|
v
DEPLOYMENT PIPELINE
-------------------
Staging and integration tests
|
Canary / controlled rollout
|
v
PRODUCTION APPLICATION
----------------------
User -> Gateway -> Workflow
|
Retrieval / Model
|
Tools
|
Validated output
|
v
OPERATIONAL TELEMETRY
---------------------
Logs + Metrics + Traces
Quality + Safety + Cost
|
v
ALERTS AND INVESTIGATION
|
v
FEEDBACK / CHANGE REQUEST
|
+----> Development
Important design principle
Each production request should be associated, where feasible and appropriate, with the application release and relevant model, prompt, retrieval, and tool versions.
This makes it possible to investigate which changes may have contributed to a problem.
7. Versioning and Reproducibility
Traditional applications commonly version source code. LLM applications often require additional configuration and data artifacts to be tracked.
What should be versioned?
- Application code and dependencies.
- Prompt templates and instructions.
- Model identifiers and supported settings.
- Tool schemas and permission policies.
- Retrieval source snapshots and index versions.
- Evaluation datasets and scoring methods.
- Safety-control configurations.
- Release and infrastructure configurations.
Example release manifest
{
"release_id": "knowledge-assistant-1.4.0",
"model_id": "approved-model-version",
"prompt_version": "prompt-12",
"retrieval_index": "policies-2026-11",
"tool_schema_version": "tools-3",
"eval_suite": "enterprise-evals-7",
"safety_policy": "safety-5",
"approval_status": "approved"
}
The identifiers above are illustrative.
Reproducibility has limits
Even with fixed configurations, generative model outputs may vary. Hosted models may also change according to provider policies and versioning practices.
The practical objective is to reproduce the relevant environment and compare behavior within documented tolerances, not to promise identical text in every run.
↑ Back to contents8. Release Management and CI/CD for AI
CI/CD refers to continuous integration and continuous delivery or deployment.
For LLM applications, a release pipeline should include more than code compilation and ordinary software tests.
Example release pipeline
- Validate source code and dependencies.
- Validate prompt and configuration formats.
- Check tool permissions and access rules.
- Test retrieval and data-source compatibility.
- Run functional evaluations.
- Run safety and adversarial evaluations.
- Measure latency and estimated cost.
- Review release evidence.
- Deploy to staging.
- Perform controlled production rollout.
- Monitor release-specific outcomes.
- Expand rollout or initiate rollback.
Canary deployment
A canary rollout exposes a limited portion of traffic to a new configuration before broader release.
For example, a team might initially route 5% of eligible requests to a candidate version. This percentage is illustrative, not a universal recommendation.
Rollback
Rollback may involve reverting application code, prompt configuration, model selection, retrieval indexes, or tool permissions.
Not every data change is immediately reversible. Teams must plan rollback and recovery according to the artifacts involved.
↑ Back to contents9. Continuous Evaluation: The Quality Gate
LLMOps depends on evaluation before and after important changes.
Offline evaluations
Run curated test cases before deployment to assess expected behavior.
Integration evaluations
Test retrieval, tool execution, authorization, and workflow behavior together.
Production evaluations
Use privacy-aware sampling, approved feedback, and appropriate quality-assessment methods to detect regressions.
Human evaluations
Qualified reviewers may be needed for complex or consequential cases.
Evaluation dimensions
- Answer correctness and grounding.
- Task completion.
- Tool-call success.
- Structured-output validity.
- Safety and policy compliance.
- Latency and reliability.
- Cost per successful task.
- Human escalation appropriateness.
10. Production Observability and Feedback
Production systems must be observed under actual operating conditions.
Technical signals
- Availability and error rates.
- p50 and p95 response latency.
- Model and tool-call failures.
- Request volume and queueing.
- Token usage and cost.
AI-specific signals
- Answer quality and grounding.
- Retrieval freshness and relevance.
- Structured-output validation failures.
- Prompt-injection and safety events.
- Human escalation and overrides.
- Task success and user feedback.
Feedback loop
Production signal
|
v
Investigate affected requests
|
v
Identify a possible failure mode
|
v
Create regression test cases
|
v
Propose configuration change
|
v
Evaluate and release
|
v
Verify production recovery
Feedback should be curated and reviewed before it is used for training, fine-tuning, or changes to evaluation datasets.
User feedback alone may be incomplete, biased, or difficult to interpret.
↑ Back to contents11. Security, Safety, and Governance Throughout the Lifecycle
LLMOps provides an operational mechanism for implementing AI governance decisions.
Security
Manage credentials, authorization, data access, tool permissions, and application vulnerabilities.
Safety
Test harmful outputs, prompt injection, excessive agency, and foreseeable misuse.
Privacy
Control data collection, retention, logging, and access to sensitive information.
Accountability
Assign owners for releases, evaluation, risk acceptance, and incident response.
Auditability
Maintain appropriate evidence of configuration changes, approvals, test outcomes, and operational incidents.
Lifecycle governance
Define when a system should be modified, restricted, suspended, or retired.
12. DevOps, MLOps, and LLMOps: What Is Different?
| Discipline | Primary focus | Characteristic concerns |
|---|---|---|
| DevOps | Software delivery and operations | Builds, tests, deployment, infrastructure, reliability |
| MLOps | Machine-learning lifecycle | Training data, experiments, models, drift, deployment |
| LLMOps | LLM-based application lifecycle | Prompts, retrieval, model APIs, agents, evaluations, safety, cost |
| AI Governance | Direction, oversight, and accountability | Policies, risk ownership, compliance, assurance |
LLMOps builds on DevOps and MLOps. It is not a complete replacement for either.
The practical difference is that LLM application behavior often depends on a changing combination of prompts, models, external data, tools, and runtime context.
↑ Back to contents13. Industry Applications
Enterprise knowledge management
Manage document freshness, retrieval indexes, access policies, and response quality.
Customer support
Track prompt versions, tool integrations, escalation behavior, and cost per resolved case.
Financial services
Maintain controlled releases, evidence, authorization, and oversight for sensitive information workflows.
Healthcare administration
Manage privacy-aware deployment, workflow validation, and human review.
Manufacturing
Version approved technical documents, test safety-sensitive outputs, and control maintenance workflow changes.
Software engineering agents
Control coding-agent permissions, tool versions, test execution, and production deployment approvals.
Education
Evaluate instructional quality, safety, accessibility, and changing curricula.
↑ Back to contents14. Detailed Industry Case: Enterprise Knowledge Assistant
The following is a fictional, industry-grounded teaching scenario.
Business context
A multinational organization deploys an internal AI assistant for employees to search approved HR, travel, procurement, and policy documents.
Initial production release
The assistant launches with:
- An approved model configuration.
- Versioned system prompts.
- A retrieval index of internal policies.
- Access-controlled document retrieval.
- Evaluation cases for common questions.
- Production telemetry and support procedures.
Three months later
Employees report that some travel-expense answers refer to outdated reimbursement rules.
At the same time, the platform team has introduced a new prompt version and changed its model configuration.
Step 1 — Detect
Production quality evaluations and employee feedback indicate an increase in answers containing outdated policy information.
Step 2 — Correlate
Engineers compare affected requests with release identifiers, prompt versions, retrieval-index versions, and source-document timestamps.
Step 3 — Investigate
The team discovers that a revised travel policy was approved but not successfully incorporated into the production retrieval index.
Separately, a prompt change has increased response length and token usage.
Step 4 — Contain
The affected policy-answering workflow is temporarily restricted to approved, verified responses or human escalation.
Step 5 — Correct
- Repair the document ingestion process.
- Rebuild and validate the affected retrieval index.
- Restore or revise the prompt configuration.
- Run targeted policy-answering evaluations.
- Check authorization and document access.
Step 6 — Release
The corrected configuration is deployed through a controlled release process.
Step 7 — Verify
The team monitors affected answer quality, document freshness, latency, and cost after deployment.
Step 8 — Prevent recurrence
Engineers add a release gate that verifies critical policy documents are indexed before a relevant knowledge release is approved.
All incident details and outcomes in this example are illustrative.
↑ Back to contents15. Engineering Decisions and Trade-offs
| Decision | Benefit | Trade-off |
|---|---|---|
| Frequent releases | Faster improvements | Greater regression-management burden |
| Strict release gates | Higher assurance | Longer release cycles |
| Automated evaluations | Repeatable testing | Incomplete coverage and evaluator errors |
| Human approvals | Accountability for high-impact changes | Review effort and delay |
| Extensive telemetry | Better investigations | Cost and privacy exposure |
| Frequent index updates | Fresher information | Additional processing and validation |
| Multiple model providers | Flexibility and resilience | More integration and evaluation complexity |
| Rollback capability | Faster recovery | Artifact and data compatibility challenges |
There is no universal release process appropriate for every AI application. Controls should reflect business importance, potential harm, operating scale, and organizational requirements.
↑ Back to contents16. Hands-On Capstone: Build an LLMOps Release System
Project title: Enterprise AI Assistant — Release, Monitor, Improve.
Purpose: Integrate the engineering concepts introduced across the 24-lesson course.
Level: Beginner to intermediate.
Tools: Spreadsheet, JSON files, optional Python, and simulated application data.
No paid API is required for the basic version.
Phase 1 — Define the application
Create a fictional internal policy assistant for travel, HR, and procurement questions.
Specify intended users, authorized sources, acceptable tasks, and prohibited actions.
Phase 2 — Prepare versioned artifacts
Create:
- Two prompt versions.
- Two fictional document-index versions.
- A model configuration manifest.
- A simulated tool permission policy.
- A release identifier.
Phase 3 — Build an evaluation dataset
Prepare 30 fictional test cases:
- 10 ordinary policy questions.
- 5 ambiguous questions.
- 5 outdated-document scenarios.
- 5 access-control cases.
- 5 safety or prompt-injection cases.
Phase 4 — Define release gates
Set explicit acceptance criteria for quality, safety, retrieval freshness, and operational performance.
Use illustrative targets for the exercise and document why they were chosen.
Phase 5 — Simulate deployment
Create release records for version 1.0 and version 1.1.
Compare evaluation results before approving the candidate release.
Phase 6 — Introduce a regression
Modify one policy document without updating the simulated retrieval index.
Observe which evaluation cases fail.
Phase 7 — Investigate
Use version records and simulated production traces to identify the outdated index.
Phase 8 — Repair and release
Update the index version, rerun evaluations, and document the approval decision.
Phase 9 — Verify recovery
Compare the corrected configuration with the previous release.
Phase 10 — Produce the capstone report
Submit:
- Application architecture.
- Version manifest.
- Evaluation dataset.
- Release-gate checklist.
- Simulated deployment record.
- Incident report.
- Corrective-action record.
- Recovery evaluation.
- Continuous-improvement proposal.
17. Capstone Evaluation Criteria
| Dimension | Evidence of achievement |
|---|---|
| Architecture | Clearly defined application components and boundaries |
| Versioning | Traceable prompt, model, data, and tool configurations |
| Evaluation | Repeatable test cases and explicit acceptance criteria |
| Deployment | Documented release and rollback decisions |
| Observability | Useful operational and quality indicators |
| Safety | Authorization and high-risk controls preserved |
| Cost | Resource usage and economic impact considered |
| Incident response | Evidence-supported diagnosis and containment |
| Recovery | Regression tests demonstrate corrected behavior |
| Improvement | Incident lessons incorporated into future controls |
Suggested grading rubric
- Architecture and versioning — 20 points.
- Evaluation and release gates — 25 points.
- Observability and incident investigation — 20 points.
- Safety, security, and accountability — 20 points.
- Recovery and continuous improvement — 15 points.
Total: 100 points.
For a training exercise, a score of 75 or above may be used as a suggested completion threshold, provided essential safety requirements are met.
↑ Back to contents18. The Connected 24-Term AI Engineering Glossary
LLMOps brings together every concept introduced throughout this course.
Chapter 1 — Foundations of AI Engineering
- Token — Usage and processing units.
- Embedding — Semantic representations.
- Vector Database — Similarity-based retrieval infrastructure.
- RAG — Grounding responses in retrieved sources.
- Context Window — Managing information available to a model.
- Prompt Engineering — Designing effective model instructions.
Chapter 2 — Engineering AI for Reliability
- Evals — Measuring AI behavior.
- Hallucination — Managing unsupported outputs.
- Grounding — Connecting claims to evidence.
- Guardrails — Enforcing defined constraints.
- Observability — Understanding system behavior.
- Human-in-the-Loop — Designing meaningful human review.
Chapter 3 — Connecting AI to Tools, Workflows, and Agents
- Tool Calling — Connecting models to authorized capabilities.
- Workflows — Organizing task execution.
- AI Agents — Managing goal-directed model-driven actions.
- Agent Orchestration — Coordinating agent execution.
- Model Context Protocol (MCP) — Standardizing connections to external capabilities.
- Structured Outputs — Producing machine-readable results.
Chapter 4 — Deploying, Scaling, and Operating AI Systems
- Deployment — Moving applications into production.
- Scalability — Supporting growing demand.
- Production Observability — Monitoring real-world behavior.
- Cost Optimization — Improving economic efficiency.
- AI Safety and Risk — Managing potential harms.
- Lifecycle Operations (LLMOps) — Managing the complete operational lifecycle.
19. Living Update: Keeping LLMOps Current
Editorial baseline: November 2026.
LLMOps practices must evolve as AI platforms, model capabilities, tool protocols, security threats, and organizational requirements change.
Areas to monitor include:
- Model and API versioning practices.
- Agent evaluation and execution tracing.
- Prompt and retrieval artifact management.
- Automated quality and safety evaluation.
- Production feedback and data governance.
- Multi-model and multi-provider deployments.
- Inference efficiency and cost controls.
- AI security and applicable regulatory requirements.
Engineers should consult current official documentation and test changes in their own environment rather than assuming that a platform feature or model behavior remains unchanged.
↑ Back to contents20. Key Takeaway and One Action
Understand the Term: LLMOps manages the lifecycle of LLM-based applications.
Connect the Technology: Prompts, models, retrieval, tools, evaluations, deployment, observability, safety, and cost become parts of one managed operational system.
Apply the Knowledge: Design a traceable release process that can detect and correct regressions.
One Action: Create a release checklist for one AI application containing version identifiers, evaluation results, deployment approval, monitoring signals, and rollback instructions.
One Term. One Industry Connection. One Practical Insight. One Action.
↑ Back to contents21. Authoritative References
- MLflow Documentation — Experiment tracking and model-related lifecycle capabilities.
- OpenTelemetry Documentation — Distributed tracing, metrics, and logs.
- GitHub Actions Documentation — Automated build, test, and deployment workflows.
- Kubernetes Documentation — Application deployment and infrastructure operations.
- Google Site Reliability Engineering Books — Production reliability, monitoring, and incident management.
- NIST AI Risk Management Framework — Risk governance and AI-system management.
- OWASP Top 10 for LLM Applications — Security risks in LLM applications.
- OpenAI API Documentation — Model integration and application engineering guidance.
No comments:
Post a Comment