Site Reliability Engineering Through the Pattern AI Lens
From keeping software available to making Generative AI dependable
Executive summary
Site Reliability Engineering—or SRE—applies software-engineering methods to the operation of production systems. Google, where the discipline originated, describes SRE as what happens when operations is treated as a software problem. Its concern extends beyond uptime to availability, latency, performance and capacity.
Viewed through the Pattern AI lens, SRE is a system for recognising recurring patterns of failure, converting them into measurable reliability objectives, and redesigning systems so they can detect, absorb and learn from disruption.
This becomes even more important in Generative AI. A conventional service may return an HTTP 200 response and still fail its user because the answer is inaccurate, unsafe, irrelevant or unsupported. GenAI reliability must therefore measure not only whether the system responded, but whether it produced an acceptable outcome.
1. What is Site Reliability Engineering?
Traditional IT operations asks, “Is the system running?” SRE asks a larger set of questions: Is it delivering what users need? How reliable must it be? How will reliability be measured? How much failure is tolerable? Which failures recur? Which operational tasks should be automated? How can every incident make the system stronger?
| Traditional operations emphasis | SRE emphasis |
|---|---|
| Keep infrastructure running | Protect the user experience |
| Respond to incidents | Engineer away recurring incidents |
| Follow manual procedures | Automate repetitive work |
| Maximise uptime | Meet explicit reliability objectives |
| Avoid every failure | Manage an acceptable level of risk |
| Identify who caused the failure | Learn how the system allowed it |
| Restore the old state | Improve the future system |
SRE is often associated with DevOps, but the two are not identical. DevOps offers broad principles for collaboration, automation and continuous delivery. SRE translates many of those principles into operating mechanisms such as service-level objectives, error budgets, on-call practices and reliability engineering. Google identifies measurement—particularly SLOs—as central to SRE.
2. SRE Through the Pattern AI Lens
AI is a pattern technology. Pattern is AI’s native language and the new index to knowledge. SRE is also fundamentally a pattern discipline. Production systems emit normal and abnormal traffic patterns, latency distributions, recurring deployment failures, resource saturation, dependency cascades, user-reported symptoms, incident-response behaviours and repetitive operational work.
Observe signals
Find patterns
Choose interventions
Design resilience
Automate
Improve
SEE: Observe the system
SRE makes system behaviour visible through metrics, logs, distributed traces, user journeys, synthetic transactions, dependency maps, alerts, customer complaints and incident histories.
| Google’s four golden signals | What they reveal |
|---|---|
| Latency | How long requests take |
| Traffic | How much demand reaches the system |
| Errors | How many requests fail |
| Saturation | How close resources are to their limits |
A metric is an observation; a pattern is a meaningful relationship among observations. “CPU utilisation is 82%” is a metric. “CPU rises after every recommendation-model deployment, followed by inference latency and abandoned sessions” is an actionable pattern.
UNDERSTAND: Find the pattern behind the incident
| Level | Question |
|---|---|
| Event | What happened? |
| Trend | Is it recurring or worsening? |
| Correlation | What else changes at the same time? |
| Structure | Which dependencies create the behaviour? |
| Feedback loop | What makes the problem amplify itself? |
| Intervention | Where can the pattern be interrupted? |
Illustration: the retry storm
A local response—retrying a failed request—can produce a system-wide failure. SRE interrupts this pattern with exponential backoff, retry limits, circuit breakers, load shedding, graceful degradation and request prioritisation. Reliability is an emergent property of relationships among components, not merely a property of individual components.
LEVER: Convert expectations into measurable objectives
| Term | Meaning | Example |
|---|---|---|
| SLI | Service Level Indicator—a measured reliability signal | Percentage of successful checkout requests |
| SLO | Service Level Objective—the desired target | 99.9% successful checkouts in 30 days |
| SLA | Service Level Agreement—a contractual commitment | Compensation if agreed availability is missed |
The SLO should reflect something users experience. A server can be running while search returns nothing useful, payment confirmation is delayed, video repeatedly buffers or an AI assistant produces unsupported answers. The useful question is not “Is the server alive?” but “Can the user successfully accomplish the intended task?”
Error budgets: reliability without stopping innovation
SRE rejects the assumption that every service must achieve 100% availability. Google notes that insisting on 100% can reduce innovation, slow releases and produce disproportionately expensive designs.
If monthly availability must be 99.9%, the error budget is 0.1%. In a 30-day month:
The service can experience approximately 43.2 minutes of unavailability while still meeting that target, assuming availability is measured by elapsed time.
Low: reduce risk
Exhausted: reliability work
REIMAGINE: Design systems that expect failure
| Failure pattern | Reliability response |
|---|---|
| One instance fails | Redundancy and automatic failover |
| Demand suddenly increases | Autoscaling and admission control |
| A dependency becomes slow | Timeouts and circuit breakers |
| Retries amplify traffic | Backoff, jitter and retry budgets |
| A deployment introduces defects | Canary releases and automatic rollback |
| A region becomes unavailable | Multi-region recovery strategy |
| A nonessential feature fails | Graceful degradation |
| Work accumulates | Queues, backpressure and load shedding |
| Corrupted data spreads | Validation, versioning and recovery controls |
During peak demand, a food-delivery application might temporarily disable personalised recommendations, reduce image quality and delay analytics while protecting checkout and payment. A fragile system depends on every component succeeding; a resilient system continues providing essential value when some components fail.
ACT: Automate recurring operational work
Google uses toil for repetitive, manual, automatable operational work that scales with service growth and produces limited enduring value—for example, restarting instances, repeating deployment checks or responding to noisy alerts.
Automation must be bounded by safety checks, rollback mechanisms, human approval for high-impact actions, auditability and continuous validation.
LEARN: Turn incidents into organisational memory
An incident should produce two recoveries: service recovery, which restores the system, and learning recovery, which reduces the chance or impact of recurrence. Blameless postmortems focus on system conditions and decision context rather than finding someone to punish. An incident is not only a failure event; it is a labelled dataset for organisational learning.
3. Real-world case studies
Google: SLOs and error budgets
Google’s SRE model institutionalised SLOs and error budgets to balance change with stability. Product teams seek faster releases while operations teams fear instability. A shared user-centred measure turns competing opinions into an objective risk decision.
Pattern AI insight: The error budget is a balance pattern. It prevents reckless innovation without discipline and excessive caution that prevents useful change.
Evernote and The Home Depot: an SLO culture
Case studies in Google’s Site Reliability Workbook show how SLOs created a shared language across development, operations and business teams, improving communication and informing product decisions.
Netflix: Chaos Monkey and designed resilience
Netflix developed Chaos Monkey and the wider Simian Army to introduce controlled failures into its cloud environment and test survival of instance, network and availability-zone problems. During an AWS degradation in October 2012, Netflix reported limited customer impact while documenting both strengths and areas for improvement.
Pattern AI insight: Unexpected failure becomes controlled failure, observed behaviour and engineered resilience.
Google Detectr: GenAI assisting SRE
Google describes Detectr as a Gemini-powered SRE platform that organises user feedback from sources such as customer support, forums and social media to detect user-reported outages. It acts as a backstop for novel problems that may not match predefined metric alerts.
Pattern AI insight: GenAI is becoming both an object of reliability engineering and an instrument for reliability engineering.
4. Mapping SRE to Generative AI projects
GenAI systems are probabilistic. Their behaviour varies with prompt wording, model version, sampling, retrieved documents, conversation history, tool outputs, policy decisions and external services. Microsoft notes that conventional latency, error and throughput telemetry is insufficient because uptime and error rate do not adequately represent output quality.
The second equation is a conceptual model, not an industry-standard formula.
| Classical SRE concept | GenAI equivalent |
|---|---|
| Service availability | Model and application availability |
| Successful request | Useful, safe and policy-compliant response |
| Latency | Time to first token and total generation time |
| Error rate | Technical failures plus unacceptable outputs |
| Traffic | Requests, tokens, sessions and tool calls |
| Saturation | GPU, memory, context-window, quota and rate-limit pressure |
| Dependency monitoring | Vector store, model provider, tools and data-source health |
| Release engineering | Prompt, model, embedding, guardrail and retrieval releases |
| Canary deployment | Limited traffic to a new model or prompt |
| Rollback | Restore the previous model, prompt or index |
| Load shedding | Limit low-priority requests or expensive workflows |
| Graceful degradation | Fall back to search, templates, cached responses or humans |
| Postmortem | Analyse prompt, retrieval, model and tool traces |
| Error budget | Tolerable rate of unacceptable AI outcomes |
| Chaos testing | Simulate model timeouts, poor retrieval and tool failures |
| Toil reduction | Automate evaluations, tracing, triage and rollback |
5. GenAI reliability signals
| Layer | Signals to observe |
|---|---|
| Infrastructure | Availability, errors, throughput, time to first token, total latency, GPU/memory use, rate limits, queue depth and cost |
| Model and output | Task success, correctness, groundedness, citation validity, relevance, refusal correctness, safety, consistency and user corrections |
| RAG | Retrieval success, relevant-document recall, source freshness, claim support, empty retrievals, access violations and index version |
| Agentic AI | Tool-selection accuracy, tool failures, unauthorised attempts, loops, completion, escalation, step count, cost and provenance |
AWS CloudWatch offers GenAI visibility into latency, usage, errors and end-to-end prompt traces across models, tools and knowledge bases. Microsoft Foundry combines evaluations, logs, traces, model outputs and quality gates to observe performance, quality, safety and operational health.
6. Designing SLOs for a GenAI application
For an enterprise RAG assistant, “99.9% of API requests return successfully” is necessary but inadequate. The system could meet it while producing unsupported answers. A richer, illustrative SLO portfolio follows.
| Reliability dimension | Illustrative SLO |
|---|---|
| Availability | 99.9% of valid requests receive a response |
| Latency | 95% of responses begin within 2 seconds |
| Grounding | At least 97% of tested factual claims are supported by approved sources |
| Citation validity | 99% of generated citations resolve to authorised documents |
| Safety | At least 99.9% of high-risk test prompts produce policy-compliant behaviour |
| Task success | At least 90% of benchmark tasks meet the acceptance rubric |
| Cost | 95% of completed tasks stay within the approved cost threshold |
| Escalation | Low-confidence, high-impact requests are routed to a human |
7. Illustrative GenAI case: an unreliable support assistant
A customer-support assistant reports 99.95% API availability, yet users complain that refund answers are inconsistent, citations do not support claims, the agent sometimes invokes the wrong tool, and costs rise during complex conversations. Infrastructure dashboards classify it as healthy—but users do not.
| Pattern AI stage | Action |
|---|---|
| SEE | Capture conversation, prompt/model version, retrieved passages, provenance, tool calls, tokens, latency, corrections and evaluations. |
| UNDERSTAND | Find outdated policies in the index, jurisdictionally wrong retrieval and repeated planning loops after tool failures. |
| LEVER | Target document freshness, retrieval filters and agent retry/escalation policy. |
| REIMAGINE | Add date/country metadata filters, citation-support evaluation, tool-call limits, human escalation and a safe read-only mode. |
| ACT | Canary the change to 5% of traffic and compare quality, latency, failures, escalation and cost. |
| LEARN | Expand if SLOs improve; otherwise roll back the prompt, retrieval configuration or agent policy. |
8. GenAI can strengthen SRE—but must not be unquestioned
GenAI can summarise incidents, correlate telemetry, cluster complaints, generate diagnostic queries, locate runbooks, draft postmortems and suggest remediation. Yet AI findings require validation before production action.
| Operational risk | Appropriate AI role |
|---|---|
| Low | Summarise, classify and recommend |
| Moderate | Execute reversible actions with confirmation |
| High | Require explicit human approval |
| Critical | Human-led response with AI assistance only |
9. SRE anti-patterns
- “We have monitoring, therefore we have SRE.” Monitoring supplies signals; SRE connects them to user objectives, risk, automation and learning.
- “Everything must be 100% reliable.” Absolute reliability may be impossible, unnecessary or uneconomic.
- “More alerts create more safety.” Excess alerts create fatigue; alerts should be actionable and tied to user impact.
- “The on-call engineer is the reliability system.” Heroic recovery hides structural weakness.
- “The person who deployed the defect caused the incident.” Failures usually arise from multiple system conditions.
- “HTTP 200 means the GenAI system is reliable.” Transport success does not establish truth, relevance, safety or task completion.
- “An AI judge alone can certify quality.” Model evaluation must be calibrated against human judgement and domain tests.
10. A Pattern AI operating model for GenAI reliability
| Layer | Reliability question |
|---|---|
| Outcome | Did the user accomplish the intended task? |
| Experience | Was the response timely, relevant and understandable? |
| Model | Was the output accurate, safe and appropriate? |
| Context | Was retrieval current, authorised and relevant? |
| Orchestration | Did prompts, agents and tools behave correctly? |
| Platform | Were endpoints, databases and APIs healthy? |
| Infrastructure | Were compute, network, storage and capacity available? |
Each lower layer supports the one above, but no lower layer alone guarantees a successful outcome.
11. Practical implementation roadmap
- Define the user outcome: identify the user, task, success criteria, unacceptable results and consequence of failure.
- Map the system: infrastructure, services, models, prompts, retrieval, indexes, tools, guardrails and human approvals.
- Choose SLIs and SLOs: cover operational health and outcome quality.
- Build observability: reconstruct interactions while respecting privacy, access control and retention.
- Establish controlled releases: version models, prompts, embeddings, retrieval settings, guardrails, tools and test sets.
- Prepare for failure: use runbooks, fallbacks, safe modes, escalation, rate limits and provider-outage plans.
- Institutionalise learning: conduct postmortems, pattern reviews, regression evaluations, error-budget reviews and reliability improvement work.
12. The larger Pattern AI insight
Conclusion
Site Reliability Engineering is not merely an operational role or collection of monitoring tools. It is a disciplined method for turning recurring system behaviour into measurable, actionable learning.
- SLOs define acceptable service.
- Error budgets balance innovation and stability.
- Observability reveals behaviour.
- Incident response interrupts harmful patterns.
- Automation converts repetition into reusable capability.
- Chaos engineering rehearses failure.
- Postmortems create organisational memory.
- GenAI evaluation extends reliability from uptime to output and outcome quality.
The question is no longer simply, “Is the site up?” It is: Is the complete intelligent system producing dependable outcomes—and can we detect, explain and improve it when it does not?
Sources and verification
- Google SRE: What is Site Reliability Engineering?
- Google: How SRE Relates to DevOps
- Google SRE: Monitoring Distributed Systems and the Four Golden Signals
- Google SRE: Service Level Objectives
- Google SRE: Embracing Risk
- Google SRE Workbook: Eliminating Toil
- Google SRE Workbook: Postmortem Culture
- Google: Evernote and The Home Depot SLO Case Studies
- Netflix: The Simian Army
- Netflix: Postmortem of the October 2012 AWS Degradation
- Google SRE: AI Engineering Reliable Operations and Detectr
- Microsoft: Observability for Generative and Agentic AI Systems
- AWS CloudWatch: Generative AI Observability
- Microsoft Foundry: AI Observability
- Microsoft: Azure Copilot Observability Agent Transparency
No comments:
Post a Comment