AI Engineering Living Glossary #020 — Scalability: Designing AI Systems for Growing Demand

AI ENGINEERING — LIVING GLOSSARY #020

Lesson 20 of 24 | Chapter 4

SCALABILITY

Designing AI Systems for Growing Demand

Chapter 4: Deploying, Scaling, and Operating AI Systems

AIYING — Accomplishing with AI

The AI Assistant That Could Not Survive a Sale

An e-commerce company launches an AI shopping assistant. During a pilot, the assistant answers product questions, retrieves catalog information, and explains order status.

The pilot serves a small number of shoppers successfully. The company then announces a major promotional sale.

Thousands of customers arrive within minutes.

Suddenly, some responses take 30 seconds. Others fail. The model provider reports rate-limit errors. Product search becomes slow. Repeated requests increase costs.

The assistant has not necessarily become less intelligent. Its system capacity is inadequate for the workload.

Central engineering insight: Scalability is the ability to accommodate changing demand while meeting defined requirements for latency, throughput, reliability, quality, and cost.

Scaling an AI system requires understanding the complete request path, not simply increasing the number of model calls.

↑ Back to contents

Contents

1. What Is Scalability?
2. Why AI Scalability Is Different
3. Essential Scaling Metrics
4. Vertical and Horizontal Scaling
5. Scalable AI Architecture
6. Practical Scaling Techniques
7. Capacity Planning
8. Bottlenecks and Failure Modes
9. Scalability vs Related Concepts
10. Industry Applications
11. Industry Case: Retail Shopping Assistant
12. Engineering Design Decisions
13. Hands-On Mini Project
14. Evaluation and Acceptance Criteria
15. Connected Glossary Terms
16. Living Update
17. Key Takeaway
18. References

1. What Is Scalability?

Scalability is the ability of a system to handle increased or changing workload by adjusting resources, architecture, or execution methods while maintaining specified service objectives.

For an AI application, the workload may grow through:

• More concurrent users.
• More requests per second.
• Longer prompts and generated responses.
• Larger document collections.
• More retrieval operations.
• Additional tool calls or agent steps.
• More complex reasoning tasks.

Scalability is not unlimited capacity. Every system has constraints, including provider quotas, compute resources, network limits, and budgets.

A scalable design anticipates these limits and makes growth manageable.

Important: A system that serves ten times more users at ten times the cost is not necessarily cost-efficient. Scaling must be evaluated against service and economic objectives.

↑ Back to contents

2. Why AI Scalability Is Different

Conventional web services often perform relatively predictable operations. AI workloads can vary substantially from request to request.

One request might contain 200 input tokens and produce a short answer. Another might contain thousands of input tokens, retrieve documents, invoke multiple tools, and generate a long response.

Important AI-specific constraints include:

• Token demand: Prompt and completion lengths affect processing time and consumption.
• Model inference: Large-model execution can be computationally expensive.
• Provider limits: Requests-per-minute and tokens-per-minute quotas may constrain throughput.
• Retrieval: Vector search and database access introduce additional latency.
• Agent loops: One user request may trigger many model and tool calls.
• Variable quality: Faster or cheaper execution strategies may change answer quality.
• External dependencies: A slow catalog API can delay the entire response.

The slowest or most constrained component can determine end-to-end performance.

↑ Back to contents

3. Essential Scaling Metrics

Metric	Meaning	Engineering Question
Throughput	Completed requests or tasks per unit of time.	How much work can the system finish?
Concurrency	Requests being processed simultaneously.	How many tasks are in progress?
Latency	Time taken to complete a request.	How long does the user wait?
p95 latency	95th-percentile response time.	How slow are the slower requests?
Error rate	Fraction of requests that fail.	Is overload causing failures?
Token consumption	Input and output tokens used.	What inference capacity is required?
Queue depth	Work waiting to be processed.	Is demand exceeding service capacity?
Cost per successful task	Total relevant cost divided by successful outcomes.	Is scaling economically sustainable?
Task quality	Accuracy or success against evaluation criteria.	Does quality remain acceptable under load?

Concurrency is not throughput

A service may have 100 requests in progress while completing only 10 per second.

Under relatively stable conditions, Little's Law gives a useful relationship:

Average in-flight work
  = Throughput × Average time in system

L = λ × W

For example, at 20 completed requests per second and an average end-to-end time of 5 seconds, average in-flight work is approximately 100 requests.

This relationship helps with capacity estimation, but actual systems must account for variable workloads, queues, bursts, and failure conditions.

↑ Back to contents

4. Vertical and Horizontal Scaling

Vertical scaling — Scale up

Increase the capacity of an individual machine or service instance, such as adding CPU, memory, or GPU resources.

Advantage: May be straightforward for a bounded workload.

Limitation: Hardware and service limits eventually constrain growth.

Horizontal scaling — Scale out

Add more service instances and distribute work among them.

Advantage: Can support greater capacity and resilience when components are designed for distribution.

Limitation: Requires load balancing, shared-state design, coordination, and careful dependency management.

Scaling down

Reduce allocated capacity when demand falls, where supported, to avoid unnecessary cost.

                Load Balancer
                     |
           +---------+---------+
           |         |         |
           v         v         v
        App 1      App 2      App 3
           |         |         |
           +---------+---------+
                     |
                     v
             Model / Data APIs

Adding application instances does not automatically increase the quota of a shared model provider or the capacity of a shared database.

↑ Back to contents

5. Scalable AI Architecture

A scalable retail AI assistant may use:

             Customer Requests
                     |
                     v
          API Gateway / Rate Limits
                     |
                     v
                Load Balancer
                     |
             +-------+-------+
             |       |       |
             v       v       v
          AI App   AI App   AI App
             |       |       |
             +-------+-------+
                     |
           +---------+---------+
           |         |         |
           v         v         v
        Cache    Retrieval   Task Queue
           |         |         |
           |         v         v
           |     Vector DB   Workers
           |                   |
           +---------+---------+
                     |
                     v
             Model Gateway
                     |
                     v
            Model Provider(s)

Cross-cutting:
Authorization | Tracing | Evals
Budgets | Alerts | Fallbacks

Stateless application services

Where practical, keep application instances stateless and store necessary shared session or task state in suitable external systems.

Model gateway

A model gateway can centralize provider routing, rate-limit handling, observability, and policy enforcement.

Background processing

Long-running or non-interactive tasks can be processed asynchronously, with explicit task status and completion handling.

Dependency isolation

Separate components so that an overloaded reporting workload does not necessarily block customer-facing requests.

↑ Back to contents

6. Practical Scaling Techniques

A. Load balancing

Distribute requests across available application instances. Apply health checks to avoid routing traffic to unhealthy instances.

B. Autoscaling

Adjust service capacity using suitable demand signals such as CPU utilization, request concurrency, or queue depth.

Autoscaling is not instantaneous. Plan for startup delays and sudden traffic spikes.

C. Caching

Reuse eligible results rather than repeating expensive computation or retrieval.

For example, public product descriptions may be cached. Personalized order details require stricter access controls, isolation, and freshness rules.

D. Batching

Combine compatible operations where the model or processing service supports efficient batch execution.

Batching can improve throughput but may increase waiting time for individual requests.

E. Queues and backpressure

Place non-urgent tasks in bounded queues and control how quickly workers consume them.

When capacity is exhausted, return an explicit overload response or apply a defined admission policy rather than accepting unlimited work.

F. Rate limiting

Control requests by user, tenant, application, or other appropriate scope to protect shared resources.

G. Token budgeting

Reduce unnecessary context and cap output lengths where appropriate, while checking that answer quality remains acceptable.

H. Model routing

Use different approved models for different task requirements, based on measured quality, latency, and cost.

I. Retrieval optimization

Improve indexing, filtering, query design, and retrieval configuration rather than assuming all delays originate in the language model.

J. Graceful degradation

When capacity is limited, offer simpler verified functionality, defer non-urgent work, or provide clear fallback responses.

Scaling principle: First measure the bottleneck. Then change the component or behavior that limits the system.

↑ Back to contents

7. Capacity Planning

Capacity planning estimates the resources required for a projected workload.

Illustrative token-demand calculation

Assume an AI assistant receives:

• 30 requests per second.
• An average of 1,000 input tokens per request.
• An average of 300 output tokens per request.

Average tokens per request
= 1,000 + 300
= 1,300 tokens

Average token demand
= 30 × 1,300
= 39,000 tokens/second

Equivalent average demand
= 2,340,000 tokens/minute

This is an aggregate planning estimate, not a claim that a particular model service can deliver that throughput.

Real providers may enforce separate input, output, request, and concurrency limits. Output generation can also impose different performance constraints from input processing.

Why averages are insufficient

A promotional sale may produce short bursts far above average demand. Capacity planning should therefore include peak traffic, request-size distributions, provider quotas, retries, and safety margins.

Measure actual usage before selecting capacity.

↑ Back to contents

8. Bottlenecks and Failure Modes

Problem	Likely Cause	Possible Response
High model latency	Long requests, inference saturation.	Measure token demand; optimize or add permitted capacity.
Rate-limit errors	Provider quota exceeded.	Admission control, bounded retries, capacity planning.
Slow retrieval	Database contention or inefficient queries.	Indexing, filtering, query tuning, caching.
Growing queue	Arrival rate exceeds processing rate.	Add effective worker capacity or reduce admitted demand.
High cost	Repeated calls, excessive tokens, inefficient routing.	Measure per-task usage; optimize safely.
Cross-user data exposure	Unsafe caching or authorization design.	Enforce access controls and tenant isolation.
Retry storm	Many clients repeatedly retry a failing service.	Backoff, jitter, retry limits, circuit breaking.
Quality regression	Truncation, aggressive optimization, model changes.	Re-run evaluations and compare against baseline.

A system can fail under load even when every component passes isolated functional tests.

↑ Back to contents

9. Scalability vs Related Concepts

Concept	Main Question
Deployment	How do we release the system for use?
Scalability	How do we accommodate changing workload?
Performance	How fast and efficiently does the system operate?
Reliability	Does the system behave correctly and consistently?
Availability	Is the service accessible when needed?
Observability	Can we understand its behavior and failures?
Optimization	Can we improve cost, speed, or resource use?
LLMOps	How do we manage the system across its lifecycle?

These concepts overlap but are not interchangeable. A system can be highly available at low traffic yet fail to scale under a sudden increase in demand.

↑ Back to contents

10. Industry Applications

• E-commerce: Handle seasonal peaks in shopping assistance and order inquiries.
• Customer support: Support increased ticket volume without uncontrolled response delays.
• Enterprise search: Serve many simultaneous retrieval and summarization requests.
• Software engineering: Scale code-analysis and testing agents across teams.
• Document processing: Process large backlogs through asynchronous pipelines.
• Education: Support many learners using AI tutoring services at the same time.
• Operations: Handle bursts of incident alerts and diagnostic tasks.

Different applications need different scaling strategies. Interactive chat prioritizes responsiveness, while overnight document processing may prioritize total throughput and cost.

↑ Back to contents

11. Industry Case: Scaling a Retail Shopping Assistant

Illustrative industry scenario, not a documented deployment by a named retailer.

The business problem

An online retailer expects demand for its AI assistant to increase sharply during a major sale.

The assistant answers product questions, retrieves catalog details, and checks order status for authenticated customers.

Initial architecture

A single application instance calls one model endpoint and a product-search service.

It performs adequately during the pilot but has limited concurrency, no request queue, and insufficient visibility into bottlenecks.

Improved design

1. Measure baseline latency, throughput, quality, and cost.
2. Separate customer-facing and background workloads.
3. Scale stateless application instances behind a load balancer.
4. Introduce rate limits and bounded admission control.
5. Cache eligible public catalog information with defined freshness limits.
6. Optimize product retrieval queries.
7. Estimate model token demand and verify provider quotas.
8. Use bounded queues for non-urgent processing.
9. Configure monitoring for p95 latency, errors, queue depth, and cost.
10. Load-test the complete system before the sale.

Failure A: Model quota exceeded

Increasing application replicas causes more simultaneous model requests, but the shared provider quota remains unchanged.

Response: Apply admission control and capacity planning rather than assuming more replicas solve the problem.

Failure B: Unsafe caching

A personalized order response is cached and accidentally returned to another customer.

Response: Enforce authorization, use appropriate cache keys and isolation, and avoid caching sensitive results where safe reuse cannot be guaranteed.

Failure C: Retry storm

A provider slowdown triggers repeated retries, further increasing load.

Response: Limit retries, use backoff with jitter, and implement overload protection.

Failure D: Faster but less accurate answers

The team shortens retrieval context to reduce latency, but the assistant begins omitting important product conditions.

Response: Compare quality evaluations before and after optimization.

Industry insight: Successful scaling preserves the value of the application as demand increases. Higher throughput is not success if answers become unreliable, unauthorized, or economically unsustainable.

↑ Back to contents

12. Engineering Design Decisions

1. What is the expected average and peak workload?
2. Which operations are interactive and which can be asynchronous?
3. What are the latency and availability objectives?
4. Where is the current bottleneck?
5. Which components can scale horizontally?
6. What provider quotas limit total capacity?
7. Which results can safely be cached?
8. What is the acceptable maximum queue length or wait time?
9. How will overload be communicated to users?
10. What quality checks must remain in place under heavy load?
11. How will per-user and per-tenant fairness be maintained?
12. What is the maximum acceptable cost per successful task?
13. How will scaling decisions be tested before production?

Decision principle: Scale the constrained part of the system, not the most visible part.

↑ Back to contents

13. Hands-On Mini Project

Project: Load-Test a Simulated AI Shopping Assistant

Objective: Measure how a simple AI application behaves as request volume and concurrency increase.

No real model API or paid cloud infrastructure is required. A simulated model service is sufficient for the exercise.

Step 1: Create the simulated service

Build a simple endpoint that accepts a product question and returns a fictional response after a configurable delay.

Step 2: Define workload profiles

• Light: 5 concurrent users.
• Medium: 20 concurrent users.
• Heavy: 50 concurrent users.

These are illustrative test levels, not production capacity recommendations.

Step 3: Record baseline performance

Measure successful requests, average latency, p95 latency, and error rate.

Step 4: Add a concurrency limit

Allow only a defined number of simulated model calls at once.

Step 5: Add a bounded queue

Queue eligible requests up to a maximum depth and reject excess work explicitly.

Step 6: Introduce caching

Cache only safe, reusable public product information. Do not cache personalized customer records.

Step 7: Simulate failures

• Slow model responses.
• Provider rate-limit errors.
• Sudden traffic burst.
• Retrieval-service delay.
• Exhausted queue capacity.

Step 8: Compare designs

Compare the original service with the version using bounded concurrency, queueing, and caching.

Step 9: Prepare a scaling report

Document bottlenecks, performance changes, quality checks, and the remaining limits of the design.

↑ Back to contents

14. Evaluation and Acceptance Criteria

Measure	Evaluation Question	Exercise Goal
Throughput	How many requests complete per second?	Measure at each load level.
p95 latency	How slow are the slower requests?	Compare before and after changes.
Error rate	How often does processing fail?	Distinguish failures from deliberate overload rejection.
Queue depth	Is work accumulating?	Remain within the configured bound.
Rate-limit handling	Are provider limits respected?	No uncontrolled retry storms.
Cache correctness	Are reused results appropriate?	No cross-user data exposure.
Task quality	Are answers still correct?	No regression on critical test cases.
Cost estimate	How much does each successful task cost?	Record and compare alternatives.

Students should explain trade-offs rather than claiming that one scaling technique always improves every metric.

For example, a queue may reduce immediate failures but increase waiting time. Caching may reduce load but introduce freshness risks.

↑ Back to contents

15. Connected Glossary Terms

• #001 — Token: Token volume influences inference demand and cost.
• #003 — Vector Database: Retrieval performance can limit throughput.
• #005 — Context Window: Context size affects workload characteristics.
• #007 — Evals: Verify quality remains acceptable at scale.
• #011 — Observability: Reveal latency, errors, and bottlenecks.
• #014 — Workflows: Coordinate asynchronous and multi-step workloads.
• #019 — Deployment: Establishes the production environment that must support growing demand.

Previous lesson: #019 — Deployment.

Next lesson: #021 — Production Observability: Monitoring AI Systems Under Real-World Conditions.

↑ Back to contents

16. Living Update

Editorial baseline: November 2026.

AI scalability practices evolve with inference infrastructure, model APIs, hardware, and application frameworks.

Areas to monitor include:

• Provider quotas and throughput tiers.
• Inference batching and scheduling.
• Model routing and capacity allocation.
• Autoscaling for GPU and CPU workloads.
• Long-context inference performance.
• Agent concurrency and tool-call budgets.
• Prompt and retrieval caching.
• Cost-aware load balancing.
• Evaluation under production-like load.

Living Glossary note: Revisit this entry as inference services and deployment practices change. Measure actual system behavior rather than relying on generic performance claims.

↑ Back to contents

17. Key Takeaway

Scalability enables AI systems to handle growing demand while preserving acceptable quality, responsiveness, reliability, and cost.

The engineering objective is not maximum traffic at any price. It is sustainable service delivery at the required workload.

One Action: Take an AI application, simulate three levels of demand, measure p95 latency and error rate, and identify the first component that becomes a bottleneck.

Think like an AI engineer: If user demand doubles tomorrow, which component will fail first—and how would you know?

↑ Back to contents

18. References and Further Reading

• Google — Site Reliability Engineering
• AWS Well-Architected Framework
• Kubernetes — Deployments
• Kubernetes — Horizontal Pod Autoscaling
• OpenTelemetry Documentation
• Locust — Load Testing Documentation
• Grafana k6 — Performance Testing

↑ Back to contents

AIYING — Accomplishing with AI

The Language of AI Engineering

One Term. One Industry Connection. One Practical Insight. One Action.

Chapter 4 | Lesson 20 | SCALABILITY

Understand the Term. Connect the Technology. Apply the Knowledge.

↑ Return to top