AI ENGINEERING — LIVING GLOSSARY #020
Lesson 20 of 24 | Chapter 4
SCALABILITY
Designing AI Systems for Growing Demand
Chapter 4: Deploying, Scaling, and Operating AI Systems
AIYING — Accomplishing with AI
The AI Assistant That Could Not Survive a Sale
An e-commerce company launches an AI shopping assistant. During a pilot, the assistant answers product questions, retrieves catalog information, and explains order status.
The pilot serves a small number of shoppers successfully. The company then announces a major promotional sale.
Thousands of customers arrive within minutes.
Suddenly, some responses take 30 seconds. Others fail. The model provider reports rate-limit errors. Product search becomes slow. Repeated requests increase costs.
The assistant has not necessarily become less intelligent. Its system capacity is inadequate for the workload.
Scaling an AI system requires understanding the complete request path, not simply increasing the number of model calls.
↑ Back to contents1. What Is Scalability?
Scalability is the ability of a system to handle increased or changing workload by adjusting resources, architecture, or execution methods while maintaining specified service objectives.
For an AI application, the workload may grow through:
- More concurrent users.
- More requests per second.
- Longer prompts and generated responses.
- Larger document collections.
- More retrieval operations.
- Additional tool calls or agent steps.
- More complex reasoning tasks.
Scalability is not unlimited capacity. Every system has constraints, including provider quotas, compute resources, network limits, and budgets.
A scalable design anticipates these limits and makes growth manageable.
2. Why AI Scalability Is Different
Conventional web services often perform relatively predictable operations. AI workloads can vary substantially from request to request.
One request might contain 200 input tokens and produce a short answer. Another might contain thousands of input tokens, retrieve documents, invoke multiple tools, and generate a long response.
Important AI-specific constraints include:
- Token demand: Prompt and completion lengths affect processing time and consumption.
- Model inference: Large-model execution can be computationally expensive.
- Provider limits: Requests-per-minute and tokens-per-minute quotas may constrain throughput.
- Retrieval: Vector search and database access introduce additional latency.
- Agent loops: One user request may trigger many model and tool calls.
- Variable quality: Faster or cheaper execution strategies may change answer quality.
- External dependencies: A slow catalog API can delay the entire response.
The slowest or most constrained component can determine end-to-end performance.
↑ Back to contents3. Essential Scaling Metrics
| Metric | Meaning | Engineering Question |
|---|---|---|
| Throughput | Completed requests or tasks per unit of time. | How much work can the system finish? |
| Concurrency | Requests being processed simultaneously. | How many tasks are in progress? |
| Latency | Time taken to complete a request. | How long does the user wait? |
| p95 latency | 95th-percentile response time. | How slow are the slower requests? |
| Error rate | Fraction of requests that fail. | Is overload causing failures? |
| Token consumption | Input and output tokens used. | What inference capacity is required? |
| Queue depth | Work waiting to be processed. | Is demand exceeding service capacity? |
| Cost per successful task | Total relevant cost divided by successful outcomes. | Is scaling economically sustainable? |
| Task quality | Accuracy or success against evaluation criteria. | Does quality remain acceptable under load? |
Concurrency is not throughput
A service may have 100 requests in progress while completing only 10 per second.
Under relatively stable conditions, Little's Law gives a useful relationship:
Average in-flight work = Throughput × Average time in system L = λ × W
For example, at 20 completed requests per second and an average end-to-end time of 5 seconds, average in-flight work is approximately 100 requests.
This relationship helps with capacity estimation, but actual systems must account for variable workloads, queues, bursts, and failure conditions.
↑ Back to contents4. Vertical and Horizontal Scaling
Vertical scaling — Scale up
Increase the capacity of an individual machine or service instance, such as adding CPU, memory, or GPU resources.
Advantage: May be straightforward for a bounded workload.
Limitation: Hardware and service limits eventually constrain growth.
Horizontal scaling — Scale out
Add more service instances and distribute work among them.
Advantage: Can support greater capacity and resilience when components are designed for distribution.
Limitation: Requires load balancing, shared-state design, coordination, and careful dependency management.
Scaling down
Reduce allocated capacity when demand falls, where supported, to avoid unnecessary cost.
Load Balancer
|
+---------+---------+
| | |
v v v
App 1 App 2 App 3
| | |
+---------+---------+
|
v
Model / Data APIs
Adding application instances does not automatically increase the quota of a shared model provider or the capacity of a shared database.
↑ Back to contents5. Scalable AI Architecture
A scalable retail AI assistant may use:
Customer Requests
|
v
API Gateway / Rate Limits
|
v
Load Balancer
|
+-------+-------+
| | |
v v v
AI App AI App AI App
| | |
+-------+-------+
|
+---------+---------+
| | |
v v v
Cache Retrieval Task Queue
| | |
| v v
| Vector DB Workers
| |
+---------+---------+
|
v
Model Gateway
|
v
Model Provider(s)
Cross-cutting:
Authorization | Tracing | Evals
Budgets | Alerts | Fallbacks
Stateless application services
Where practical, keep application instances stateless and store necessary shared session or task state in suitable external systems.
Model gateway
A model gateway can centralize provider routing, rate-limit handling, observability, and policy enforcement.
Background processing
Long-running or non-interactive tasks can be processed asynchronously, with explicit task status and completion handling.
Dependency isolation
Separate components so that an overloaded reporting workload does not necessarily block customer-facing requests.
↑ Back to contents6. Practical Scaling Techniques
A. Load balancing
Distribute requests across available application instances. Apply health checks to avoid routing traffic to unhealthy instances.
B. Autoscaling
Adjust service capacity using suitable demand signals such as CPU utilization, request concurrency, or queue depth.
Autoscaling is not instantaneous. Plan for startup delays and sudden traffic spikes.
C. Caching
Reuse eligible results rather than repeating expensive computation or retrieval.
For example, public product descriptions may be cached. Personalized order details require stricter access controls, isolation, and freshness rules.
D. Batching
Combine compatible operations where the model or processing service supports efficient batch execution.
Batching can improve throughput but may increase waiting time for individual requests.
E. Queues and backpressure
Place non-urgent tasks in bounded queues and control how quickly workers consume them.
When capacity is exhausted, return an explicit overload response or apply a defined admission policy rather than accepting unlimited work.
F. Rate limiting
Control requests by user, tenant, application, or other appropriate scope to protect shared resources.
G. Token budgeting
Reduce unnecessary context and cap output lengths where appropriate, while checking that answer quality remains acceptable.
H. Model routing
Use different approved models for different task requirements, based on measured quality, latency, and cost.
I. Retrieval optimization
Improve indexing, filtering, query design, and retrieval configuration rather than assuming all delays originate in the language model.
J. Graceful degradation
When capacity is limited, offer simpler verified functionality, defer non-urgent work, or provide clear fallback responses.
7. Capacity Planning
Capacity planning estimates the resources required for a projected workload.
Illustrative token-demand calculation
Assume an AI assistant receives:
- 30 requests per second.
- An average of 1,000 input tokens per request.
- An average of 300 output tokens per request.
Average tokens per request = 1,000 + 300 = 1,300 tokens Average token demand = 30 × 1,300 = 39,000 tokens/second Equivalent average demand = 2,340,000 tokens/minute
This is an aggregate planning estimate, not a claim that a particular model service can deliver that throughput.
Real providers may enforce separate input, output, request, and concurrency limits. Output generation can also impose different performance constraints from input processing.
Why averages are insufficient
A promotional sale may produce short bursts far above average demand. Capacity planning should therefore include peak traffic, request-size distributions, provider quotas, retries, and safety margins.
Measure actual usage before selecting capacity.
↑ Back to contents8. Bottlenecks and Failure Modes
| Problem | Likely Cause | Possible Response |
|---|---|---|
| High model latency | Long requests, inference saturation. | Measure token demand; optimize or add permitted capacity. |
| Rate-limit errors | Provider quota exceeded. | Admission control, bounded retries, capacity planning. |
| Slow retrieval | Database contention or inefficient queries. | Indexing, filtering, query tuning, caching. |
| Growing queue | Arrival rate exceeds processing rate. | Add effective worker capacity or reduce admitted demand. |
| High cost | Repeated calls, excessive tokens, inefficient routing. | Measure per-task usage; optimize safely. |
| Cross-user data exposure | Unsafe caching or authorization design. | Enforce access controls and tenant isolation. |
| Retry storm | Many clients repeatedly retry a failing service. | Backoff, jitter, retry limits, circuit breaking. |
| Quality regression | Truncation, aggressive optimization, model changes. | Re-run evaluations and compare against baseline. |
A system can fail under load even when every component passes isolated functional tests.
↑ Back to contents9. Scalability vs Related Concepts
| Concept | Main Question |
|---|---|
| Deployment | How do we release the system for use? |
| Scalability | How do we accommodate changing workload? |
| Performance | How fast and efficiently does the system operate? |
| Reliability | Does the system behave correctly and consistently? |
| Availability | Is the service accessible when needed? |
| Observability | Can we understand its behavior and failures? |
| Optimization | Can we improve cost, speed, or resource use? |
| LLMOps | How do we manage the system across its lifecycle? |
These concepts overlap but are not interchangeable. A system can be highly available at low traffic yet fail to scale under a sudden increase in demand.
↑ Back to contents10. Industry Applications
- E-commerce: Handle seasonal peaks in shopping assistance and order inquiries.
- Customer support: Support increased ticket volume without uncontrolled response delays.
- Enterprise search: Serve many simultaneous retrieval and summarization requests.
- Software engineering: Scale code-analysis and testing agents across teams.
- Document processing: Process large backlogs through asynchronous pipelines.
- Education: Support many learners using AI tutoring services at the same time.
- Operations: Handle bursts of incident alerts and diagnostic tasks.
Different applications need different scaling strategies. Interactive chat prioritizes responsiveness, while overnight document processing may prioritize total throughput and cost.
↑ Back to contents11. Industry Case: Scaling a Retail Shopping Assistant
Illustrative industry scenario, not a documented deployment by a named retailer.
The business problem
An online retailer expects demand for its AI assistant to increase sharply during a major sale.
The assistant answers product questions, retrieves catalog details, and checks order status for authenticated customers.
Initial architecture
A single application instance calls one model endpoint and a product-search service.
It performs adequately during the pilot but has limited concurrency, no request queue, and insufficient visibility into bottlenecks.
Improved design
- Measure baseline latency, throughput, quality, and cost.
- Separate customer-facing and background workloads.
- Scale stateless application instances behind a load balancer.
- Introduce rate limits and bounded admission control.
- Cache eligible public catalog information with defined freshness limits.
- Optimize product retrieval queries.
- Estimate model token demand and verify provider quotas.
- Use bounded queues for non-urgent processing.
- Configure monitoring for p95 latency, errors, queue depth, and cost.
- Load-test the complete system before the sale.
Failure A: Model quota exceeded
Increasing application replicas causes more simultaneous model requests, but the shared provider quota remains unchanged.
Response: Apply admission control and capacity planning rather than assuming more replicas solve the problem.
Failure B: Unsafe caching
A personalized order response is cached and accidentally returned to another customer.
Response: Enforce authorization, use appropriate cache keys and isolation, and avoid caching sensitive results where safe reuse cannot be guaranteed.
Failure C: Retry storm
A provider slowdown triggers repeated retries, further increasing load.
Response: Limit retries, use backoff with jitter, and implement overload protection.
Failure D: Faster but less accurate answers
The team shortens retrieval context to reduce latency, but the assistant begins omitting important product conditions.
Response: Compare quality evaluations before and after optimization.
12. Engineering Design Decisions
- What is the expected average and peak workload?
- Which operations are interactive and which can be asynchronous?
- What are the latency and availability objectives?
- Where is the current bottleneck?
- Which components can scale horizontally?
- What provider quotas limit total capacity?
- Which results can safely be cached?
- What is the acceptable maximum queue length or wait time?
- How will overload be communicated to users?
- What quality checks must remain in place under heavy load?
- How will per-user and per-tenant fairness be maintained?
- What is the maximum acceptable cost per successful task?
- How will scaling decisions be tested before production?
Decision principle: Scale the constrained part of the system, not the most visible part.
↑ Back to contents13. Hands-On Mini Project
Project: Load-Test a Simulated AI Shopping Assistant
Objective: Measure how a simple AI application behaves as request volume and concurrency increase.
No real model API or paid cloud infrastructure is required. A simulated model service is sufficient for the exercise.
Step 1: Create the simulated service
Build a simple endpoint that accepts a product question and returns a fictional response after a configurable delay.
Step 2: Define workload profiles
- Light: 5 concurrent users.
- Medium: 20 concurrent users.
- Heavy: 50 concurrent users.
These are illustrative test levels, not production capacity recommendations.
Step 3: Record baseline performance
Measure successful requests, average latency, p95 latency, and error rate.
Step 4: Add a concurrency limit
Allow only a defined number of simulated model calls at once.
Step 5: Add a bounded queue
Queue eligible requests up to a maximum depth and reject excess work explicitly.
Step 6: Introduce caching
Cache only safe, reusable public product information. Do not cache personalized customer records.
Step 7: Simulate failures
- Slow model responses.
- Provider rate-limit errors.
- Sudden traffic burst.
- Retrieval-service delay.
- Exhausted queue capacity.
Step 8: Compare designs
Compare the original service with the version using bounded concurrency, queueing, and caching.
Step 9: Prepare a scaling report
Document bottlenecks, performance changes, quality checks, and the remaining limits of the design.
↑ Back to contents14. Evaluation and Acceptance Criteria
| Measure | Evaluation Question | Exercise Goal |
|---|---|---|
| Throughput | How many requests complete per second? | Measure at each load level. |
| p95 latency | How slow are the slower requests? | Compare before and after changes. |
| Error rate | How often does processing fail? | Distinguish failures from deliberate overload rejection. |
| Queue depth | Is work accumulating? | Remain within the configured bound. |
| Rate-limit handling | Are provider limits respected? | No uncontrolled retry storms. |
| Cache correctness | Are reused results appropriate? | No cross-user data exposure. |
| Task quality | Are answers still correct? | No regression on critical test cases. |
| Cost estimate | How much does each successful task cost? | Record and compare alternatives. |
Students should explain trade-offs rather than claiming that one scaling technique always improves every metric.
For example, a queue may reduce immediate failures but increase waiting time. Caching may reduce load but introduce freshness risks.
↑ Back to contents15. Connected Glossary Terms
- #001 — Token: Token volume influences inference demand and cost.
- #003 — Vector Database: Retrieval performance can limit throughput.
- #005 — Context Window: Context size affects workload characteristics.
- #007 — Evals: Verify quality remains acceptable at scale.
- #011 — Observability: Reveal latency, errors, and bottlenecks.
- #014 — Workflows: Coordinate asynchronous and multi-step workloads.
- #019 — Deployment: Establishes the production environment that must support growing demand.
Previous lesson: #019 — Deployment.
Next lesson: #021 — Production Observability: Monitoring AI Systems Under Real-World Conditions.
↑ Back to contents16. Living Update
Editorial baseline: November 2026.
AI scalability practices evolve with inference infrastructure, model APIs, hardware, and application frameworks.
Areas to monitor include:
- Provider quotas and throughput tiers.
- Inference batching and scheduling.
- Model routing and capacity allocation.
- Autoscaling for GPU and CPU workloads.
- Long-context inference performance.
- Agent concurrency and tool-call budgets.
- Prompt and retrieval caching.
- Cost-aware load balancing.
- Evaluation under production-like load.
Living Glossary note: Revisit this entry as inference services and deployment practices change. Measure actual system behavior rather than relying on generic performance claims.
↑ Back to contents17. Key Takeaway
The engineering objective is not maximum traffic at any price. It is sustainable service delivery at the required workload.
One Action: Take an AI application, simulate three levels of demand, measure p95 latency and error rate, and identify the first component that becomes a bottleneck.
Think like an AI engineer: If user demand doubles tomorrow, which component will fail first—and how would you know?
↑ Back to contents18. References and Further Reading
- Google — Site Reliability Engineering
- AWS Well-Architected Framework
- Kubernetes — Deployments
- Kubernetes — Horizontal Pod Autoscaling
- OpenTelemetry Documentation
- Locust — Load Testing Documentation
- Grafana k6 — Performance Testing
