Showing posts with label Model Routing. Show all posts

Thursday, November 12, 2026

AI Engineering Living Glossary #022 — Cost Optimization: Engineering AI Systems for Sustainable Economics

AI Engineering — Living Glossary #022

Chapter 4 Lesson 22 of 24 Production AI

Cost Optimization: Engineering AI Systems for Sustainable Economics

From Building AI Applications to Running Reliable Production Systems

AIYING — Accomplishing with AI

Understand the Term. Connect the Technology. Apply the Knowledge.

The central idea: The goal is not the cheapest model call. The goal is the lowest sustainable cost per successful, quality-assured business outcome.

1. The Industry Problem: When Successful AI Becomes Expensive

An online retailer launches an AI customer-support assistant. The assistant answers questions about order tracking, returns, refunds, delivery policies, and products.

The pilot is successful. Users receive helpful answers and the business decides to expand the service.

Then a new problem appears. AI operating costs grow rapidly.

Engineers investigate and discover several causes:

Every question is sent to a premium language model, including simple tracking questions.
Long system prompts and conversation histories are repeatedly transmitted.
The retrieval system returns more documents than needed.
Some tool calls repeat information already available.
Failed requests trigger expensive retries.
Costs are tracked at the provider-account level, not by customer journey or business outcome.

The team initially asks, "How can we reduce the token bill?"

A better engineering question is:

How can we redesign the AI workflow to complete more customer tasks successfully, at lower total cost, while maintaining service quality?

This change of question is the foundation of AI cost optimization.

↑ Back to contents

2. What Is AI Cost Optimization?

AI cost optimization is the systematic practice of designing, measuring, and operating AI systems so that they deliver required business outcomes at an economically sustainable total cost.

It involves decisions about:

Which models to use for different tasks
How much input and output context is necessary
When to retrieve information or call tools
What results can safely be cached
How workloads are routed, batched, and scaled
When a human or deterministic system is preferable
How to measure the value and cost of completed work

Working definition: Cost optimization is not indiscriminate cost cutting. It is the engineering of efficient resource use subject to quality, latency, security, and reliability requirements.

A low-cost response that gives the wrong refund policy may create more expense than a higher-cost response that resolves the issue correctly.

↑ Back to contents

3. Why Cost Optimization Matters

Economic sustainability

A useful prototype is not necessarily a viable production product. As adoption grows, variable AI costs can materially affect operating margins.

Scalability

Traffic growth multiplies requests, tokens, retrieval queries, and tool calls. Cost-efficient architecture makes scaling more sustainable.

Product accessibility

Lower cost per successful task can make AI assistance affordable for more customers and organizations.

Engineering discipline

Measuring cost alongside quality and reliability encourages explicit architectural choices rather than relying on one expensive default configuration.

Business accountability

Cost information becomes actionable when associated with product features, tenants, workflows, and successful outcomes.

↑ Back to contents

4. Understanding AI Cost Economics

Generative AI introduces cost drivers that differ from many conventional web applications.

4.1 Model usage costs

For usage-priced model APIs, costs may depend on input tokens, output tokens, cached tokens, and other provider-specific features.

Model cost = (input tokens × input rate) + (output tokens × output rate) + other applicable charges

Rates must use consistent units. For example, if a provider quotes prices per million tokens, divide token counts by one million before multiplying.

4.2 Retrieval costs

Retrieval-augmented generation may incur costs for embeddings, index storage, vector searches, database operations, and data transfer.

4.3 Tool and agent costs

Agentic workflows may perform multiple model calls, external API requests, database operations, and retries before producing a final answer.

4.4 Infrastructure costs

Self-hosted inference introduces GPU or accelerator capacity, CPU, memory, storage, networking, and engineering operations costs.

4.5 Quality and failure costs

Incorrect answers, failed workflows, escalations, rework, and unnecessary human reviews can also increase the total cost of delivering an outcome.

Cost per successful task = Total relevant operating cost ÷ Number of successfully completed tasks

This metric requires a clear definition of success. A generated answer alone is not necessarily a completed task.

Important: Token price is not total cost of ownership. A complete economic analysis also considers infrastructure, data, tooling, monitoring, engineering effort, and operational consequences.

↑ Back to contents

5. Eight Engineering Levers for Cost Optimization

Lever 1 — Model selection

Use the least expensive model that reliably meets the quality requirements of a specific task.

A straightforward order-status lookup may need deterministic software and a concise response, rather than a sophisticated reasoning model.

Lever 2 — Model routing

Route tasks according to complexity and risk.

Simple structured lookup: conventional API or template
Routine language task: smaller capable model
Complex reasoning: stronger model
High-risk exception: human review

Routing introduces its own classification errors, operational complexity, and evaluation requirements.

Lever 3 — Prompt and context efficiency

Remove irrelevant instructions, duplicated context, and unnecessary history while preserving information essential to correct task execution.

Shorter is not automatically better. A prompt that becomes ambiguous may cause retries or errors.

Lever 4 — Retrieval optimization

Retrieve relevant passages instead of entire documents. Use filters, appropriate chunk sizes, reranking where justified, and a controlled context budget.

Measure answer quality as well as retrieved-token volume.

Lever 5 — Caching

Reuse safe, stable results when appropriate.

Cache public policy explanations with version awareness.
Cache embeddings for unchanged source material.
Use provider-supported prompt caching when available.
Avoid reusing personalized answers across customers.

Cache keys must account for tenant, permissions, data version, locale, and other relevant context.

Lever 6 — Batching and asynchronous processing

Group suitable non-urgent tasks to improve throughput or access lower-cost processing options when supported.

Batching is often appropriate for document enrichment, classification, and offline analysis. It may be unsuitable for interactive tasks with strict response-time requirements.

Lever 7 — Infrastructure right-sizing

Match provisioned capacity to workload demand. For self-hosted inference, evaluate utilization, throughput, queueing, accelerator memory, and availability requirements.

For managed APIs, monitor provider quotas and rate limits. Autoscaling an application does not automatically increase the model provider's capacity.

Lever 8 — Workflow simplification

Remove unnecessary model calls, tool loops, duplicated validation, and avoidable retries.

Often the best optimization is architectural: Do not ask a language model to perform work that a reliable database query, rule, or existing API can complete more simply.

↑ Back to contents

6. Designing a Cost-Aware AI Architecture

Consider the following reference workflow:

User request
↓
Authentication and policy checks
↓
Intent classification and risk assessment
↓
Deterministic API, cache, or model route
↓
Targeted retrieval and required tool calls
↓
Response validation and safety checks
↓
User response or human escalation
↓
Quality, latency, usage, and cost telemetry

Architecture principle: decide before spending

Before each expensive operation, ask whether it is necessary for the current task.

Architecture principle: preserve traceability

Record a workflow identifier, model version, usage counters, tool-call counts, latency, estimated or billed cost, and evaluation outcome.

Architecture principle: enforce limits

Use request budgets, output-token limits, retry ceilings, timeouts, rate limits, and per-tenant spending controls.

Architecture principle: fail safely

If the budget or service limit is reached, provide a safe fallback, defer non-urgent work, or escalate instead of silently producing an unsupported answer.

↑ Back to contents

7. Cost Optimization Versus Related Concepts

Concept	Main question	Relationship
Cost optimization	What does a successful outcome cost?	Balances economics with required outcomes.
Performance optimization	How fast and efficiently does the system respond?	Lower latency can increase or decrease cost.
Scalability	Can the system handle growing demand?	Growth requires sustainable unit economics.
Observability	What is happening inside the system?	Provides evidence needed to find waste.
FinOps	How do teams manage technology spending and value?	Adds financial accountability and governance.
LLMOps	How are language-model applications operated?	Integrates cost with quality and lifecycle controls.

These disciplines reinforce one another, but they are not interchangeable.

↑ Back to contents

8. Metrics That Make Cost Visible

Useful cost metrics must connect resource consumption to service delivery and business results.

Metric	What it reveals
Cost per request	Average spend for each request received.
Cost per successful task	Spend relative to correctly completed work.
Input/output tokens per task	Where context and generation costs arise.
Model calls per workflow	Potential over-orchestration or retries.
Cache hit rate	How often eligible work is reused.
Task success rate	Whether the system accomplishes its purpose.
p95 end-to-end latency	Response time experienced by slower requests.
Escalation rate	How often automation requires human support.
Spend per tenant or feature	Which users and workflows drive expenditure.

Example of a misleading metric

Suppose System A costs $0.01 per attempt and succeeds on 50% of attempts.

System B costs $0.015 per attempt and succeeds on 90% of attempts.

For a simplified comparison using independent attempts and no additional costs:

A: $0.01 ÷ 0.50 = $0.020 per successful attempt B: $0.015 ÷ 0.90 ≈ $0.0167 per successful attempt

System B is more expensive per attempt but cheaper per successful attempt.

Actual production comparisons must account for workflow differences, human escalation, retries, and outcome quality.

↑ Back to contents

9. Protecting Quality, Safety, and Reliability

Cost changes should be treated as engineering changes, not merely financial adjustments.

Quality regression testing

Compare optimized and baseline configurations using representative evaluation datasets.

Grounding and factual accuracy

Ensure that shorter prompts or smaller retrieval contexts do not remove necessary evidence.

Security and privacy

Never reduce cost by weakening authorization, exposing customer information through shared caches, or skipping essential security checks.

Reliability under load

Test timeouts, retries, rate limits, queue depth, and fallback behavior at realistic demand levels.

Human oversight

Preserve escalation paths for high-risk or ambiguous decisions, even when human review increases direct operating cost.

Non-negotiable principle: An optimization that violates required safety, correctness, privacy, or service-level standards is not a successful optimization.

↑ Back to contents

10. Where Cost Optimization Matters in Industry

Retail and e-commerce

Route order-status queries to existing APIs while reserving model reasoning for complex customer-support situations.

Banking and financial services

Use structured systems for balances and transactions; apply AI selectively to explanations and document workflows, with appropriate security and human oversight.

Healthcare administration

Reduce repetitive document processing costs without compromising privacy or required review.

Manufacturing

Batch non-urgent maintenance report summarization and use targeted retrieval for equipment documentation.

Education

Manage the cost of tutoring interactions while maintaining instructional quality, accessibility, and learner safety.

Enterprise knowledge assistants

Optimize document indexing, access-controlled retrieval, context size, and repeated searches.

↑ Back to contents

11. Detailed Industry Case: The Retail Support Assistant

The following is an illustrative engineering scenario, not a reported company deployment.

Business context

A retailer processes 100,000 customer-support requests per month through an AI assistant.

The original system sends every request to one premium model, includes a long prompt, retrieves multiple documents, and uses the same workflow for simple and complex questions.

Baseline assumptions

Measure	Illustrative baseline
Monthly requests	100,000
Average AI processing cost per request	$0.08
Monthly AI processing cost	$8,000
Successfully resolved requests	80%
Successful resolutions	80,000
Cost per successful resolution	$0.10

These figures are invented for teaching purposes. They exclude human-support and other operating costs.

Step 1 — Examine request distribution

The team classifies requests into:

Simple order and delivery lookups
Routine policy questions
Complex return and refund situations
Exceptions requiring human review

Step 2 — Redesign the workflow

The team introduces deterministic order lookups, task-specific model routing, targeted retrieval, context limits, safe caching, and bounded retries.

Step 3 — Validate the new configuration

The optimized design is evaluated against a representative test set that includes difficult refunds, outdated policies, missing orders, and adversarial inputs.

Step 4 — Compare outcomes

Metric	Before	After
Monthly requests	100,000	100,000
AI processing cost per request	$0.080	$0.045
Monthly AI processing cost	$8,000	$4,500
Successful resolution rate	80%	84%
Successful resolutions	80,000	84,000
AI cost per successful resolution	$0.100	≈ $0.0536

Illustrative result: AI processing expenditure falls by $3,500 per month, or 43.75%, while successful resolution improves by four percentage points.

However, these are hypothetical results. A real implementation must establish measured performance and include the cost of routing, engineering, monitoring, infrastructure, and human escalations.

The engineering lesson

The improvement comes from changing the work allocation, not simply choosing a cheaper model.

↑ Back to contents

12. Engineering Decisions and Trade-offs

Decision	Potential benefit	Risk or trade-off
Smaller model	Lower inference cost	Possible quality regression
Shorter context	Fewer input tokens	Missing critical evidence
Caching	Fewer repeated operations	Stale or unauthorized results
Batch processing	Potentially lower unit cost	Longer completion time
Self-hosted inference	Greater deployment control	Capacity and operations burden
Deterministic workflows	Predictable cost and behavior	Less flexibility for ambiguous tasks
Reduced agent steps	Lower latency and cost	May remove necessary verification

Engineering teams should evaluate each change against explicit acceptance criteria rather than assuming that lower expenditure automatically represents progress.

↑ Back to contents

13. Hands-On Mini-Project: Build an AI Cost Optimizer

Project: Design a cost-aware routing strategy for a retail customer-support assistant.

Level: Beginner to intermediate. No paid model API is required for the first version.

Step 1 — Create a sample dataset

Prepare 40 fictional customer requests:

15 order-status questions
10 general policy questions
10 complex refund or return questions
5 sensitive or exceptional cases

For each request, record its category, expected handling method, and acceptable outcome.

Step 2 — Define routing rules

Create a simple decision table:

Request category	Proposed route
Order status	Authenticated order API
General policy	Approved policy retrieval and concise answer
Complex return	More capable model with verified context
High-risk exception	Human review

Step 3 — Assign fictional unit costs

Use illustrative cost assumptions for each route. Keep them clearly labeled as simulated rather than actual provider prices.

Step 4 — Simulate two architectures

Baseline: Every request uses the same premium model workflow.
Optimized: Requests follow the cost-aware routing table.

Step 5 — Measure the outcomes

Calculate:

Total estimated processing cost
Average cost per request
Successful task completion rate
Cost per successful task
Percentage of requests requiring human review
Incorrect-routing rate

Step 6 — Add a quality gate

Set minimum acceptable success and safety criteria. Reject any optimization that violates them.

Step 7 — Conduct a failure test

Introduce an outdated policy document, an ambiguous refund request, and a request containing another customer's order identifier.

Verify that the system does not expose private information or produce unsupported answers.

Step 8 — Produce an engineering recommendation

Write a one-page report containing:

Baseline and optimized designs
Assumptions and calculations
Quality and safety results
Recommended routing policy
Limitations and next tests

Success criterion: Demonstrate a lower simulated cost per successful task while meeting the same minimum quality and safety requirements.

↑ Back to contents

14. Connected Living Glossary

Cost optimization becomes easier to understand when connected to earlier AI engineering concepts.

#001 Token — The unit commonly used to meter language-model processing.
#004 RAG — Retrieval choices affect relevance, latency, and total cost.
#005 Context Window — Context budgets influence processing requirements.
#007 Evals — Verify that cheaper configurations remain effective.
#011 Observability — Measure model and workflow behavior.
#014 Workflows — Avoid unnecessary steps and repeated operations.
#015 AI Agents — Multi-step autonomy requires budgets and limits.
#018 Structured Outputs — Reliable structured results can reduce downstream rework.
#021 Production Observability — Connect runtime traces, cost, and quality outcomes.

↑ Back to contents

15. Living Update: What Engineers Should Keep Watching

AI cost optimization is a moving target. Model capabilities, provider pricing, context caching, batch offerings, inference engines, and hardware efficiency continue to evolve.

For every major change, review:

Current model capabilities and official prices
Input, output, and cached-token billing rules
Rate limits and processing constraints
Quality on the organization's own evaluation set
Latency and reliability under representative load
Data governance and regional requirements
Total operating cost, not just model charges

A cost-optimal architecture today may not remain optimal after model, workload, or pricing changes.

↑ Back to contents

16. Key Takeaway and One Action

Key takeaway: Sustainable AI engineering means delivering the required outcome at the lowest responsible total cost, without weakening quality, safety, reliability, or user trust.

One action: Select one AI workflow and measure its cost per successful task. Then identify one unnecessary model call, retrieval step, or repeated operation that could be eliminated or redesigned.

One Term. One Industry Connection. One Practical Insight. One Action.

↑ Back to contents

17. Authoritative References and Further Learning

The following official resources support further study. Consult current documentation for actual prices, feature availability, and implementation requirements.

OpenAI — API Pricing. Model pricing and available usage options.
OpenAI — Latency Optimization. Practical design considerations for inference efficiency.
AWS — Well-Architected Framework. Cost optimization and performance-efficiency principles.
FinOps Foundation — FinOps Framework. Managing technology value and financial accountability.
Kubernetes — Horizontal Pod Autoscaling. Scaling application workloads.
OpenTelemetry — Documentation. Instrumentation, metrics, traces, and logs.
NIST — AI Risk Management Framework. Risk-aware AI system design and operation.

↑ Back to contents

Friday, November 6, 2026

The Effective AI Engineer #012: The Engineer Who Chose the Smaller Model

The Effective AI Engineer • Case #012 • 12 of 16

The Engineer Who Chose the Smaller Model

Generating alternatives and selecting model capability against the requirements of completed work.

By Ravindra Dastikop

Applying Lateral Thinking to Create Meaningful Impact with AI

Choose model capability against the work’s requirements, and measure efficiency at the point where the work is successfully completed.

Lateral Thinking tool: Alternatives, organised around a fixed purpose.

Engineering focus: Model selection, task decomposition, evaluation, latency and total workflow cost.

Professional benefit: Learn to compare models and workflow designs against explicit quality requirements, including the cost of retries, escalation and human correction.

About this case: This is research-grounded fiction. Characters, events and comparison findings are illustrative. Technical sources were checked on 6 November 2026. No model ranking, measured saving or universal performance advantage is claimed.

1. The situation

A software company had built an internal assistant to prepare incoming support requests for investigation.

For each request, the assistant identified the product and version, classified the issue, extracted relevant details and produced a short handoff note for a support engineer.

The team used its strongest available general-purpose model for every request.

It handled difficult messages reasonably well. It also spent time interpreting straightforward submissions whose product, version and error code were already supplied in form fields.

As request volume grew, the team faced pressure to reduce operating cost and waiting time.

The initial proposal was to replace the model with a less expensive alternative.

That made the support lead uneasy.

“We chose the stronger model because these requests matter,” she said.

The team appeared to face a familiar choice:

Keep the stronger model and accept the cost, or use a smaller model and accept worse quality.

↑ Back to contents

2. The story

Nikhil, the engineer responsible for the assistant, opened a routine request.

The form contained a product identifier, a version number and an error code. The assistant had read the message, inferred the same identifiers and written a paragraph explaining its classification.

“What decision did we need the model to make here?” he asked.

“Prepare the request for the right support queue,” said Leela, the support lead.

“Does that require this model?”

“Sometimes.”

“Which part?”

They examined several requests together.

One was a completed form reporting a known error. Another contained a long conversation with conflicting version information. A third described an unfamiliar failure across two products.

The application treated all three as the same task.

A purpose that opened alternatives

Nikhil wrote a purpose on the board:

Produce a correct, evidence-linked support handoff, or explicitly identify what remains unresolved.

Then he asked the team to generate alternatives that could fulfil that purpose.

Keep the current model and shorten unnecessary output.
Use a lower effort setting where supported.
Use deterministic code for validated form fields and known mappings.
Test a smaller model for bounded language interpretation.
Send every request directly to a support engineer.
Route different request types through different combinations.

They deliberately postponed choosing.

“The smaller model is one alternative,” Nikhil said. “It is not the whole design.”

The direct replacement exposed a quality gap

They first established a baseline. Support engineers reviewed representative requests and defined what a usable handoff needed: correct product and version, relevant evidence, an appropriate queue and visible unresolved information.

A missing version was not automatically a failure. Inventing a version was.

Then they tested a smaller model as a direct replacement.

It performed acceptably on many routine requests. But several messages containing conflicting details produced clean-looking handoffs that concealed the conflict.

Leela pointed to one.

“The fields are complete. The handoff is wrong.”

Their schema validator had accepted it. All required fields existed and had the right types.

The team added checks for factual support and unresolved contradictions. Structural validity alone had not established correctness.

Short did not mean simple

Next, they introduced a route for apparently simple requests.

A request went to the smaller model when it was short and mentioned a known product. Longer requests went to the stronger model.

The first serious routing error arrived in one sentence:

“Before today’s upgrade it worked; after rolling back it still fails, although the console shows the new version.”

Short did not mean simple.

They discarded message length as the main decision rule.

The revised design used verified form completeness, supported product categories and tested request patterns to define a narrow routine path. Conflicting information and unsupported categories followed a different path. Uncertain cases remained eligible for human investigation.

Even then, the team did not assume that the router could recognise every difficult case. They included hard examples that looked routine in the evaluation set.

Separate the work before selecting the model

They also changed the division of work.

Application code copied validated identifiers and applied maintained mappings. The smaller model interpreted the remaining language and drafted a concise note from the supplied evidence.

It was instructed to preserve missing information rather than complete every field.

The stronger model remained available for selected requests requiring broader interpretation. It could also return an unresolved result. Its presence did not make insufficient evidence sufficient.

The cost report measured the attempt

The next comparison exposed another problem.

The smaller model made an incorrect classification. A retry corrected it. The system had recorded only the cheaper first call in its summary cost report.

Nikhil followed the request through the logs.

There had been an initial call, a retry, an escalation and a support engineer’s correction.

“We have priced the attempt,” he said. “We have not priced the completed work.”

They linked every attempt and review to the original request. Their comparison now included routing, retries, escalation, human handling and completion time.

The small-first cascade was unattractive for one category: requests that frequently needed the stronger model anyway. Those requests waited through two model calls.

The team routed that category directly to the stronger model.

Another category gained little from generation at all. A validated form and maintained lookup produced the required handoff.

For the bounded language-interpretation category, the smaller model remained promising.

A limited selection

During load testing, its usual response time was good, but the slowest requests still exceeded the team’s target. They investigated service capacity, queueing and fallback behaviour instead of treating the model label as a latency guarantee.

Finally, a new product version appeared in the incoming requests.

It was outside the evaluated coverage. The workflow marked it unresolved and sent it for manual handling rather than applying an old mapping.

The team approved a limited pilot.

Its design used code for established facts, a smaller model for a tested language task, a stronger model for selected complex requests and people for unresolved cases.

Leela reviewed the decision.

“So you chose the smaller model.”

“For a defined part of the work,” Nikhil replied. “We chose the workflow against the requirement.”

↑ Back to contents

3. The Lateral Thinking tool: Alternatives

Edward de Bono explains that alternatives need a reference point—a fixed point. One useful fixed point is the purpose being achieved. Holding that purpose steady allows different ways of achieving it to be explored. [1]

In this case, the fixed point was:

Produce a correct, evidence-linked support handoff, or explicitly identify what remains unresolved.

That purpose allowed the team to consider more than replacement models.

Its alternatives included code, maintained mappings, model settings, task decomposition, different models and manual handling.

The thinking sequence was:

Identify the purpose behind the current implementation.
Make that purpose explicit.
Generate different ways to achieve it.
Develop the promising options.
Evaluate them against the same requirements.

No PO statement is needed here. Deliberate alternative generation provides the change in direction.

The original method comes from de Bono’s Serious Creativity, particularly the fixed-point discussion on printed pages 126–127. The engineering architecture is a contemporary application, not a method attributed to the cited technology companies.

↑ Back to contents

4. Making the thinking visible

The team’s progress depended on changing five questions.

What must this work accomplish?

The handoff requirement became the basis for comparison. General model capability was relevant, but it was not the acceptance criterion.

Which parts require language interpretation?

Validated identifiers and maintained mappings could be handled by application code. Ambiguous descriptions required a different kind of work.

Are the output’s claims supported?

A well-formed result could still contain an incorrect version or conceal conflicting evidence. Schema compliance was only one check.

What observable conditions justify this route?

The team treated routing as a fallible component requiring its own evaluation.

What does successful completion require?

Retries, escalation and human correction could change the apparent economics of a cheaper first call.

Reusable prompt

We are designing an AI workflow for [task].

Define a fixed purpose in terms of the completed work, including what should happen when evidence is insufficient.

Generate at least five ways to achieve that purpose. Include a deterministic approach, the current model, a different model or effort setting, a decomposed workflow and a manual alternative.

For each option, identify the capability required, evidence available, likely failure modes, validation limits and operational overhead.

Propose a comparison using representative cases and difficult cases that look routine. Include quality, latency, retries, escalation, human effort and total cost per acceptable completion.

Do not assume that smaller means cheaper overall, that a larger model is always correct, or that self-reported confidence establishes correctness.

↑ Back to contents

5. Comparing the alternatives

Model and workflow options
Alternative	Potential benefit	Limitation to test
Current stronger model for every request	Simple architecture and broad capability.	May spend unnecessary effort on bounded tasks.
Current model with adjusted effort or shorter output	May improve efficiency without another model integration.	Capability and quality can change with settings.
Smaller model as a direct replacement	Potentially lower inference cost or latency.	May fail on important request categories.
Code plus a smaller model for a bounded task	Separates established facts from language interpretation.	Requires maintained mappings and clear task boundaries.
Routed combination of models	Can match different requests to different capabilities.	Adds routing errors, monitoring and operational complexity.
Manual preparation	Flexible handling of exceptional cases.	Requires human capacity and introduces waiting.

The fictional team selected a combination. Another workload might justify keeping one model.

The title describes a decision within the case. It is not a recommendation to favour smaller models in general.

A note on terminology: Here, “smaller” describes the fictional candidate. For commercial models whose parameter counts are undisclosed, engineers should describe observable properties—capability, price, latency and supported features—rather than infer physical model size.

↑ Back to contents

6. Research evidence and limits

Model selection is workload-specific

Anthropic’s model-selection guidance recommends considering capability, speed, cost and effort settings, then testing actual prompts and data. It describes both efficiency-first and capability-first starting approaches. [2]

This supports comparing candidates against the task rather than assuming one starting strategy always wins.

Routing introduces its own engineering tradeoffs

Microsoft’s Azure Architecture Center discusses task fit, single-model and multi-model designs, and manual versus automatic selection. It also identifies predictability, observability and architectural complexity as routing considerations. [3]

Routing therefore needs evidence of value beyond a lower advertised model price.

Evaluate the system that completes the task

Anthropic’s evaluation guidance distinguishes attempts, traces and outcomes. It recommends multiple trials where outputs vary and explains that an agent evaluation assesses the model working with its surrounding harness. It also identifies latency, usage, cost and errors as useful baseline measures. [4]

The proposition to test

A bounded task may be served adequately by a less demanding model configuration, provided the complete workflow meets its quality and operating requirements.

These sources support the evaluation approach. They do not establish the fictional team’s results or prove that a smaller model will improve another application.

↑ Back to contents

7. Practice and evaluation

Accessible exercise

Choose one recurring AI task.

Define an acceptable completed result.
Identify unacceptable errors separately from minor defects.
Mark which inputs are already established facts.
Identify which steps genuinely require language interpretation.
Generate five alternatives before selecting one.
Include an option that uses no generative model.
Decide what happens when the available information is insufficient.

Technical exercise

Build a controlled comparison of a small set of plausible configurations.

Record the model identifier, settings, prompt and application version.
Use representative workload cases.
Maintain a separate challenge set.
Include difficult requests that appear routine.
Keep development examples separate from held-out evaluation.
Use comparable evidence and tool access.
Give candidates a documented development budget.
Distinguish a direct model comparison from a redesigned workflow comparison.
Validate factual support as well as output structure.
Test missing information, contradictions and unsupported categories.
Record every attempt, escalation and human correction.
Test realistic load and service failures.

Measure quality before declaring savings

Define minimum requirements for important case categories. Do not allow a strong average result to conceal an unacceptable error in a consequential category.

Acceptable completion rate.
Unsupported factual claims.
Incorrect routing.
Appropriate unresolved outcomes.
Unnecessary escalation.
Human correction time.
Median and tail completion latency.
Total operating cost across all attempts.

A useful aggregate measure is:

Total operating cost during the evaluation ÷ number of acceptable completions.

Include the costs of unsuccessful requests in the numerator. State which costs are measured, estimated or excluded, and report the failure rate alongside the ratio.

API charges and human labour may also be shown separately to make the tradeoff visible. Account for integration and maintenance effort when deciding whether the change is worth adopting.

Pilot discipline

Use a narrow rollout with defined quality limits, monitoring and a rollback path. Reassess the choice when request patterns, prompts, model versions, prices or service behaviour change.

↑ Back to contents

8. Professional benefit and transfer

For AI engineers

Translate product needs into model-selection criteria.
Separate deterministic processing from language interpretation.
Generate architectural alternatives.
Evaluate routers and validators as fallible components.
Trace costs through retries and escalation.
Compare completed workflows rather than isolated calls.

For technical leaders

Set quality requirements before cost targets.
Ask where advanced capability creates useful value.
Consider engineering and reviewer workload.
Avoid unnecessary multi-model complexity.
Fund bounded comparisons with explicit acceptance criteria.
Keep model choices open to revision.

Limits of transfer

The approach can transfer to classification, extraction, summarisation, document preparation and other defined tasks. Success in one category does not establish suitability for a different domain or a more consequential action.

Reflection question

Which part of your AI workflow receives more model capability than the task requires—and what evidence would justify changing it?

Professional takeaway

Select the model as part of a complete workflow, and judge efficiency by acceptable work delivered.

↑ Back to contents

9. Reusable artifact: Model and Workflow Selection Record

Document the following.

Task and intended user outcome.
Fixed purpose used to generate alternatives.
Minimum quality requirements by case category.
Unacceptable errors and consequences.
Representative workload and challenge cases.
Development and held-out evaluation boundaries.
Candidate models, configurations and non-model alternatives.
Model identifiers, prompts, settings and application versions.
Deterministic steps and maintained data dependencies.
Routing conditions and known detection limitations.
Validation checks and what they cannot establish.
Unresolved, escalation and failure-handling paths.
Retries, human effort and cost-accounting boundaries.
Quality, latency and load-test results with uncertainty.
Pilot scope, acceptance limits and rollback conditions.
Decision owner and reassessment triggers.

Completion check: Show that the selected configuration meets the defined requirements, explain where it fails and demonstrate why its full workflow is preferable to the alternatives considered.

↑ Back to contents

References

Edward de Bono. Serious Creativity: Using the Power of Lateral Thinking to Create New Ideas. 1992. Alternatives and the fixed point, printed pages 126–127. Original source supplied with the project.
Anthropic. Choosing the Right Model . Documentation checked 6 November 2026.
Microsoft. Choose the Right AI Model for Your Workload . Azure Architecture Center. Documentation checked 6 November 2026.
Anthropic. Demystifying Evals for AI Agents . 9 January 2026.

Research note: Sources checked 6 November 2026. Model capabilities, service behaviour and documentation can change. The story and its comparison findings are fictional. No current pricing comparison or universally superior model configuration is asserted.

↑ Back to contents

Develop your engineering judgment

The Effective AI Engineer is a 16-part series combining Lateral Thinking with AI engineering. Each independent case includes a professional scenario, an explained thinking tool, research-grounded discussion and practical exercises.

Apply this case to one recurring task: define acceptable completion, generate alternatives and test whether a different configuration improves the complete workflow.

Join the course: For course and enrolment details, workshops or professional collaboration, contact Ravindra Dastikop and mention The Effective AI Engineer.

Email: ravindra.dastikop@gmail.com
Cell: 8453212131