The Effective AI Engineer #012: The Engineer Who Chose the Smaller Model

The Effective AI Engineer • Case #012 • 12 of 16

The Engineer Who Chose the Smaller Model

Generating alternatives and selecting model capability against the requirements of completed work.

By Ravindra Dastikop

Applying Lateral Thinking to Create Meaningful Impact with AI

Choose model capability against the work’s requirements, and measure efficiency at the point where the work is successfully completed.

Lateral Thinking tool: Alternatives, organised around a fixed purpose.

Engineering focus: Model selection, task decomposition, evaluation, latency and total workflow cost.

Professional benefit: Learn to compare models and workflow designs against explicit quality requirements, including the cost of retries, escalation and human correction.

About this case: This is research-grounded fiction. Characters, events and comparison findings are illustrative. Technical sources were checked on 6 November 2026. No model ranking, measured saving or universal performance advantage is claimed.

In this case

1. The situation
2. The story
3. The Lateral Thinking tool
4. Making the thinking visible
5. Comparing the alternatives
6. Research evidence and limits
7. Practice and evaluation
8. Professional benefit and transfer
9. Reusable artifact
10. References

1. The situation

A software company had built an internal assistant to prepare incoming support requests for investigation.

For each request, the assistant identified the product and version, classified the issue, extracted relevant details and produced a short handoff note for a support engineer.

The team used its strongest available general-purpose model for every request.

It handled difficult messages reasonably well. It also spent time interpreting straightforward submissions whose product, version and error code were already supplied in form fields.

As request volume grew, the team faced pressure to reduce operating cost and waiting time.

The initial proposal was to replace the model with a less expensive alternative.

That made the support lead uneasy.

“We chose the stronger model because these requests matter,” she said.

The team appeared to face a familiar choice:

Keep the stronger model and accept the cost, or use a smaller model and accept worse quality.

↑ Back to contents

2. The story

Nikhil, the engineer responsible for the assistant, opened a routine request.

The form contained a product identifier, a version number and an error code. The assistant had read the message, inferred the same identifiers and written a paragraph explaining its classification.

“What decision did we need the model to make here?” he asked.

“Prepare the request for the right support queue,” said Leela, the support lead.

“Does that require this model?”

“Sometimes.”

“Which part?”

They examined several requests together.

One was a completed form reporting a known error. Another contained a long conversation with conflicting version information. A third described an unfamiliar failure across two products.

The application treated all three as the same task.

A purpose that opened alternatives

Nikhil wrote a purpose on the board:

Produce a correct, evidence-linked support handoff, or explicitly identify what remains unresolved.

Then he asked the team to generate alternatives that could fulfil that purpose.

• Keep the current model and shorten unnecessary output.
• Use a lower effort setting where supported.
• Use deterministic code for validated form fields and known mappings.
• Test a smaller model for bounded language interpretation.
• Send every request directly to a support engineer.
• Route different request types through different combinations.

They deliberately postponed choosing.

“The smaller model is one alternative,” Nikhil said. “It is not the whole design.”

The direct replacement exposed a quality gap

They first established a baseline. Support engineers reviewed representative requests and defined what a usable handoff needed: correct product and version, relevant evidence, an appropriate queue and visible unresolved information.

A missing version was not automatically a failure. Inventing a version was.

Then they tested a smaller model as a direct replacement.

It performed acceptably on many routine requests. But several messages containing conflicting details produced clean-looking handoffs that concealed the conflict.

Leela pointed to one.

“The fields are complete. The handoff is wrong.”

Their schema validator had accepted it. All required fields existed and had the right types.

The team added checks for factual support and unresolved contradictions. Structural validity alone had not established correctness.

Short did not mean simple

Next, they introduced a route for apparently simple requests.

A request went to the smaller model when it was short and mentioned a known product. Longer requests went to the stronger model.

The first serious routing error arrived in one sentence:

“Before today’s upgrade it worked; after rolling back it still fails, although the console shows the new version.”

Short did not mean simple.

They discarded message length as the main decision rule.

The revised design used verified form completeness, supported product categories and tested request patterns to define a narrow routine path. Conflicting information and unsupported categories followed a different path. Uncertain cases remained eligible for human investigation.

Even then, the team did not assume that the router could recognise every difficult case. They included hard examples that looked routine in the evaluation set.

Separate the work before selecting the model

They also changed the division of work.

Application code copied validated identifiers and applied maintained mappings. The smaller model interpreted the remaining language and drafted a concise note from the supplied evidence.

It was instructed to preserve missing information rather than complete every field.

The stronger model remained available for selected requests requiring broader interpretation. It could also return an unresolved result. Its presence did not make insufficient evidence sufficient.

The cost report measured the attempt

The next comparison exposed another problem.

The smaller model made an incorrect classification. A retry corrected it. The system had recorded only the cheaper first call in its summary cost report.

Nikhil followed the request through the logs.

There had been an initial call, a retry, an escalation and a support engineer’s correction.

“We have priced the attempt,” he said. “We have not priced the completed work.”

They linked every attempt and review to the original request. Their comparison now included routing, retries, escalation, human handling and completion time.

The small-first cascade was unattractive for one category: requests that frequently needed the stronger model anyway. Those requests waited through two model calls.

The team routed that category directly to the stronger model.

Another category gained little from generation at all. A validated form and maintained lookup produced the required handoff.

For the bounded language-interpretation category, the smaller model remained promising.

A limited selection

During load testing, its usual response time was good, but the slowest requests still exceeded the team’s target. They investigated service capacity, queueing and fallback behaviour instead of treating the model label as a latency guarantee.

Finally, a new product version appeared in the incoming requests.

It was outside the evaluated coverage. The workflow marked it unresolved and sent it for manual handling rather than applying an old mapping.

The team approved a limited pilot.

Its design used code for established facts, a smaller model for a tested language task, a stronger model for selected complex requests and people for unresolved cases.

Leela reviewed the decision.

“So you chose the smaller model.”

“For a defined part of the work,” Nikhil replied. “We chose the workflow against the requirement.”

↑ Back to contents

3. The Lateral Thinking tool: Alternatives

Edward de Bono explains that alternatives need a reference point—a fixed point. One useful fixed point is the purpose being achieved. Holding that purpose steady allows different ways of achieving it to be explored. [1]

In this case, the fixed point was:

Produce a correct, evidence-linked support handoff, or explicitly identify what remains unresolved.

That purpose allowed the team to consider more than replacement models.

Its alternatives included code, maintained mappings, model settings, task decomposition, different models and manual handling.

The thinking sequence was:

1. Identify the purpose behind the current implementation.
2. Make that purpose explicit.
3. Generate different ways to achieve it.
4. Develop the promising options.
5. Evaluate them against the same requirements.

No PO statement is needed here. Deliberate alternative generation provides the change in direction.

The original method comes from de Bono’s Serious Creativity, particularly the fixed-point discussion on printed pages 126–127. The engineering architecture is a contemporary application, not a method attributed to the cited technology companies.

↑ Back to contents

4. Making the thinking visible

The team’s progress depended on changing five questions.

What must this work accomplish?

The handoff requirement became the basis for comparison. General model capability was relevant, but it was not the acceptance criterion.

Which parts require language interpretation?

Validated identifiers and maintained mappings could be handled by application code. Ambiguous descriptions required a different kind of work.

Are the output’s claims supported?

A well-formed result could still contain an incorrect version or conceal conflicting evidence. Schema compliance was only one check.

What observable conditions justify this route?

The team treated routing as a fallible component requiring its own evaluation.

What does successful completion require?

Retries, escalation and human correction could change the apparent economics of a cheaper first call.

Reusable prompt

We are designing an AI workflow for [task].

Define a fixed purpose in terms of the completed work, including what should happen when evidence is insufficient.

Generate at least five ways to achieve that purpose. Include a deterministic approach, the current model, a different model or effort setting, a decomposed workflow and a manual alternative.

For each option, identify the capability required, evidence available, likely failure modes, validation limits and operational overhead.

Propose a comparison using representative cases and difficult cases that look routine. Include quality, latency, retries, escalation, human effort and total cost per acceptable completion.

Do not assume that smaller means cheaper overall, that a larger model is always correct, or that self-reported confidence establishes correctness.

↑ Back to contents

5. Comparing the alternatives

Model and workflow options

Alternative	Potential benefit	Limitation to test
Current stronger model for every request	Simple architecture and broad capability.	May spend unnecessary effort on bounded tasks.
Current model with adjusted effort or shorter output	May improve efficiency without another model integration.	Capability and quality can change with settings.
Smaller model as a direct replacement	Potentially lower inference cost or latency.	May fail on important request categories.
Code plus a smaller model for a bounded task	Separates established facts from language interpretation.	Requires maintained mappings and clear task boundaries.
Routed combination of models	Can match different requests to different capabilities.	Adds routing errors, monitoring and operational complexity.
Manual preparation	Flexible handling of exceptional cases.	Requires human capacity and introduces waiting.

The fictional team selected a combination. Another workload might justify keeping one model.

The title describes a decision within the case. It is not a recommendation to favour smaller models in general.

A note on terminology: Here, “smaller” describes the fictional candidate. For commercial models whose parameter counts are undisclosed, engineers should describe observable properties—capability, price, latency and supported features—rather than infer physical model size.

↑ Back to contents

6. Research evidence and limits

Model selection is workload-specific

Anthropic’s model-selection guidance recommends considering capability, speed, cost and effort settings, then testing actual prompts and data. It describes both efficiency-first and capability-first starting approaches. [2]

This supports comparing candidates against the task rather than assuming one starting strategy always wins.

Routing introduces its own engineering tradeoffs

Microsoft’s Azure Architecture Center discusses task fit, single-model and multi-model designs, and manual versus automatic selection. It also identifies predictability, observability and architectural complexity as routing considerations. [3]

Routing therefore needs evidence of value beyond a lower advertised model price.

Evaluate the system that completes the task

Anthropic’s evaluation guidance distinguishes attempts, traces and outcomes. It recommends multiple trials where outputs vary and explains that an agent evaluation assesses the model working with its surrounding harness. It also identifies latency, usage, cost and errors as useful baseline measures. [4]

The proposition to test

A bounded task may be served adequately by a less demanding model configuration, provided the complete workflow meets its quality and operating requirements.

These sources support the evaluation approach. They do not establish the fictional team’s results or prove that a smaller model will improve another application.

↑ Back to contents

7. Practice and evaluation

Accessible exercise

Choose one recurring AI task.

• Define an acceptable completed result.
• Identify unacceptable errors separately from minor defects.
• Mark which inputs are already established facts.
• Identify which steps genuinely require language interpretation.
• Generate five alternatives before selecting one.
• Include an option that uses no generative model.
• Decide what happens when the available information is insufficient.

Technical exercise

Build a controlled comparison of a small set of plausible configurations.

• Record the model identifier, settings, prompt and application version.
• Use representative workload cases.
• Maintain a separate challenge set.
• Include difficult requests that appear routine.
• Keep development examples separate from held-out evaluation.
• Use comparable evidence and tool access.
• Give candidates a documented development budget.
• Distinguish a direct model comparison from a redesigned workflow comparison.
• Validate factual support as well as output structure.
• Test missing information, contradictions and unsupported categories.
• Record every attempt, escalation and human correction.
• Test realistic load and service failures.

Measure quality before declaring savings

Define minimum requirements for important case categories. Do not allow a strong average result to conceal an unacceptable error in a consequential category.

• Acceptable completion rate.
• Unsupported factual claims.
• Incorrect routing.
• Appropriate unresolved outcomes.
• Unnecessary escalation.
• Human correction time.
• Median and tail completion latency.
• Total operating cost across all attempts.

A useful aggregate measure is:

Total operating cost during the evaluation ÷ number of acceptable completions.

Include the costs of unsuccessful requests in the numerator. State which costs are measured, estimated or excluded, and report the failure rate alongside the ratio.

API charges and human labour may also be shown separately to make the tradeoff visible. Account for integration and maintenance effort when deciding whether the change is worth adopting.

Pilot discipline

Use a narrow rollout with defined quality limits, monitoring and a rollback path. Reassess the choice when request patterns, prompts, model versions, prices or service behaviour change.

↑ Back to contents

8. Professional benefit and transfer

For AI engineers

• Translate product needs into model-selection criteria.
• Separate deterministic processing from language interpretation.
• Generate architectural alternatives.
• Evaluate routers and validators as fallible components.
• Trace costs through retries and escalation.
• Compare completed workflows rather than isolated calls.

For technical leaders

• Set quality requirements before cost targets.
• Ask where advanced capability creates useful value.
• Consider engineering and reviewer workload.
• Avoid unnecessary multi-model complexity.
• Fund bounded comparisons with explicit acceptance criteria.
• Keep model choices open to revision.

Limits of transfer

The approach can transfer to classification, extraction, summarisation, document preparation and other defined tasks. Success in one category does not establish suitability for a different domain or a more consequential action.

Reflection question

Which part of your AI workflow receives more model capability than the task requires—and what evidence would justify changing it?

Professional takeaway

Select the model as part of a complete workflow, and judge efficiency by acceptable work delivered.

↑ Back to contents

9. Reusable artifact: Model and Workflow Selection Record

Document the following.

1. Task and intended user outcome.
2. Fixed purpose used to generate alternatives.
3. Minimum quality requirements by case category.
4. Unacceptable errors and consequences.
5. Representative workload and challenge cases.
6. Development and held-out evaluation boundaries.
7. Candidate models, configurations and non-model alternatives.
8. Model identifiers, prompts, settings and application versions.
9. Deterministic steps and maintained data dependencies.
10. Routing conditions and known detection limitations.
11. Validation checks and what they cannot establish.
12. Unresolved, escalation and failure-handling paths.
13. Retries, human effort and cost-accounting boundaries.
14. Quality, latency and load-test results with uncertainty.
15. Pilot scope, acceptance limits and rollback conditions.
16. Decision owner and reassessment triggers.

Completion check: Show that the selected configuration meets the defined requirements, explain where it fails and demonstrate why its full workflow is preferable to the alternatives considered.

↑ Back to contents

References

1. Edward de Bono. Serious Creativity: Using the Power of Lateral Thinking to Create New Ideas. 1992. Alternatives and the fixed point, printed pages 126–127. Original source supplied with the project.
2. Anthropic. Choosing the Right Model . Documentation checked 6 November 2026.
3. Microsoft. Choose the Right AI Model for Your Workload . Azure Architecture Center. Documentation checked 6 November 2026.
4. Anthropic. Demystifying Evals for AI Agents . 9 January 2026.

Research note: Sources checked 6 November 2026. Model capabilities, service behaviour and documentation can change. The story and its comparison findings are fictional. No current pricing comparison or universally superior model configuration is asserted.

↑ Back to contents

Develop your engineering judgment

The Effective AI Engineer is a 16-part series combining Lateral Thinking with AI engineering. Each independent case includes a professional scenario, an explained thinking tool, research-grounded discussion and practical exercises.

Apply this case to one recurring task: define acceptable completion, generate alternatives and test whether a different configuration improves the complete workflow.

Join the course: For course and enrolment details, workshops or professional collaboration, contact Ravindra Dastikop and mention The Effective AI Engineer.

Email: ravindra.dastikop@gmail.com
Cell: 8453212131