The Effective AI Engineer · Case 16 of 16
The Engineer Who Kept the Decision Open
Applying Lateral Thinking to Create Meaningful Impact with AI
A successful pilot earns a decision about its next use, not automatic permission to expand.
An AI pilot can produce convincing demonstrations and useful results while leaving important questions unanswered. The professional task is to establish which uses the evidence supports, which need further work and which should stop.
Research-grounded fiction: The people, organisation and events are illustrative. Six Thinking Hats comes from Edward de Bono; the evaluation and release practices are contemporary engineering applications. The fictional pilot is not a report about the organisations cited.
1. The situation and learning outcomes
A company had completed a pilot of an AI assistant for its customer support team. The assistant retrieved guidance and prepared response drafts. Specialists reviewed each draft before sending it.
The launch presentation showed encouraging results:
- Drafts appeared quickly.
- Many specialists found them useful.
- Routine questions often received suitable responses.
- Usage had increased during the pilot.
The proposed next step was to expand the assistant across the support department.
Meera, the engineer responsible for evaluation, had helped build the system. She wanted it to succeed. She also knew that the pilot results did not describe one consistent experience.
What you will learn
- Interpret pilot results within their actual scope.
- Distinguish useful evidence from unsupported conclusions.
- Use Six Thinking Hats to structure a review.
- Generate alternatives to unrestricted expansion.
- Identify questions that require further testing.
- Document a decision with ownership and review conditions.
2. The story
A convincing demonstration
At the expansion review, the product manager opened with a demonstration. A routine request arrived. The assistant retrieved the relevant guidance and produced a clear draft.
“This is what we can give every specialist,” he said.
A senior support specialist raised a concern. “For standard requests, yes. For unusual ones, checking the draft sometimes takes longer than writing the response.”
The product manager pointed to the positive feedback. The specialist described a difficult case. Someone mentioned the approaching renewal date for the service contract.
The conversation became a contest between examples.
Meera stopped sharing the dashboard.
“We are discussing evidence, enthusiasm, concern and investment in the same breath,” she said. “Can we examine them separately before deciding?”
Blue Hat: define the decision
Meera proposed a short Six Thinking Hats sequence. Everyone would use the same thinking mode together. Nobody would become the permanent optimist or critic.
She wrote the decision:
Which request categories, users and operating conditions are supported by the pilot evidence?
The options included expansion, limited continuation, redesign, further investigation and stopping a particular use.
The purpose was to reach a decision the evidence could support. Agreement with the original launch proposal was not an acceptance requirement.
White Hat: establish what is known
The team inspected what its dashboard actually measured.
“Response time” meant time until a draft appeared. It did not include the specialist’s review, correction or follow-up work.
The positive feedback came from employees who answered an optional survey. Several participants had not responded.
The pilot had also included many routine requests and relatively few unusual ones.
The existing records were useful, but they did not establish department-wide improvement.
Meera separated three statements:
- Routine cases frequently produced usable drafts.
- Some specialists reported substantial correction effort.
- The total effect on completed work remained insufficiently measured.
The third statement was uncomfortable. It was also more accurate than declaring either success or failure.
Red Hat: acknowledge reactions
Under the Red Hat, participants briefly expressed their feelings without being required to justify them.
The product manager felt proud and impatient to move forward.
A specialist felt relieved when a useful draft appeared, but anxious when a polished response concealed an uncertain interpretation.
Meera felt protective of the work her team had invested.
These reactions mattered to the discussion. They did not become evidence of performance.
Yellow Hat: identify supported value
The team examined where the assistant could help. For routine requests, retrieved guidance and an editable draft could reduce repetitive preparation.
Newer specialists could see useful examples of how to structure a response. The assistant could also make relevant documentation easier to find.
Each benefit was recorded with a condition. A draft was useful when the guidance applied, the response was accurate and review effort remained reasonable.
The team could recognise those benefits while continuing to investigate unresolved categories.
Black Hat: examine reasons for caution
The team examined weaknesses and failure conditions. Unusual requests sometimes required policy interpretation that the assistant handled inconsistently.
Some drafts combined applicable guidance with a plausible but unsupported instruction.
Reviewers could miss such additions, particularly when the rest of the response looked familiar.
Expansion could also change the workload. More users and a different request mix could create conditions the pilot had not exercised.
The specialist who had raised concerns now helped identify categories where the system looked dependable. The product manager helped specify conditions under which expansion would be inappropriate.
Participants were changing thinking activities rather than defending fixed roles.
Green Hat: generate other next steps
The original choice had been framed as expand or do not expand. The team now generated alternatives:
- Expand only routine categories with adequate evidence.
- Offer source retrieval without drafting for selected complex cases.
- Use a structured preparation form where required information was predictable.
- Keep uncertain categories with the existing specialist workflow.
- Redesign the review interface to make unsupported additions easier to detect.
- Stop drafting for categories where correction burden exceeded its value.
Not every idea was selected. The team had nevertheless developed several ways to preserve useful work.
Return to the White Hat: close a critical evidence gap
Discussion could not resolve the missing measurement of total handling effort.
The team arranged a further comparison using representative tasks. Where practical, it assigned comparable work across assisted and existing workflows. It accounted for differences in task difficulty and reviewer experience.
It measured work through completion, including correction and rework. It also checked quality separately. Faster handling could not compensate for failing an essential requirement.
The fictional comparison supported a narrower conclusion than the original launch presentation.
Some routine categories justified limited expansion. Other categories required redesign or more evidence. A small category was removed from AI drafting because its review burden and errors did not support continued use in that form.
Blue Hat: make a bounded decision
At the follow-up review, Meera presented the decision by category.
Each approved use had:
- A defined scope.
- Evidence supporting continuation.
- A responsible owner.
- Monitoring requirements.
- Conditions for pausing or changing the use.
- A review date.
The team retained useful components from the discontinued category, including its test cases and organised source material. Stopping one implementation did not erase the learning.
The product manager looked at the revised plan. “We are expanding less than we expected.”
“We are expanding what we can currently support,” Meera replied.
“And the rest?”
“We have specified what would change the decision.”
3. The thinking method: Six Thinking Hats
Edward de Bono’s Six Thinking Hats separates different thinking activities so that they can receive deliberate attention. [1]
The hats represent temporary modes, not personality types. Participants can examine the same subject through different modes without becoming permanently identified with one position.
| Hat | Thinking activity | Application in the case |
|---|---|---|
| Blue | Organise and direct the thinking | Define the decision, sequence the discussion and record the outcome |
| White | Examine information and information needs | Inspect measures, missing evidence and limitations of the pilot |
| Red | Express feelings and intuitions | Acknowledge pride, anxiety and attachment without treating them as facts |
| Yellow | Explore value and reasons for benefit | Identify useful applications and the conditions supporting them |
| Black | Examine difficulties and reasons for caution | Identify failure conditions, review burden and limits of expansion |
| Green | Generate alternatives | Develop narrower uses, redesigns and different operating arrangements |
The sequence used in this case
Blue → White → Red → Yellow → Black → Green → White → Blue
This is one practical arrangement, not a mandatory order. The team returns to the White Hat when it discovers a question requiring additional evidence.
Method distinction: Six Thinking Hats structures the examination. It does not calculate the correct decision, guarantee consensus or replace experiments, specialist judgment and accountable ownership. No PO statement is required in this case.
4. Making the thinking visible
-
Define the decision precisely.
Replace “Was the pilot successful?” with a question about particular uses, users and conditions. -
Inspect what the measures mean.
Time to draft and time to completed work answer different questions. -
Separate observations from conclusions.
Positive feedback supports a claim about reported experience. It does not independently establish productivity or quality. -
Acknowledge investment and emotion.
Pride, fatigue and anxiety can be expressed without being disguised as evidence. -
Explore benefits and weaknesses deliberately.
Give each appropriate attention rather than allowing them to cancel each other through argument. -
Generate additional options.
Narrowing, redesigning or changing a function may preserve value. -
Obtain evidence the meeting cannot supply.
A structured conversation cannot fill missing performance data. -
Make the decision revisable.
Record its scope, owner, review conditions and the evidence that could change it.
Reusable prompt
We need to decide the next use of this AI system: [describe it].
Structure the review using Six Thinking Hats. Begin by defining the decision and available options.
Separate recorded facts, missing information, feelings, potential benefits, reasons for caution and alternative designs.
Do not assign permanent hats to participants or invent evidence. Identify which questions require testing outside the meeting.
Finish with a decision record stating supported scope, unresolved limitations, ownership, monitoring and conditions for reconsideration.
Do not force consensus or treat completion of the exercise as proof that deployment is justified.
5. Comparing the next-step options
| Option | When it may be appropriate | Evidence or preparation needed |
|---|---|---|
| Expand a defined use | Quality and operational requirements are met for the proposed scope | Representative results, capacity checks and monitoring |
| Continue a limited pilot | Potential value exists, but important uncertainty remains | A specific learning question and bounded test |
| Redesign the interaction | The current approach creates avoidable effort or errors | A demonstrated problem and comparative testing |
| Reduce the AI function | A narrower capability provides useful support | Tests of the remaining function and handovers |
| Pause a use | Requirements are not currently met or necessary evidence is unavailable | Clear user communication and an alternative process |
| Retire an implementation | Continued operation is not justified | A transition plan preserving necessary work and records |
Past expenditure should be recorded accurately. It does not establish the value of future expansion.
A use can also require several changes together. For example, limited expansion may depend on a redesigned review interface and a narrower set of supported categories.
6. Research evidence and limits
Evaluate outcomes as well as execution
Anthropic’s evaluation guidance distinguishes the recorded execution of a run from its final outcome. It also describes combining evaluations with production monitoring, experiments and user research. [2]
These distinctions support examining completed work rather than relying only on a convincing demonstration or the text of a response. They do not establish the performance of the fictional assistant.
Interpret findings within their studied setting
Research reported by Cui and colleagues used randomised field experiments with software developers. The combined analysis found increased completed tasks, while noting noisy individual experiments and differences associated with experience. [3]
These findings illustrate why study conditions, uncertainty and variation matter. They cannot be transferred directly to a customer support assistant or used to justify an unrelated deployment.
The proposition to test
Separating thinking activities and examining evidence at the level of the proposed use can support a more explicit, reviewable decision.
The case illustrates that proposition. It does not establish that Six Thinking Hats improves deployment outcomes, meeting speed or decision accuracy.
Evidence boundary: The pilot observations and comparison results in the story are fictional. The cited sources provide method and engineering context.
7. Practice and evaluation
Accessible exercise
Choose one AI tool or workflow you already use.
- State the decision you need to make about its future use.
- Record what you know and what remains unmeasured.
- Express your immediate reaction separately.
- Identify benefits and the conditions required for them.
- Examine difficulties and reasons for caution.
- Generate alternatives to continuing unchanged.
- Write what evidence would change your decision.
Technical exercise
Review one AI pilot.
- Define the unit of analysis: task, user group and operating condition.
- Inspect sampling, missing feedback and comparison quality.
- Measure review, correction and rework.
- Check quality against authoritative requirements.
- Include failures, exceptions and abandoned tasks.
- Assess the workload expected after expansion.
- Use a Six Thinking Hats session to organise the review.
- Assign tests for unresolved questions.
- Record a decision for each proposed use.
Evaluation measures
- Successful completion against requirements.
- Total handling effort.
- Correction and rework.
- Error severity and frequency.
- Results across relevant task categories.
- User experience and accessibility.
- Cost per successfully completed task.
- Operational capacity and maintenance effort.
Interpret the comparison carefully
Before-and-after differences can reflect changes in workload, staffing or experience. Use a comparison design appropriate to the claim being made and report uncertainty.
Set acceptance requirements before using results to justify expansion. Document any later changes and their reasons.
Do not use an average result to conceal a category that fails an essential requirement. At the same time, avoid drawing confident conclusions from very small subgroups.
8. Professional benefit and transfer
For AI engineers
- Interpret results within their actual scope.
- Distinguish model performance from workflow value.
- Make missing evidence visible.
- Preserve useful components while redesigning weak implementations.
- Define conditions for reconsideration.
For technical leaders
- Avoid turning launch reviews into contests between advocates and critics.
- Allocate attention to evidence, value, concerns and alternatives.
- Accept different decisions for different uses.
- Assign responsibility beyond the demonstration.
- Treat stopping an unsuitable implementation as a legitimate outcome.
Limits of transfer
A structured discussion cannot guarantee candour, eliminate power differences or repair a poorly designed experiment. Participants need access to relevant evidence and a genuine opportunity to question conclusions.
Some requirements are mandatory. A benefit identified under the Yellow Hat does not cancel a failure to meet them.
Reflection question
What evidence would make you narrow or stop an AI system you helped build?
Professional takeaway
Make the next decision specific, evidence-based and open to revision.
9. Reusable artifact: AI Continuation Decision Record
Complete one record for a defined use. Attach evidence or link to authorised records where appropriate.
- System and implementation version.
- User group, task and operating conditions.
- Decision being considered.
- Acceptance requirements.
- Evidence sources and comparison method.
- Missing information and sampling limitations.
- Total handling effort and quality results.
- Benefits and their supporting conditions.
- Difficulties and failure conditions.
- Feelings or intuitions recorded separately.
- Alternative implementations considered.
- Additional tests and responsible owners.
- Decision and permitted scope.
- Transition or handover arrangements.
- Monitoring and pause conditions.
- Review date and evidence that would change the decision.
Completion check: Explain what is supported, what remains uncertain, who owns the next step and when the decision will be reconsidered.
10. Final course application
Select one real AI workflow and prepare a reviewable recommendation for its next use.
-
Define the purpose.
State what the person or organisation needs to accomplish. -
Examine an established choice.
Identify its purpose and whether other implementations could serve it. -
Generate alternatives.
Include changes to the workflow, information, interaction and AI role. -
Test the complete task.
Evaluate outcomes, correction effort, exceptions and operational limits. -
Investigate failures.
Separate observations from explanations and seek distinguishing evidence. -
Prepare for interruption.
Establish what continues, waits or stops, and how work recovers. -
Review the next decision.
Use Six Thinking Hats to examine evidence, reactions, value, difficulties and alternatives. -
Document the recommendation.
State scope, ownership, limitations, review conditions and the evidence that could change it.
Final deliverable: An AI Continuation Decision Record supported by task evidence, alternative designs and an explicit account of uncertainty.
The aim is to make a justified next decision. Expansion, redesign, limited continuation and retirement can each be appropriate outcomes.
References
- Edward de Bono. Six Thinking Hats. Little, Brown and Company, 1985. Project-supplied source, including the colour functions and discussions of the individual thinking modes.
- Anthropic. Demystifying evals for AI agents . 9 January 2026.
- Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers . Microsoft Research publication page, June 2025.
Research note: Sources were consulted during preparation of this case on 6 November 2026. Online material can change. The sources do not validate the fictional pilot or establish a measured benefit from the thinking exercise.
