THE EFFECTIVE AI ENGINEER
CASE #003 · CASE 3 OF 16
The Engineer Who Tested Before Building
Applying Lateral Thinking to Create Meaningful Impact with AI
By Ravindra Dastikop
Engineering focus: Evaluation-driven development, acceptance criteria and architecture discovery.
1. The situation and impact sought
Meera had three weeks to demonstrate an assistant that arranged internal training sessions. It would read a request, check a test calendar and prepare a booking for approval.
The team had not agreed what counted as a successful booking. A polished confirmation sounded persuasive, but participants needed the right time, duration and room. An ambiguous request might require clarification rather than a reservation.
Meera could use approved examples, a sandbox calendar and two training coordinators. The prototype could not send invitations or modify the production calendar.
The intended impact was less coordinator effort without more wrong bookings, duplicates or missed clarifications. The current manual process was the baseline. Time to an accepted plan, correction effort and actual calendar state would matter more than fluent confirmation text.
2. The story
At the first meeting, Meera sketched a conversational interface.
Someone suggested adding calendar tools. Someone else proposed separate agents for scheduling, room selection and confirmation.
“We can evaluate it once there’s something to evaluate,” her manager said.
Meera wrote that sentence in her notebook.
That afternoon, she tried an odd reversal: imagine the tests choosing the system that would build it.
She pictured a test sitting in the project meeting.
“What can you establish?” it asked the proposed components.
The calendar lookup could establish availability. A language model could suggest what a request meant. Neither could decide which of two conflicting business instructions the team intended to follow.
The imaginary conversation opened several directions. A test might ask users for a missing rule. It might invite different implementations to demonstrate a capability. It might identify the smallest useful product instead of grading an entire assistant.
Meera brought a different artifact to the next meeting: six request cards.
One asked for an available room at a specified time. Another asked for a full room. A third said “next Friday afternoon.” A fourth repeated an earlier request after a connection failure.
“What would a correct system do here?” she asked.
The coordinators agreed on the first card.
They disagreed on the second. One would offer another time. The other would preserve the requested time and find a different room.
“Both might be useful,” Meera said. “Which matters to the requester?”
They changed the task: when the preference was unknown, the assistant should ask. The evaluation would accept different clear questions that resolved the uncertainty.
Before connecting a model, Meera wrote a human reference response for each card and checked whether it passed her scoring rules.
One did not.
Her grader looked for a particular sentence. A coordinator had expressed the same valid question differently.
Meera fixed the grader before she blamed any future assistant.
She asked an approved AI assistant to expand the scenarios, keeping a list of the capabilities each one required. It proposed interrupted lookups, conflicting time zones and unavailable equipment.
It also invented a rule about minimum notice.
The coordinators removed it. Their policy said nothing of the kind.
AI had helped broaden the questions. The team still had to supply the answers.
Meera separated visible development examples from reserved evaluation cases. She used invented names and sandbox records, and kept reference answers outside the candidate system’s inputs.
Then she compared three small approaches: a form with validation, a model that interpreted language inside a fixed workflow, and an agent that could investigate alternatives through tools.
The form handled explicit requests. The workflow handled some free-text requests without delegating every scheduling decision to a model. The agent could investigate less predictable requests, but introduced more behavior to inspect.
The scores still contained a surprise.
A candidate wrote, “Your session is booked,” in a warm, confident message. The language grader liked it.
The calendar contained no reservation.
Meera changed the evaluation to inspect the resulting state. In this task, a confirmation was correct only if the appropriate sandbox reservation actually existed. If approval was required, a pending proposal had to remain distinct from a completed booking.
On another trial, an interrupted tool call was retried and created two reservations.
The team added a request identifier and duplicate checks at the execution boundary. A prompt asking the model to avoid duplicates was insufficient.
Those discoveries made the first product smaller.
Meera proposed a supervised scheduling draft for a defined set of requests. It would interpret the request, collect missing details, show verified availability and prepare a proposal. A coordinator would approve the action. The implementation would recheck availability before writing.
The team tested that narrow design against the manual process and other prototypes. The fictional trials suggested it deserved a bounded pilot. They did not establish a reliable time saving or justify unattended scheduling.
At the demonstration, Meera showed a request the assistant could handle and one it correctly left unresolved.
Her manager asked whether the tests had delayed the build.
“They changed the build,” she said.
The first useful output of the evaluation had been a clearer product decision.
3. The thinking tool explained
The tool is reversal provocation, followed by deliberate movement.
In Serious Creativity, de Bono describes reversal as changing the normal direction of an action. Movement develops an idea from the provocation, including by extracting a principle or examining the difference. See printed pages 316–319 in the supplied edition.
Normal relationship: The engineer builds a system and uses tests to assess it.
Original modern provocation:
PO: The tests choose and commission their engineer.
Tests cannot literally hire or direct a team. The useful movement is to explore what changes when evidence requirements take the initiative: they can expose unresolved decisions, invite competing implementations and bound the first product.
Three directions emerge from that movement:
- Tests elicit missing decisions from stakeholders.
- Tests act as a common audition for different architectures.
- Tests help define the smallest capability worth building.
The story combines those directions. Reversal supplies a thinking stimulus; engineering supplies the practical methods.
This PO is our original adaptation, not a quotation from de Bono. Evaluation-driven development is established practice. The case illustrates one way to discover its relevance; it does not claim to invent it or prove that provocation improves delivery.
4. The thinking made visible—including AI’s contribution
Focus: Reduce the effort required to arrange a correct training session.
Assumption: The implementation must exist before evaluation can do useful work.
Provocation: Tests choose and commission their engineer.
Movement by extracting a principle: A test describes evidence of a needed capability, so identifying that evidence can guide implementation choices.
Movement by examining the difference: The product no longer defines its own success after it is built. Stakeholders must decide what should count beforehand.
The alternatives are architectural and organizational. Meera can clarify policy before coding, compare small implementations, or narrow the product to a supervised proposal rather than autonomous booking.
AI helps propose scenario variations and alternative implementations. Coordinators validate requirements. Meera controls the evaluation, examines grading failures and interprets the results.
Reusable prompt:
Explore this provocation: PO, the tests choose and commission their engineer.
Task and useful outcome: [description].
Known policies and available evidence: [verified facts].
Unresolved decisions: [list].
Constraints and authority: [limits].
Extract a useful principle and develop three different implementation directions.
For each, propose a small capability test, acceptable outcomes, cases requiring clarification and an approach without an autonomous agent.
Keep invented scenarios separate from verified policy.
Flag information needed to judge success.
Do not invent results or write a production-readiness verdict.
A model can generate candidate tests. That does not make it an authoritative source of requirements or an independent validator of its own answers.
5. The engineering decision
| Alternative | Potential benefit | Cost or difficulty | Critical uncertainty | Decision |
|---|---|---|---|---|
| Form and validation | Explicit inputs and easy inspection | More input effort | Will users accept structured intake? | Keep as baseline |
| Model interpretation in a workflow | Natural-language entry with controlled execution | Ambiguity and semantic errors | Is the interpretation reviewable? | Test narrowly |
| Tool-using agent | Investigates flexible alternatives | More variable behavior and evaluation work | Is dynamic exploration needed? | Compare on suitable cases |
| Supervised scheduling proposal | Human approval before action | Review queue and handoff effort | Does it reduce total coordinator work? | Bounded pilot candidate |
The pilot candidate uses model interpretation within a controlled workflow. Approval is an operating arrangement that could also apply to other architectures; the alternatives are not all mutually exclusive.
Tests must permit valid alternative solutions. Exact wording or one preferred tool sequence should not be required unless it is part of the actual specification.
Availability can change after a proposal is generated. The executor must recheck before writing and return a clear failure when the action can no longer be completed.
Any time saving must include review, clarification and maintenance. Passing the test set establishes performance on those cases, not general production reliability.
6. The evidence behind the case
Anthropic recommends defining capability evaluations before agents can fulfil them. It also discusses reference solutions, balanced tasks, isolated trials and calibrated graders [1].
OpenAI recommends early, task-specific evaluation, representative inputs, continuous measurement and human calibration [2].
Anthropic has documented a benchmark-contamination incident involving accessible answer material [3]. Its specific findings are not evidence that every private test will leak. They reinforce the need to examine what a candidate can access and how a score was obtained.
The scheduling scenario, commissioning PO and resulting design are our adaptations. No source establishes that this fictional prototype saves time, or that the organizations use de Bono’s methods.
The case also draws on familiar software testing and execution practices. Their value must be assessed in context; beginning with tests does not eliminate the need for exploratory prototypes or user observation.
7. Try the move
Choose a proposed AI feature. Write five representative task cards before selecting a framework.
For each, identify the inputs available at the time, required evidence, acceptable outcomes, clarification conditions and unacceptable actions. Use verified policies; mark unknowns.
Create one valid reference solution and one plausible incorrect response. Check whether your scoring process distinguishes them.
Apply the reversal and generate three ways the evaluation could shape the project. It may reveal an unresolved requirement, change the architecture or reduce the initial scope.
For an accessible exercise, run the cards through human role-play and paper prototypes. Record disagreements rather than hiding them in a single score.
For a technical extension, build a minimal harness: load a test fixture, run a candidate, capture tool activity, inspect final state and apply relevant checks. Reset state between trials.
Use development cases for iteration and reserve separate cases for assessment. Once a reserved case is repeatedly used to tune the system, stop describing it as untouched validation.
Compare the current process, a simple baseline and one AI candidate. Repeat variable trials. Report sample size, task categories, failures, review effort, time and cost.
Define pilot conditions and stopping criteria before final assessment. Do not choose an arbitrary pass percentage as a universal deployment threshold; consequences and workflow requirements should inform the decision.
8. Take it into your work—including limits of transfer
For the engineer: before building a large system, identify one capability whose success you can judge and one requirement you cannot yet judge.
For the technical leader: ask which evidence would change the investment decision. Resource domain review and grader inspection, not just implementation.
This approach fits projects with costly ambiguity about success. A small evaluation can also help compare prototypes already under development.
It fits less cleanly when the useful outcome is still being discovered. Begin with observation or a lightweight exploratory prototype, then revise criteria as understanding improves. Prematurely freezing tests can preserve the wrong objective.
Evaluation has a cost. Use a small informative set first and expand it as the decision warrants. Automated evaluation should be complemented by appropriate live monitoring and user feedback.
No measured return on investment is claimed. The practical aim is to avoid committing to a design before understanding the evidence it must produce.
Transfer question:
Which acceptance criterion would you struggle to explain to a domain colleague before your next build begins?
9. Reusable artifact and sources
Capability Before Construction Card
- User and useful outcome:
- Task and available inputs:
- Verified policy or reference:
- Unresolved requirement:
- Acceptable outputs and actions:
- Conditions requiring clarification:
- Required final-state evidence:
- Valid reference solution:
- Plausible failure:
- Grader and known limitations:
- Three possible implementation directions:
- Development and reserved cases:
- Baseline and comparison conditions:
- Review effort, time, cost and failure consequences:
- Pilot criteria, owner and review date:
References
Checked 4 November 2026.
- Anthropic. Demystifying evals for AI agents. 9 January 2026.
- OpenAI. Evaluation best practices.
- Anthropic. Eval awareness in Claude Opus 4.6’s BrowseComp performance. 6 March 2026.
- Edward de Bono. Serious Creativity, 1992, printed pages 316–319, supplied project edition.
