The Effective AI Engineer #018: The AI Builder Who Watched the User Work

THE EFFECTIVE AI ENGINEER · A CASE-BASED APPROACH

Case #018: The Builder Who Watched the User Work

Choose Work That Matters

By Ravindra Dastikop

AIYING — Accomplishing with AI

A convincing answer can still leave the user’s task unfinished.

About this case

This original fictional teaching case draws on published material from OpenAI, Anthropic, Microsoft Research and the UK Government Digital Service. CedarWorks, its people, workflow, timings, budget and example outcomes are author-created teaching material. No fictional result is attributed to a real company.

Evidence checked: 7 November 2026. No actual user study, production integration or business-benefit measurement was conducted for this case.

Contents

1. Learning objectives
2. The case: an answer and an unfinished task
3. The decision
4. Exhibit 1: the surrounding workflow
5. Exhibit 2: four delivery options
6. Exhibit 3: nine task scenarios
7. What published practice contributes
8. Your recommendation
9. Worked analysis and practical guide
10. Evaluation and pilot design
11. Exercises and reusable prompts
12. Teaching and assessment
13. References and evidence limits
14. Author and contact

1. Learning objectives

After working through this case, you should be able to:

• Define a useful task outcome beyond the quality of an AI answer.
• Separate observed behaviour, interpretation, hypotheses and design decisions.
• Map verification, correction, exceptions and handoffs around an AI interaction.
• Compare an AI workflow with a credible non-AI alternative.
• Specify a bounded prototype and an evaluation plan before claiming success.

For: students, freshers, freelancers, AI builders, developers, engineers, product owners, functional teams and faculty. Coding: optional.

↑ Back to contents

2. The case: an answer and an unfinished task

The demonstration

Nikhil Desai, an independent AI builder, arrived at CedarWorks Services with a polished equipment-request assistant. The business handled requests from employees joining projects or replacing unsuitable devices. Sana, the operations sponsor, wanted fewer repetitive questions for the coordination team.

Nikhil’s assistant searched an approved policy collection. It explained which equipment categories suited different roles, identified required information and described when an exception was needed. Its answers were readable and included references.

During the demonstration, Sana asked about a new analyst who needed a portable computer. The assistant suggested a standard category and supplied a plausible justification. Sana liked the result.

The prototype was deliberately read-only. It could not verify live stock, create a request record or submit anything. Nikhil had no integration agreement and had kept those actions outside his scope.

Then Sana asked: “How many requests will this remove from the coordinator’s queue?”

Nikhil had tested answers to prepared questions. He had not watched a request move from an employee’s need to a packet ready for authorised review. His demonstration could not answer Sana’s question.

What happened after the answer

Before leaving, Nikhil asked to watch Priya, a coordinator, prepare a request using synthetic details. She read the assistant’s answer and said it was useful. Then she opened another window.

She needed a cost centre. She checked the applicable policy version, found delivery details, searched for an existing request and verified whether the item was standard. She checked the age of the stock export. Finally, she copied information into the request form and attached the supporting reference.

The assistant had answered a question. Priya still had to assemble an artifact that another person could review.

A missing cost centre could not be guessed from a familiar team name. Old inventory information could not justify a promise of availability. A nonstandard request required an authorised decision. An existing request should be linked rather than duplicated.

A useful observation, with limits

Nikhil began recording what happened after the answer. He saw possible opportunities to reduce copying and make missing information visible. He also saw necessary checks that should not simply disappear.

However, one replay could not establish a representative workflow. Perhaps Priya handled unusual exceptions. Perhaps requesters could provide information directly. Perhaps clearer labels and validation in the existing form would solve most of the problem.

Some users might only want a policy explanation. Replacing every conversation with a long intake form could make their experience worse.

The delivery constraint

Nikhil had a fictional capacity limit of 30 person-hours: 24 builder hours, four process-owner or coordinator hours and two technical-review hours. Sana expected a revised demonstration the following week.

He could improve the chat, create guided request preparation, investigate autonomous submission, or improve the existing form. He could not silently increase the budget or assume unlimited access to staff.

What should Nikhil deliver next—and what evidence would make that decision defensible?

↑ Back to contents

3. The decision

Recommend the next increment with the information available. No later business outcome is supplied.

• Who is the primary user, and what task are they trying to complete?
• Does success mean understanding policy, preparing a request or submitting one?
• Which steps require clarification, verification or an authorised human decision?
• Which work could a better form remove?
• What evidence would justify changing the recommendation?

The title describes the builder’s action. It does not prove that observation led to a superior product.

↑ Back to contents

4. Exhibit 1: the surrounding workflow

The following figures are invented for an arithmetic and reasoning exercise. They are not measurements or company results.

Synthetic active effort, in minutes

Step	Manual	Chat-assisted
Clarify the need	3	5
Find policy evidence	4	2
Check eligibility and stock	3	4
Enter the request	4	4
Review the packet	2	1
Route for review	2	1
Total active effort	18	17

Waiting time is excluded. A faster answer does not establish an equivalent reduction in total task effort.

Separate observation from interpretation

Observed in the fictional replay	Hypothesis to investigate
Priya opened another record.	Required context may be missing from the interaction.
She checked for an existing request.	Duplicate handling may be part of the task.
She checked policy and stock dates.	The output may need visible evidence freshness.
She copied text into a form.	A reviewable packet or better form may reduce transfer effort.

Ask what each action accomplishes. Do not infer frequency, motivation or organisation-wide practice from a single replay.

↑ Back to contents

5. Exhibit 2: four delivery options

Option	Proposed work	Indicative builder hours	Main uncertainty
A. Improve chat	Better retrieval, wording and references.	12–18	How much downstream work remains?
B. Prepare a packet	Guided fields, source links, clarification and reviewable draft.	18–24	Can corrections and exceptions fit the scope?
C. Submit autonomously	Request creation and submission tools.	More than 24; not reliably estimated.	Identity, authority, retries and recovery are undefined.
D. Improve the existing form	Required fields, clearer help and routing guidance.	8–14	How much value actually requires a language model?

These are fictional planning ranges. The additional six hours of process-owner and reviewer capacity are separate from the builder’s allocation.

Authority boundary: the current prototype can prepare material for review. It cannot approve a request, reserve inventory, purchase equipment or submit a live transaction.

↑ Back to contents

6. Exhibit 3: nine task scenarios

Before reading the worked analysis, decide how a request-preparation prototype should respond to each scenario.

1. R001: Complete standard request with current evidence.
2. R002: Cost centre is missing.
3. R003: Stock evidence is stale.
4. R004: The requested item is nonstandard.
5. R005: An existing request matches the need.
6. R006: The requester is not entitled to use this route.
7. R007: Delivery date is missing.
8. R008: Applicable policy versions conflict.
9. R009: A second complete standard request.

A useful system must distinguish a prepared draft from a question, an exception or a record that already exists.

↑ Back to contents

7. What published practice contributes

• OpenAI’s internal data agent: its published account connects organisational context, permissions, existing work surfaces and evaluation. It is a first-party implementation account, not proof of a benefit at CedarWorks. [1]
• Anthropic’s tool-design guidance: it recommends meaningful workflows and evaluation tasks with verifiable outcomes. This supports examining the unit of work before adding tools. [2]
• GOV.UK Chat: its reported testing combines research methods and identifies useful conversational experiences alongside clarification and support needs. Chat remains a credible option. [3]
• Microsoft Research’s interaction guidelines: capability communication, contextual relevance, correction and uncertainty provide useful design questions. [4]
• GDS observation guidance: studying activity in context can help reveal surrounding work, with appropriate planning and consent. [5]

These sources cannot determine the client’s bottleneck, exception frequency, implementation cost or return on investment. Those require local evidence.

↑ Back to contents

8. Your recommendation

Write a 700–1,000 word decision memo containing:

1. The primary user, task and precise endpoint.
2. A comparison of at least three options, including a non-AI alternative.
3. A role-by-role allocation of the available 30 person-hours.
4. A workflow map showing clarification, correction, review and handoff.
5. An example output packet and a task-level evaluation plan.
6. Evidence that would change the decision and conditions that would stop a pilot.

Consider accessibility: can a keyboard-only user find missing fields, inspect sources and correct mistakes? If you use AI, disclose its contribution and explain what you checked.

Pause here before opening the worked analysis.

↑ Back to contents

9. Worked analysis and practical guide

Open the author’s conditional recommendation

Use a small amount of the remaining capacity to investigate contrasting situations. Then compare guided request preparation with an improved existing form on equivalent tasks. Retain a separate policy-enquiry path. Keep autonomous submission outside this increment.

This is a proposed response, not a demonstrated outcome. If the form achieves the same endpoint with less effort and maintenance, choose it. If most use is policy explanation, improving chat may be the better decision.

A feasible allocation

Activity	Builder	Process owner	Technical reviewer
Follow-up research and synthesis	4	2	0
Compare low-fidelity designs	4	1	0
Build bounded local prototype	9	0	0
Evaluate and inspect failures	5	1	2
Handover and scope decision	2	0	0
Total hours	24	4	2

Define the output states

• READY: a complete packet prepared for authorised review.
• CLARIFY: required user-supplied information is missing.
• HANDOFF: authority, policy or evidence needs another role.
• EXISTING: a matching request already exists.

READY never means approved or submitted. Every prototype packet should preserve submitted: false.

Expected dispositions for the teaching scenarios

R001: READY; R002: CLARIFY; R003: HANDOFF; R004: HANDOFF; R005: EXISTING; R006: HANDOFF; R007: CLARIFY; R008: HANDOFF; R009: READY.

Keep the implementation bounded

1. Offer a policy-enquiry path and a request-preparation path.
2. Retrieve applicable policy with a source and version.
3. Collect required fields; distinguish suggestions from confirmed values.
4. Check completeness, eligibility, freshness and duplicates.
5. Show sources, missing information and an editable draft.
6. Identify the next owner without performing a live write.

A paper form or local prototype can test these interactions. Live identity, inventory, model integration and transactional recovery should be justified and scoped separately.

↑ Back to contents

10. Evaluation and pilot design

Evaluate the whole task and the work distributed across people.

• Task outcome: an appropriate packet, clarification, handoff or existing-record link.
• False completion: unsupported readiness, approval, availability or submission claims.
• Human effort: requester, coordinator and reviewer time, including corrections.
• Elapsed time: time to the agreed endpoint, including waits.
• Usability: source inspection, correction, accessibility and abandonment.
• Operating burden: maintenance, exception handling and support.

Set acceptance rules before testing. A proposed design target is zero false completion on the supplied critical scenarios, explicit missing information, inspectable sources and no live writes. Meeting that target on nine synthetic tasks would not establish production readiness.

The optional local exercise

The accompanying Word teaching pack includes nine synthetic task contracts, scripted baseline and reference outputs, a Python checker and 12 unit tests. All 12 tests passed during preparation.

The deliberately flawed baseline matches two of nine contracts; the reference matches nine of nine. These are constructed teaching outcomes. They are not model accuracy scores, an AI improvement benchmark or evidence of business benefit.

The checker validates declared fields. It cannot establish that evidence is true, identity is genuine or an interface is usable.

Stop or revise when

• The prototype invents required facts or hides conflicting sources.
• It implies an approval or submission that did not occur.
• Correction work increases beyond the agreed limit.
• A simpler form achieves the same outcome more reliably.
• The necessary users, policy owners or evidence are unavailable.

↑ Back to contents

11. Exercises and reusable prompts

Foundation: map one task

Choose a familiar activity. Record what happens before and after an AI answer. Identify the artifact that lets the next person continue. Mark observations separately from assumptions.

Practitioner: compare two designs

Create a paper or local guided workflow and a clearer form. Use the nine scenarios, add a correction step and compare remaining human work. Keep the task endpoint consistent.

Advanced: scope a future submission capability

Specify identity, entitlement, duplicate prevention, transaction confirmation, retries, cancellation and recovery. Explain which evidence would justify moving from preparation to an authorised write.

Student project

Apply the case to travel requests, onboarding, maintenance or customer returns. Submit an evidence map, two alternative designs, contrasting task scenarios and a decision memo. Obtain appropriate permission before observing people or collecting organisational data.

Prompt: observation synthesis

Review these observation notes.
Separate:
1. Recorded behaviour
2. Participant explanation
3. Our interpretation
4. Evidence gaps

Propose at least one alternative explanation for each interpretation.
Do not invent frequency, motives, time savings or outcomes.

Prompt: scope review

Given this user task and prototype:
- Define the exact endpoint.
- Identify what remains for the user and reviewer.
- Compare chat, guided preparation and a non-AI form.
- Flag missing authority, unsupported completion claims and hidden correction work.
- State what evidence would change the recommendation.

Prompt: packet inspection

Inspect this draft packet against the supplied contract.
Distinguish user-confirmed facts from suggestions.
Identify missing fields, conflicting sources and stale evidence.
Recommend READY, CLARIFY, HANDOFF or EXISTING with a concise justification.
Do not infer approval, availability or submission.

Freelancer application

Offer workflow discovery and bounded prototype development as a defined engagement. Agree inputs, user access, process ownership, acceptance criteria, exclusions and handover. Price discovery, implementation and live integration separately.

↑ Back to contents

12. Teaching and assessment

Assign the narrative and exhibits before revealing the worked analysis. Ask learners to submit an initial recommendation, debate alternatives and revise their decision after inspecting the task scenarios.

Suggested 90-minute session: endpoint definition, 10 minutes; observation and hypothesis sorting, 15; option comparison, 20; packet exercise, 20; counterarguments and revision, 15; transfer and exit memo, 10.

Assessment criterion	Weight
Problem and evidence reasoning	20%
Alternatives and judgment	20%
Workflow design	20%
Verification, correction and recovery	25%
Communication and professional practice	15%

Reward a defensible conclusion rather than agreement with the author. A recommendation to retain chat or improve the form can be strong when it addresses the task, evidence and constraints.

↑ Back to contents

13. References and evidence limits

1. Xu, Bonnie; Suresh, Aravind; Tang, Emma. Inside OpenAI’s in-house data agent. OpenAI, 29 January 2026. First-party engineering account.
2. Aizawa, Ken. Writing effective tools for AI agents—using AI agents. Anthropic, 11 September 2025. Engineering guidance and experience.
3. Dub, Sam; McDonald, Sharon. 5 things we learned testing GOV.UK Chat: an AI assistant for government. Government Digital Service, 16 March 2026. Reported public-service testing.
4. Amershi, Saleema, and colleagues. Guidelines for Human-AI Interaction. CHI 2019. Research framework for interaction design.
5. GOV.UK Service Manual. Contextual research and observation. Published 1 September 2017. User-research method guidance.

Evidence boundary: the sources inform the questions and design principles. The CedarWorks narrative, timings, capacity, task contract, recommendations and prototype criteria are original teaching designs. First-party company accounts are not independent replications. No source establishes that the proposed intervention will save time at CedarWorks.

Review status: source-grounded, AI-assisted author preparation with local exercise testing. Independent subject-expert review and classroom validation have not been obtained.

↑ Back to contents

14. Author and contact

Ravindra Dastikop
AIYING — Accomplishing with AI
The Effective AI Engineer: A Case-Based Approach

Email: ravindra.dastikop@gmail.com
Phone: +91 84532 12131

Choose Work That Matters

After the assistant answers, what must someone still do?

↑ Back to contents

The Effective AI Engineer #002: The Engineer Who Removed the Agent

THE EFFECTIVE AI ENGINEER

CASE #002 · CASE 2 OF 16

The Engineer Who Removed the Agent

Applying Lateral Thinking to Create Meaningful Impact with AI

By Ravindra Dastikop

Engineering focus: Architecture selection and the allocation of model judgment.

Research-grounded fiction. All characters, events and test observations are invented.
Research checked: 4 November 2026.

Contents

1. The situation and impact sought
2. The story
3. The thinking tool explained
4. The thinking made visible—including AI’s contribution
5. The engineering decision
6. The evidence behind the case
7. Try the move
8. Take it into your work—including limits of transfer
9. Reusable artifact and sources

1. The situation and impact sought

Rohan’s team had built an agent that prepared onboarding checklists for customers of a software platform. It read requests, looked up account information and chose tools to assemble the checklist.

The prototype handled useful examples. Now the team needed to decide what deserved production investment.

Success meant an accurate checklist based on the customer’s plan, region and requested integrations, with unresolved information visible to a specialist. The system was not authorized to activate accounts or change subscriptions.

Rohan had a test environment, approved request examples and access-controlled account lookup services. He needed a baseline for correctness, specialist review time, response time and operating cost.

The question was whether model-directed planning was necessary for this particular task. Removing it was an option to investigate, not the project’s starting conclusion.

↑ Back to contents

2. The story

On Wednesday, Rohan watched the agent complete a difficult example.

The request mentioned two products by their old names, referred to an earlier conversation and included a half-finished integration description. The agent found relevant records, asked for missing details and produced a useful draft.

His manager smiled. “That’s the flexibility we wanted.”

Rohan agreed.

At the next architecture review, he opened with that success. Then he asked the team to describe a normal request.

Maya, the onboarding specialist, drew five boxes: identify the account, confirm its plan, check regional availability, identify integrations and prepare the checklist.

“What determines the order?” Rohan asked.

“Dependencies,” she said. “You need the account before you can trust the plan lookup.”

“And when does the order need to change?”

They examined examples. Some needed clarification, but most still followed the same dependencies.

A colleague pointed at the prototype. “It already works. Why redesign it?”

“I want to know what else could work,” Rohan said.

He opened a fresh design note. In one column he listed the agent’s responsibilities. In another he wrote what each responsibility accomplished.

The agent chose when to look up the account. But the account was always needed.

It chose whether to check availability. But availability had to be checked for every relevant product.

It decided when the checklist was complete. But Maya already had explicit requirements for completeness.

There was still genuine uncertainty. Customers described integrations in inconsistent language. Sometimes the request referred to an undocumented arrangement. Neither a fixed checklist nor a confident model could settle that without more evidence.

Rohan asked an approved AI assistant to generate architectures that distributed those responsibilities differently. He included the successful difficult example and told it to preserve that capability in the comparison.

The first suggestions were three variations of the existing agent.

“Change who decides the next step,” he wrote. “Do not just change the model or framework.”

The next set contained a form with rules, a fixed workflow using a model for interpretation, and a workflow that escalated unusual requests to a bounded agent or specialist.

The model proposed routing by its own confidence. Maya objected.

“What does its confidence establish?”

They replaced that shortcut with observable conditions: missing fields, conflicting account records, unrecognized integration names and unsupported arrangements.

Rohan built the fixed workflow first.

The model interpreted the customer’s wording into a defined set of fields. Code checked the field types and allowed values. The backend verified that the operator could access the account. Required lookups followed a known dependency order. Rules built the checklist from verified records.

His first version looked reassuringly orderly.

Then Maya broke it.

A customer referred to a legacy integration with a nickname. The model mapped it to the wrong supported integration. The output passed the schema checks.

“Valid format,” she said, “wrong meaning.”

Rohan added a requirement to show the source phrase alongside the proposed mapping. Unrecognized or ambiguous descriptions went to review. They added aliases only after the relevant team confirmed them.

Another test exposed a changed regional-availability rule. The workflow had applied an outdated version. Rohan moved that rule behind a maintained service rather than embedding a copy in the prompt.

Neither defect had been solved merely by removing the agent.

For comparison, they ran the original agent, the fixed workflow and a form-and-rules baseline against reserved requests. Each run began with clean state and the same permitted records. They tested routine cases and exceptions separately.

The fixed workflow’s clearer execution path made the routine cases easier for the team to inspect. The form handled clean inputs without model calls. The original agent remained useful on some requests requiring exploratory investigation.

The fictional results supported a limited release of the workflow for the defined routine category, with specialists handling exceptions. An exception agent remained a candidate for a separate test; they did not deploy it simply because the architecture diagram had space for one.

At the final review, the manager asked, “So you took the intelligence out?”

Rohan pointed to the interpretation step, the verified records and Maya’s review queue.

“We changed where the decisions happen.”

The agent had been an achievement. Treating it as the only possible architecture would have limited what they could discover next.

↑ Back to contents

3. The thinking tool explained

The tool is Creative Challenge.

In Serious Creativity, de Bono distinguishes it from criticism. A critical review asks whether something is inadequate. Creative Challenge explores whether the current approach is the only possible way, even when it works well. See the Challenge chapter, printed pages 104–105, and the summary on page 314.

Apply it by:

1. Selecting an accepted design choice.
2. Explaining why it exists and what function it serves.
3. Treating its necessity as open to exploration.
4. Generating alternatives that preserve the purpose.
5. Comparing them with the existing approach afterward.

The five steps above are a practical adaptation, not a verbatim procedure from the book.

Rohan challenges the necessity of model-directed control. He does not need to declare the existing agent defective before considering other designs.

Removing a component is one possible result. Keeping it, narrowing it or discovering a more valuable use for it are equally legitimate outcomes.

No PO statement is used in this case. Creative Challenge is a distinct method, and the story should not turn it into a slogan that all agents are unnecessary.

↑ Back to contents

4. The thinking made visible—including AI’s contribution

Focus: Prepare a correct, reviewable onboarding checklist.

Accepted choice: Let the model select the next step and tool.

Assumption examined: Variable customer language requires variable control over the entire process.

Challenge: Which decisions actually require model judgment, and which follow known requirements?

The alternatives change the location of decision-making:

• Customers supply structured inputs; rules handle the rest.
• A model interprets language; code controls the sequence.
• Code handles routine work; a bounded agent investigates selected exceptions.
• A model directs the broader process within enforced limits.

AI expands architecture options and helps describe their failure modes. Rohan supplies constraints and rejects superficial variants. Maya checks whether suggested designs preserve actual domain requirements.

Reusable prompt:

Apply Creative Challenge to this architecture.

Current approach: [design].
Useful outcome: [result].
Why the approach was chosen: [reasons].
Known task dependencies: [facts].
Uncertainties: [observations].
Constraints and authority: [limits].

Do not assume the current design is faulty.

Separate language interpretation, next-step selection, authorization and execution.

Generate four materially different allocations of these responsibilities. Include a non-agent option and a case where retaining the agent is justified.

For each, explain what flexibility is lost, what remains uncertain and the smallest comparison that could inform a decision.

Do not use model confidence alone as proof of correctness or invent performance results.

The prompt is an exploration aid. It cannot establish which architecture is best without implementation and evidence.

↑ Back to contents

5. The engineering decision

Architecture	Suitable conditions	Main burden	Decision in this story
Form and rules	Inputs and choices can be specified	Input friction and rule maintenance	Keep as baseline
Model interpretation with fixed workflow	Language varies but dependencies are known	Semantic errors and exception handling	Limited routine release
Workflow with exception agent	Some exceptions need bounded investigation	Routing, coordination and added evaluation	Test separately
Model-directed agent	Useful next steps depend on discoveries	Variable trajectories and execution controls	Retain as comparator and candidate for exploratory work

“Single model call” does not mean deterministic behavior. A workflow with model interpretation still needs evaluation. A schema checks shape and permitted values; it does not prove that an integration was understood correctly.

Authorization, service validation, timeouts, retries and duplicate-action protection belong in execution code where applicable. Choosing fewer model-directed decisions does not remove ordinary distributed-system failures.

The selected scope deserves investment only if acceptable outcomes and review effort compare favourably with the simpler baseline. Maintenance and exception-handling costs count alongside inference.

↑ Back to contents

6. The evidence behind the case

Anthropic distinguishes workflows with predefined code paths from agents that direct their own processes. It recommends beginning with a simple adequate solution and considering the cost and latency of additional autonomy [1].

OpenAI’s FDE account describes replacing custom implementations with shared Codex-based capabilities as the underlying product improved [2]. This supports reassessing architecture over time, not a general prescription to remove agents.

Measuring Agents in Production reports 20 case studies and a survey of 86 practitioners across 26 domains. Simple, controllable approaches were common in that sample [3]. The study does not prove that fixed workflows outperform agents on every task.

Anthropic’s evaluation guidance explains why actual outcomes, repeated trials and controlled environments matter for agent evaluation [4].

Our onboarding workflow and its fictional observations are original case adaptations. No cited source validates this particular design or claims that its authors used Creative Challenge.

↑ Back to contents

7. Try the move

Select an AI workflow that already succeeds on some tasks.

Write down one design choice everyone treats as necessary. Explain its function before questioning it.

For a low-code exercise, list each decision and classify it as:

• A known requirement or dependency.
• Interpretation of uncertain information.
• A permission decision.
• Selection of a genuinely uncertain next step.

Ask AI for different allocations of those responsibilities. Review the classifications with a domain colleague.

For a technical experiment, implement one bounded alternative alongside the current approach and a simple baseline. Use the same information access and success criteria.

Reserve ordinary, ambiguous and unsupported examples. Test missing records, stale rules, access denial and interrupted services. Run repeated trials where model outputs vary. Keep test environments isolated and reset state.

Measure accepted checklist quality, unsupported mappings, unnecessary tool calls, review time, response time and cost. Report outcomes by category so strong routine performance cannot hide poor exception handling.

Before testing, define what would justify keeping the agent and what would justify narrowing its role. A learning experiment should permit either result.

↑ Back to contents

8. Take it into your work—including limits of transfer

For the engineer: identify the smallest part of a task that actually requires model-directed exploration. Compare alternative placements of that judgment.

For the technical leader: reward justified architecture decisions, including retaining a complex approach when it earns its cost. A short diagram is not itself evidence of effectiveness.

Creative Challenge fits design reviews, successful prototypes and periodic architecture reassessment. It can be used before failure.

A fixed workflow may fit poorly when tools, evidence and useful next steps cannot reasonably be specified in advance. Overly rigid branching can create maintenance costs and brittle exception paths.

Prefer a bounded comparison when switching costs are large. During urgent incidents, restore service first and schedule broader exploration separately.

The story establishes no measured return on investment. Its intended lesson is a method for discovering and comparing alternatives.

Transfer question:
Which successful part of your architecture have you never explored alternatives to because it already works?

↑ Back to contents

9. Reusable artifact and sources

Decision Allocation Record

• Useful outcome and acceptance criteria:
• Current architecture and reasons for it:
• Accepted design choice being challenged:
• Function that must be preserved:
• Known dependencies:
• Decisions requiring interpretation:
• Decisions requiring dynamic exploration:
• Authorization and execution boundaries:
• Three alternative allocations:
• Baseline and reserved test cases:
• Routine and exception results:
• Quality, review effort, latency and total cost:
• What would justify retaining the agent:
• Decision owner and reassessment date:

References

Checked 4 November 2026.

1. Anthropic. Building effective agents. 19 December 2024, with updated-tooling notice.
2. OpenAI. What FDE means at OpenAI. 22 October 2026.
3. Pan et al. Measuring Agents in Production. ICML 2026.
4. Anthropic. Demystifying evals for AI agents. 9 January 2026.
5. Edward de Bono. Serious Creativity, 1992, Challenge chapter, printed pages 104–105; summary page 314, supplied project edition.

Publication note: Rohan, Maya, their company and all experimental observations are fictional. The prompt, workflow and artifact are contemporary adaptations. The references support engineering principles rather than comparative performance claims about this invented system.

↑ Back to contents

The Effective AI Engineer
Case 2 of a 16-part series

By Ravindra Dastikop

AIYING — Accomplishing with AI

AI Era Realities #1 | The Death of the Billable Hour: Why Intelligence is Your New Unit of Work

The Death of the Billable Hour: Why Intelligence is Your New Unit of Work

For over a century, the professional world operated on a simple, industrial formula: Time + Effort = Value.

We measured productivity by the clock. Companies tracked billable hours, managers monitored filled desks, and employees proudly wore their 60-hour workweeks like badges of honor. Success meant opening a blank document, grinding for hours, and delivering a high volume of tasks.

But we have entered a new operational reality. In the AI era, that framework has completely collapsed.

As highlighted in the operational realities framework, Reality #1 is absolute: Intelligence becomes the new unit of work. Work is no longer measured by hours spent or tasks completed, but by the quality of intelligence embedded in outputs.

Here is a deep dive into what this shift means, how it plays out in the real world, and how you must adapt to survive.

---

1. The Core Shift: From Effort to Embedded Intelligence

When an AI assistant can draft a legal brief, write a functional software script, or build a marketing campaign in less than thirty seconds, the market value of raw execution drops to zero.

If anyone can generate volume instantly, volume ceases to be a competitive advantage.

The human contribution has fundamentally shifted. It has moved away from the mechanical act of doing and climbed up the cognitive ladder. Today, value creation centers entirely on your ability to:

• Think clearly amid data noise.
• Define problems precisely before touching a keyboard.
• Make sound judgments on what the AI produces.

📊 The Productivity Paradigm Shift

OLD METRIC (Industrial / Effort-Based)
[ 8 Hours of Labor ] ──> [ Task Completion ] ──> [ High Volume Output ]
                                                        │
                                                        ▼ (Value: Low)

NEW METRIC (AI Era / Intelligence-Based)
[ Clear Thinking ] ──> [ Precise Problem Definition ] ──> [ Embedded Intelligence ]
                                                        │
                                                        ▼ (Value: High)

In this new matrix, traditional effort-based productivity metrics are utterly replaced. The core unit of value creation is no longer how hard you work, but how deeply you think.

---

2. Real-World Example: The Software Architect vs. The Code Generator

To understand how this reality changes day-to-day work, let’s look at how a software project unfolds under the old paradigm versus the new AI operational reality.

The Old Way (Effort-Based)

A developer receives a vague feature request: "We need a user dashboard." The developer spends 20 hours writing boilerplate code, setting up databases, and debugging syntax. They are judged on the hours spent and the lines of code delivered. The output is massive, but the core intelligence is low because the problem wasn't deeply analyzed beforehand.

The New Way (Intelligence-Based)

The developer uses an AI tool that can generate the dashboard code in 10 seconds. However, if the developer defines the problem poorly, the AI will flawlessly generate a completely useless feature.

Instead of typing code, the developer spends 2 hours doing the real work:

1. Defining the problem precisely: Identifying exactly which metrics matter to the user and why.
2. Thinking clearly: Structuralizing the data flow and edge cases.
3. Making sound judgments: Reviewing the AI’s code for hidden security flaws and logical biases.

The actual unit of work wasn't the code execution; it was the intelligence embedded into the AI prompt and the final validation.

---

3. Visualizing the Leverage of Clear Thought

When intelligence is the unit of work, a bad problem definition amplified by AI creates immediate chaos. Conversely, precise thought creates massive leverage.

THE AI AMPLIFIER EFFECT

[ Vague, Muddy Thinking ] ──> 🤖 AI ──> ❌ Highly Polished, Scaled Mistakes
[ Clear, Precise Thought ] ──> 🤖 AI ──>  Elite, High-Value Solutions

If you do not think clearly before you engage with AI, you are simply automating incompetence. The precision of your initial thought dictates the value of the final asset.

---

4. How to Adapt to the New Unit of Work

To thrive in a landscape where intelligence is the currency, you must change how you approach your workday:

• Stop Tracking Time, Track Quality: Evaluate your day not by how busy you were, but by the depth of judgment you applied to your projects.
• Slow Down to Speed Up: Spend 80% of your time defining the exact boundaries of a problem before letting AI solve it.
• Become an Elite Critic: Your job is no longer to write the first draft; your job is to be the ultimate judge, filtering out generic AI outputs and injecting unique human insight.

The industrial era rewarded the compliance of the clock. The AI era rewards the clarity of the mind. To create real value tomorrow, stop focusing on doing more tasks, and start focusing on embedding higher-quality intelligence into everything you produce.

---