THE EFFECTIVE AI ENGINEER · A CASE-BASED APPROACH
Case #018: The Builder Who Watched the User Work
Choose Work That Matters
A convincing answer can still leave the user’s task unfinished.
1. Learning objectives
After working through this case, you should be able to:
- Define a useful task outcome beyond the quality of an AI answer.
- Separate observed behaviour, interpretation, hypotheses and design decisions.
- Map verification, correction, exceptions and handoffs around an AI interaction.
- Compare an AI workflow with a credible non-AI alternative.
- Specify a bounded prototype and an evaluation plan before claiming success.
For: students, freshers, freelancers, AI builders, developers, engineers, product owners, functional teams and faculty. Coding: optional.
2. The case: an answer and an unfinished task
The demonstration
Nikhil Desai, an independent AI builder, arrived at CedarWorks Services with a polished equipment-request assistant. The business handled requests from employees joining projects or replacing unsuitable devices. Sana, the operations sponsor, wanted fewer repetitive questions for the coordination team.
Nikhil’s assistant searched an approved policy collection. It explained which equipment categories suited different roles, identified required information and described when an exception was needed. Its answers were readable and included references.
During the demonstration, Sana asked about a new analyst who needed a portable computer. The assistant suggested a standard category and supplied a plausible justification. Sana liked the result.
The prototype was deliberately read-only. It could not verify live stock, create a request record or submit anything. Nikhil had no integration agreement and had kept those actions outside his scope.
Then Sana asked: “How many requests will this remove from the coordinator’s queue?”
Nikhil had tested answers to prepared questions. He had not watched a request move from an employee’s need to a packet ready for authorised review. His demonstration could not answer Sana’s question.
What happened after the answer
Before leaving, Nikhil asked to watch Priya, a coordinator, prepare a request using synthetic details. She read the assistant’s answer and said it was useful. Then she opened another window.
She needed a cost centre. She checked the applicable policy version, found delivery details, searched for an existing request and verified whether the item was standard. She checked the age of the stock export. Finally, she copied information into the request form and attached the supporting reference.
The assistant had answered a question. Priya still had to assemble an artifact that another person could review.
A missing cost centre could not be guessed from a familiar team name. Old inventory information could not justify a promise of availability. A nonstandard request required an authorised decision. An existing request should be linked rather than duplicated.
A useful observation, with limits
Nikhil began recording what happened after the answer. He saw possible opportunities to reduce copying and make missing information visible. He also saw necessary checks that should not simply disappear.
However, one replay could not establish a representative workflow. Perhaps Priya handled unusual exceptions. Perhaps requesters could provide information directly. Perhaps clearer labels and validation in the existing form would solve most of the problem.
Some users might only want a policy explanation. Replacing every conversation with a long intake form could make their experience worse.
The delivery constraint
Nikhil had a fictional capacity limit of 30 person-hours: 24 builder hours, four process-owner or coordinator hours and two technical-review hours. Sana expected a revised demonstration the following week.
He could improve the chat, create guided request preparation, investigate autonomous submission, or improve the existing form. He could not silently increase the budget or assume unlimited access to staff.
What should Nikhil deliver next—and what evidence would make that decision defensible?
3. The decision
Recommend the next increment with the information available. No later business outcome is supplied.
- Who is the primary user, and what task are they trying to complete?
- Does success mean understanding policy, preparing a request or submitting one?
- Which steps require clarification, verification or an authorised human decision?
- Which work could a better form remove?
- What evidence would justify changing the recommendation?
The title describes the builder’s action. It does not prove that observation led to a superior product.
4. Exhibit 1: the surrounding workflow
The following figures are invented for an arithmetic and reasoning exercise. They are not measurements or company results.
| Step | Manual | Chat-assisted |
|---|---|---|
| Clarify the need | 3 | 5 |
| Find policy evidence | 4 | 2 |
| Check eligibility and stock | 3 | 4 |
| Enter the request | 4 | 4 |
| Review the packet | 2 | 1 |
| Route for review | 2 | 1 |
| Total active effort | 18 | 17 |
Waiting time is excluded. A faster answer does not establish an equivalent reduction in total task effort.
Separate observation from interpretation
| Observed in the fictional replay | Hypothesis to investigate |
|---|---|
| Priya opened another record. | Required context may be missing from the interaction. |
| She checked for an existing request. | Duplicate handling may be part of the task. |
| She checked policy and stock dates. | The output may need visible evidence freshness. |
| She copied text into a form. | A reviewable packet or better form may reduce transfer effort. |
Ask what each action accomplishes. Do not infer frequency, motivation or organisation-wide practice from a single replay.
5. Exhibit 2: four delivery options
| Option | Proposed work | Indicative builder hours | Main uncertainty |
|---|---|---|---|
| A. Improve chat | Better retrieval, wording and references. | 12–18 | How much downstream work remains? |
| B. Prepare a packet | Guided fields, source links, clarification and reviewable draft. | 18–24 | Can corrections and exceptions fit the scope? |
| C. Submit autonomously | Request creation and submission tools. | More than 24; not reliably estimated. | Identity, authority, retries and recovery are undefined. |
| D. Improve the existing form | Required fields, clearer help and routing guidance. | 8–14 | How much value actually requires a language model? |
These are fictional planning ranges. The additional six hours of process-owner and reviewer capacity are separate from the builder’s allocation.
Authority boundary: the current prototype can prepare material for review. It cannot approve a request, reserve inventory, purchase equipment or submit a live transaction.
6. Exhibit 3: nine task scenarios
Before reading the worked analysis, decide how a request-preparation prototype should respond to each scenario.
- R001: Complete standard request with current evidence.
- R002: Cost centre is missing.
- R003: Stock evidence is stale.
- R004: The requested item is nonstandard.
- R005: An existing request matches the need.
- R006: The requester is not entitled to use this route.
- R007: Delivery date is missing.
- R008: Applicable policy versions conflict.
- R009: A second complete standard request.
A useful system must distinguish a prepared draft from a question, an exception or a record that already exists.
7. What published practice contributes
- OpenAI’s internal data agent: its published account connects organisational context, permissions, existing work surfaces and evaluation. It is a first-party implementation account, not proof of a benefit at CedarWorks. [1]
- Anthropic’s tool-design guidance: it recommends meaningful workflows and evaluation tasks with verifiable outcomes. This supports examining the unit of work before adding tools. [2]
- GOV.UK Chat: its reported testing combines research methods and identifies useful conversational experiences alongside clarification and support needs. Chat remains a credible option. [3]
- Microsoft Research’s interaction guidelines: capability communication, contextual relevance, correction and uncertainty provide useful design questions. [4]
- GDS observation guidance: studying activity in context can help reveal surrounding work, with appropriate planning and consent. [5]
These sources cannot determine the client’s bottleneck, exception frequency, implementation cost or return on investment. Those require local evidence.
8. Your recommendation
Write a 700–1,000 word decision memo containing:
- The primary user, task and precise endpoint.
- A comparison of at least three options, including a non-AI alternative.
- A role-by-role allocation of the available 30 person-hours.
- A workflow map showing clarification, correction, review and handoff.
- An example output packet and a task-level evaluation plan.
- Evidence that would change the decision and conditions that would stop a pilot.
Consider accessibility: can a keyboard-only user find missing fields, inspect sources and correct mistakes? If you use AI, disclose its contribution and explain what you checked.
Pause here before opening the worked analysis.
9. Worked analysis and practical guide
Open the author’s conditional recommendation
Use a small amount of the remaining capacity to investigate contrasting situations. Then compare guided request preparation with an improved existing form on equivalent tasks. Retain a separate policy-enquiry path. Keep autonomous submission outside this increment.
This is a proposed response, not a demonstrated outcome. If the form achieves the same endpoint with less effort and maintenance, choose it. If most use is policy explanation, improving chat may be the better decision.
A feasible allocation
| Activity | Builder | Process owner | Technical reviewer |
|---|---|---|---|
| Follow-up research and synthesis | 4 | 2 | 0 |
| Compare low-fidelity designs | 4 | 1 | 0 |
| Build bounded local prototype | 9 | 0 | 0 |
| Evaluate and inspect failures | 5 | 1 | 2 |
| Handover and scope decision | 2 | 0 | 0 |
| Total hours | 24 | 4 | 2 |
Define the output states
- READY: a complete packet prepared for authorised review.
- CLARIFY: required user-supplied information is missing.
- HANDOFF: authority, policy or evidence needs another role.
- EXISTING: a matching request already exists.
READY never means approved or submitted. Every prototype packet should
preserve submitted: false.
Expected dispositions for the teaching scenarios
R001: READY; R002: CLARIFY; R003: HANDOFF; R004: HANDOFF; R005: EXISTING; R006: HANDOFF; R007: CLARIFY; R008: HANDOFF; R009: READY.
Keep the implementation bounded
- Offer a policy-enquiry path and a request-preparation path.
- Retrieve applicable policy with a source and version.
- Collect required fields; distinguish suggestions from confirmed values.
- Check completeness, eligibility, freshness and duplicates.
- Show sources, missing information and an editable draft.
- Identify the next owner without performing a live write.
A paper form or local prototype can test these interactions. Live identity, inventory, model integration and transactional recovery should be justified and scoped separately.
10. Evaluation and pilot design
Evaluate the whole task and the work distributed across people.
- Task outcome: an appropriate packet, clarification, handoff or existing-record link.
- False completion: unsupported readiness, approval, availability or submission claims.
- Human effort: requester, coordinator and reviewer time, including corrections.
- Elapsed time: time to the agreed endpoint, including waits.
- Usability: source inspection, correction, accessibility and abandonment.
- Operating burden: maintenance, exception handling and support.
Set acceptance rules before testing. A proposed design target is zero false completion on the supplied critical scenarios, explicit missing information, inspectable sources and no live writes. Meeting that target on nine synthetic tasks would not establish production readiness.
The optional local exercise
The accompanying Word teaching pack includes nine synthetic task contracts, scripted baseline and reference outputs, a Python checker and 12 unit tests. All 12 tests passed during preparation.
The deliberately flawed baseline matches two of nine contracts; the reference matches nine of nine. These are constructed teaching outcomes. They are not model accuracy scores, an AI improvement benchmark or evidence of business benefit.
The checker validates declared fields. It cannot establish that evidence is true, identity is genuine or an interface is usable.
Stop or revise when
- The prototype invents required facts or hides conflicting sources.
- It implies an approval or submission that did not occur.
- Correction work increases beyond the agreed limit.
- A simpler form achieves the same outcome more reliably.
- The necessary users, policy owners or evidence are unavailable.
11. Exercises and reusable prompts
Foundation: map one task
Choose a familiar activity. Record what happens before and after an AI answer. Identify the artifact that lets the next person continue. Mark observations separately from assumptions.
Practitioner: compare two designs
Create a paper or local guided workflow and a clearer form. Use the nine scenarios, add a correction step and compare remaining human work. Keep the task endpoint consistent.
Advanced: scope a future submission capability
Specify identity, entitlement, duplicate prevention, transaction confirmation, retries, cancellation and recovery. Explain which evidence would justify moving from preparation to an authorised write.
Student project
Apply the case to travel requests, onboarding, maintenance or customer returns. Submit an evidence map, two alternative designs, contrasting task scenarios and a decision memo. Obtain appropriate permission before observing people or collecting organisational data.
Prompt: observation synthesis
Review these observation notes. Separate: 1. Recorded behaviour 2. Participant explanation 3. Our interpretation 4. Evidence gaps Propose at least one alternative explanation for each interpretation. Do not invent frequency, motives, time savings or outcomes.
Prompt: scope review
Given this user task and prototype: - Define the exact endpoint. - Identify what remains for the user and reviewer. - Compare chat, guided preparation and a non-AI form. - Flag missing authority, unsupported completion claims and hidden correction work. - State what evidence would change the recommendation.
Prompt: packet inspection
Inspect this draft packet against the supplied contract. Distinguish user-confirmed facts from suggestions. Identify missing fields, conflicting sources and stale evidence. Recommend READY, CLARIFY, HANDOFF or EXISTING with a concise justification. Do not infer approval, availability or submission.
Freelancer application
Offer workflow discovery and bounded prototype development as a defined engagement. Agree inputs, user access, process ownership, acceptance criteria, exclusions and handover. Price discovery, implementation and live integration separately.
12. Teaching and assessment
Assign the narrative and exhibits before revealing the worked analysis. Ask learners to submit an initial recommendation, debate alternatives and revise their decision after inspecting the task scenarios.
Suggested 90-minute session: endpoint definition, 10 minutes; observation and hypothesis sorting, 15; option comparison, 20; packet exercise, 20; counterarguments and revision, 15; transfer and exit memo, 10.
| Assessment criterion | Weight |
|---|---|
| Problem and evidence reasoning | 20% |
| Alternatives and judgment | 20% |
| Workflow design | 20% |
| Verification, correction and recovery | 25% |
| Communication and professional practice | 15% |
Reward a defensible conclusion rather than agreement with the author. A recommendation to retain chat or improve the form can be strong when it addresses the task, evidence and constraints.
13. References and evidence limits
- Xu, Bonnie; Suresh, Aravind; Tang, Emma. Inside OpenAI’s in-house data agent. OpenAI, 29 January 2026. First-party engineering account.
- Aizawa, Ken. Writing effective tools for AI agents—using AI agents. Anthropic, 11 September 2025. Engineering guidance and experience.
- Dub, Sam; McDonald, Sharon. 5 things we learned testing GOV.UK Chat: an AI assistant for government. Government Digital Service, 16 March 2026. Reported public-service testing.
- Amershi, Saleema, and colleagues. Guidelines for Human-AI Interaction. CHI 2019. Research framework for interaction design.
- GOV.UK Service Manual. Contextual research and observation. Published 1 September 2017. User-research method guidance.
Evidence boundary: the sources inform the questions and design principles. The CedarWorks narrative, timings, capacity, task contract, recommendations and prototype criteria are original teaching designs. First-party company accounts are not independent replications. No source establishes that the proposed intervention will save time at CedarWorks.
Review status: source-grounded, AI-assisted author preparation with local exercise testing. Independent subject-expert review and classroom validation have not been obtained.
