The Effective AI Engineer · A Case-Based Approach
The Fresher Who Could Explain the Code
Choose Work That Matters
.png)
A new developer has produced a working prototype with AI assistance. The demonstration looks promising. Then a reviewer asks her to predict what happens when one input changes.
The team now faces two connected questions: Does the software behave correctly? And can its owner understand, maintain and change it?
About this case
This is an original fictional teaching scenario informed by published research and engineering accounts. The organisation, people, dialogue, schedules, costs and ticket records are invented for instruction. They are not reported events at OpenAI, Anthropic or another real company.
Research findings are identified and referenced separately. The accompanying Python exercise has been tested locally. The proposed workplace review process has not been validated as a learning intervention.
1. Learning objectives
After working through this case, you should be able to:
- Distinguish a successful demonstration from evidence of correctness, understanding and maintainable ownership.
- Compare delivery options using business outcomes, review effort, learning and reversibility.
- Derive boundary and exception tests from an agreed business contract.
- Explain a critical code path and apply that understanding to a changed requirement.
- Write a release recommendation with acceptance conditions, named responsibilities and a fallback.
Preparation: 35–45 minutes.
Discussion: 90 minutes.
Optional lab: 60–120 minutes.
Coding prerequisites: basic conditions, loops,
dictionaries and tests. A no-code pathway is provided.
2. The case
Monday morning at FieldBridge
At 9:10 on Monday, Asha Rao opened the review request she had created late on Friday. She had joined FieldBridge Software six weeks earlier after graduating. The small services company maintained an internal dashboard for a client that repaired commercial equipment.
Its next opportunity was modest: help a coordinator review a daily ticket export without manually copying each row into a priority sheet.
Meera, the delivery lead, wanted a demonstration on Tuesday afternoon. The client had agreed to watch a prototype and consider a two-week assisted pilot. A useful demonstration could create follow-on work. A misleading one could make the client believe that exceptions had been solved when the team had barely examined them.
Asha’s assignment was a CSV-in, report-out script. It would identify ticket priorities under explicit rules and report rejected rows. It would not send messages, update the ticket system, dispatch technicians or make service commitments.
An approved AI assistant had helped Asha produce an early draft. She asked it to implement the normal path, run two examples and describe the result. On Friday, two open tickets received sensible-looking priorities and a closed ticket disappeared from the proposed queue.
She posted a screenshot. Meera replied that it looked promising.
The question that interrupted the demonstration
Dev, the senior developer reviewing the change, asked what would happen to a normal-impact open ticket exactly sixty minutes old. Asha said she thought it would become P2.
Dev asked her to trace the comparison before running the script. She paused. Then he changed the capitalisation of the status field and asked the same question.
Asha understood variables and loops. But she had accepted several details because the assistant’s explanation sounded coherent. She could describe the script’s intention more confidently than its actual behaviour.
She was also unsure whether asking for time to investigate would be interpreted as a failure to use AI effectively.
Dev did not propose banning the assistant. He wanted to know whether the team had assigned ownership without providing a reliable way to establish it. A screenshot was evidence about those exact inputs. It did not show whether Asha could diagnose the next complaint.
What the coordinator needed
Leela, the client’s coordinator, received exports from several staff members. Some cells contained leading spaces; others used upper-case words. Her main concern was overlooked work, especially a ticket blocking a customer’s operations.
She also wanted a count of records the script could not interpret. Quietly discarding a row could make the queue look cleaner while creating reconciliation work later.
The priority rules were simple: valid open blocked tickets were P1; valid open normal-impact tickets aged sixty minutes or more were P2; other valid open normal-impact tickets were P3.
Repeated identifiers raised another question. Should a later row replace the first one? Leela chose a first-occurrence rule for the exercise, with later records rejected for review. She understood that this might preserve an earlier mistake and require a separate correction process.
The team also agreed that blank age was unknown, not zero. Invalid records had to remain visible in the rejection report. Every input row needed to be accounted for.
Four legitimate concerns
Asha wanted to retain responsibility and develop competence. Dev had only ninety review minutes before another deployment. Meera needed to protect the schedule and client relationship. Leela needed someone who could explain an unexpected result and correct it.
No one had measured the complete assisted workflow. The fictional manual estimate was twenty minutes per daily batch, but it had not been systematically timed. Exporting data, reviewing rejections, reconciling totals and approving actions would still take effort.
The alternatives
At 10:00, Meera asked the team to consider four options:
- Keep the demonstration plan, generate more examples with AI assistance and ask Dev to inspect the final file.
- Ask Asha to rebuild the script without AI, accepting a narrower demonstration or a delay.
- Transfer delivery to Dev while Asha observes.
- Keep Asha as the developer, freeze the scope and use the review time for prediction, testing, explanation and a small unfamiliar modification.
More examples could repeat the draft’s blind spots. A manual rebuild could spend the window on syntax. A senior takeover could protect tomorrow’s event while transferring ownership away from Asha. The bounded review could expose important gaps, but ninety minutes might not be enough.
Meera would authorise the demonstration. Dev would recommend whether a pilot was technically supportable. Asha would own the prototype and handover. Leela would control operational use.
By 10:15, Meera wanted a plan stating what would be shown on Tuesday and what would have to wait.
3. Your decision
No outcome is supplied at this stage. The title does not guarantee that Asha passes the review.
- What would you authorise: a synthetic demonstration, an assisted pilot or neither?
- Which delivery option would you choose, and why?
- How would you allocate the ninety senior-review minutes?
- What evidence would establish correctness and maintainable ownership?
- What would stop the release, and what would the fallback be?
- What additional evidence could change your recommendation?
Assignment: Write a 600–900 word decision memo, supported by an option comparison and a one-page verification plan. Cite exhibits, label assumptions and disclose AI assistance. Coding is optional.
4. Exhibit 1: The business contract
These rules are synthetic teaching requirements, not industry-wide service standards.
-
Input consists of CSV-style records with
id,status,impactandage_minutes. - Trim the ID and preserve its case. Missing or empty IDs are rejected.
- The first nonempty ID reserves that identifier, including when its row is invalid or closed. Later occurrences of that trimmed, case-sensitive ID are rejected.
-
Trim and lower-case status and impact. Allowed values are
openorclosed, andblockedornormal. -
Age must contain only digits
0–9after trimming. Zero is valid. Blank, negative and fractional values are invalid. Missing or non-string required fields are rejected. - Validate closed records too. Exclude only fully valid closed records.
- For valid open records: blocked means P1; otherwise age greater than or equal to 60 means P2; otherwise P3.
- Preserve accepted input order and do not mutate the input records.
- Return accepted records, rejected records with reasons, and an excluded count. Every input row belongs to exactly one category.
Error precedence: missing ID → duplicate ID → invalid fields → invalid age → valid closed exclusion → priority.
Rejection row numbers start at the first data row, excluding the CSV header.
5. Exhibit 2: Synthetic tickets
| Row | ID | Status | Impact | Age in minutes |
|---|---|---|---|---|
| 1 | T001 | open | blocked | 25 |
| 2 | T002 | open | normal | 90 |
| 3 | T003 | closed | blocked | 120 |
| 4 | T004 | " OPEN " | "normal " | 60 |
| 5 | T005 | open | blocked | Empty cell |
| 6 | T006 | open | normal | -5 |
| 7 | T007 | open | normal | 20 |
| 8 | T001 | open | normal | 30 |
Spaces in row 4 are intentional. Row 5 contains an empty cell, not the words “empty cell”. Row 8 repeats the first identifier.
The original clean demonstration used only rows 1, 2 and 3. Predict the full report before running the draft.
6. Exhibit 3: The draft code
This is a deliberately defective, author-created teaching draft representing an artifact accepted too quickly. It is not a transcript of an actual model response.
def triage(rows):
accepted = []
for row in rows:
if row.get("status") == "open":
age = int(row.get("age_minutes", "0") or "0")
priority = (
"P1" if row.get("impact") == "blocked"
else ("P2" if age > 60 else "P3")
)
accepted.append({
"id": row["id"],
"priority": priority
})
return {
"accepted": accepted,
"rejected": [],
"excluded": len(rows) - len(accepted)
}Investigate before editing
- For rows 4, 5, 6 and 8, distinguish the required behaviour from the draft’s actual behaviour.
- Test a normal-impact ticket at ages 59, 60 and 61.
- Identify which rejected records could be hidden inside the draft’s excluded count.
- Explain why changing one comparison operator does not repair the entire contract.
- Propose a small counterexample that could disprove your explanation.
7. Exhibit 4: What published evidence contributes
Learning and task completion are different outcomes
Shen and Tamkin studied 52 developers learning an unfamiliar Python library. AI access was randomly assigned. The assisted group showed lower immediate comprehension, without a statistically significant average time advantage. The participants were not a sample of fresh graduates. Findings about interaction patterns were exploratory associations, not randomised comparisons of teaching methods. [S1]
Assistance can also improve measured output
Cui and colleagues combined three workplace experiments involving 4,867 developers. They report more completed tasks with coding assistance and larger gains among less experienced developers. Those output measures do not themselves establish retained learning. [S2]
The learning environment matters
Kazemitabaar and colleagues studied 69 learners aged 10–17 in a structured introductory environment. AI-supported authoring improved, while subsequent manual modification and assessments did not show a general learning penalty. The population and learning conditions differ materially from workplace onboarding. [S3]
Engineering practice and measurement require context
OpenAI’s harness-engineering account describes investment in environments and feedback around agents. It is a company-reported engineering experience, not a controlled learning study or a release policy for this fictional team. [S4]
METR’s February 2026 methodological update explains how selection and time-measurement issues weakened interpretation of its later developer-productivity experiment. [S5]
These sources support investigating several outcomes. They do not establish a universal rule that AI always helps or harms learning.
8. Practice guide and worked analysis
Pause here if using this as an assignment. Complete your recommendation before opening the analysis.
Open the worked analysis
A conditional recommendation
Keep Asha responsible, freeze the scope and run a bounded review of correctness and understanding. Authorise a synthetic, read-only demonstration only if the checks are satisfactory. Treat an assisted pilot as a separate decision.
The pilot needs an operational owner, actual workflow comparison, agreed data handling and a correction process. A successful demonstration does not establish those conditions.
The hidden assumptions
One assumption is that a completed artifact establishes its owner’s competence. Another is that assessing competence requires a full manual rebuild.
A more useful question is: What evidence would let another person trust this developer with the next change?
That moves the review toward prediction, diagnosis, modification and communication. This is our teaching reframing, not a claim that the proposed process is a proven instructional intervention.
| Option | Benefit | Main limitation |
|---|---|---|
| Generate more and inspect | Preserves the current workflow. | Examples may repeat the original blind spots. |
| Rebuild without AI | Makes independent work visible. | May consume time on syntax rather than critical reasoning. |
| Senior takeover | Transfers responsibility to an experienced developer. | Uses scarce capacity and may remove the learning opportunity. |
| Bounded review | Examines correctness and understanding while retaining ownership. | Needs a clear contract, competent observation and stop rules. |
A proposed ninety-minute plan
| Minutes | Activity | Evidence produced |
|---|---|---|
| 0–15 | Agree the contract and predict outputs. | Written predictions and identified uncertainties. |
| 15–35 | Run the draft and investigate mismatches. | Input, expected result, observed result and cause. |
| 35–60 | Repair the draft and run contract checks. | Corrected behaviour and test results. |
| 60–75 | Close AI assistance and attempt a new requirement. | Independent prediction, change and explanation. |
| 75–90 | Review evidence and record the scope decision. | Acceptance conditions, owner, limits and fallback. |
These timings are proposals, not measured optimal allocations. If critical behaviour remains unresolved, narrow the demonstration or use the manual fallback.
What Asha must understand
She need not memorise every library call. She should be able to trace the boundary, distinguish invalid from excluded records, explain duplicate handling, diagnose an unexpected result and preserve important behaviour when a requirement changes.
When another decision is defensible
A senior takeover or delay may be appropriate when the deadline cannot be met safely, no competent reviewer is available or the tool would directly change live records. The recommendation depends on the small, synthetic and reversible scope of this case.
9. Hands-on lab
Download the resource pack below. The exercise uses local synthetic data and Python’s standard library. It requires no paid AI account or external API. The supplied reference was tested with Python 3.12.14.
Learner workflow
- Predict the report for the eight records.
- Run the starter and record mismatches.
- Explain each cause before editing.
- Repair the implementation while retaining its interface.
- Run the checks and add a new contract-derived test.
- Complete the independent transfer task before reading the reference.
From the extracted resource directory:
cd lab
python starter.py tickets.csv
python -m unittest -v test_triage
The starter is deliberately wrong, so initial failures are expected.
Where appropriate, substitute python3 or py
for python.
Reveal the expected full-batch report
| Category | Expected result |
|---|---|
| Accepted | T001 P1; T002 P2; T004 P2; T007 P3, in that order. |
| Rejected | Row 5: invalid age; row 6: invalid age; row 8: duplicate ID. |
| Excluded | One fully valid closed record: T003. |
| Reconciliation | 4 accepted + 3 rejected + 1 excluded = 8 input records. |
What was tested
The reference implementation passed fifteen named contract tests. They cover boundaries, normalisation, malformed and missing values, duplicate IDs, closed-row validation, empty input, accounting, accepted order and input preservation.
The sample CSV and a threshold-change exercise were also checked. Test logs and the reference implementation are in the instructor resources.
These are local exercise results. They do not establish production readiness, workplace savings or learning gains.
10. Independent transfer task
Change the normal-impact escalation threshold from 60 to 45 minutes. Preserve P1 precedence and every validation rule.
- Predict outcomes at ages 44, 45 and 46.
- Predict the priority of a valid blocked ticket aged 10 minutes.
- Identify existing test expectations that must change.
- Identify behaviours that must remain unchanged.
- Implement or specify the change without AI assistance.
- Explain the results using the contract and observed behaviour.
Official documentation may be used for syntax if the assessment permits it. The target is reasoning about the change, not memorising an API.
Reveal the transfer-task check
Normal-impact ages 44, 45 and 46 become P3, P2 and P2. A valid blocked ticket remains P1. A previous expectation that a normal-impact 59-minute ticket is P3 must change under the new threshold.
Stop conditions
Stop the demonstration-ready claim if required priorities are wrong, input accounting fails or the named owner cannot explain a critical exception. Escalate business-rule ambiguity to the coordinator and release ambiguity to the delivery lead.
11. Measuring business value and learning
Measure the work required to obtain a trustworthy result, including export, execution, reconciliation, corrections and review.
| Question | Useful evidence |
|---|---|
| Does the tool obey the contract? | Tests, representative records and reconciled totals. |
| Can its owner maintain it? | Prediction, diagnosis, novel change and explanation. |
| Does it help the coordinator? | Comparable end-to-end workflow time and error burden. |
| Is learning retained? | A parallel task after a delay, with assistance rules recorded. |
| What does support cost? | Reviewer time, rework, exceptions and escalations. |
Net effort benefit per batch = manual end-to-end minutes − assisted end-to-end minutes.
Include setup, training, review and support before claiming a pilot benefit. The case’s twenty-minute manual estimate is unvalidated.
A delayed local assessment can guide mentoring. It is not, by itself, a causal study of AI’s effect on learning. Do not present improvement after repeated practice as proof that the proposed review process caused it.
12. Learning and workplace pathways
Foundation and no-code
Trace four records through the rules on paper. Explain a defect in ordinary language. Submit a decision memo, corrected rule flow and reconciliation.
Practitioner
Repair the starter, add a test and submit a before-and-after report. Include a handover note, actual test results and an independently completed transfer task.
Advanced
Design a stronger correction policy for repeated IDs. Compare first-occurrence rejection with latest-valid-record handling. Specify provenance, audit events, access, recovery and safe updates for a real integration.
Freelancers
Scope a bounded prototype review. Ask the client for approved rules and synthetic examples. Deliver a contract, prototype, tests and handover. Estimate implementation and review separately, and state exclusions, acceptance conditions and future support ownership.
Workforce teams
Apply the same questions to an AI-assisted spreadsheet, report, document workflow or automation: can the owner explain the important decisions, detect exceptions and respond to a changed requirement?
Reusable tutor prompt
I will provide a business rule and my prediction. Ask one question that tests my reasoning. Do not give the answer before I attempt it. Identify unsupported assumptions and suggest a small counterexample.
Reusable review prompt
Compare this draft with the agreed contract. For each possible mismatch, propose an input that would expose it and identify the rule it violates. Separate definite defects from questions for the business owner. Do not invent requirements or successful tests.
13. Instructor teaching note
For assessment use, assign the narrative and exhibits first. Withhold the worked guide and instructor resources until learners submit their predictions and recommendations. Collapsed sections on a public blog are not secure answer protection.
Opening question
Would you authorise tomorrow’s demonstration? What exactly would you be authorising?
Record the answers before discussing code. Separate a synthetic demonstration, an assisted pilot and a live deployment.
| Minutes | Activity |
|---|---|
| 0–10 | Opening vote and conditions. |
| 10–25 | Facts, assumptions and missing evidence. |
| 25–40 | Trace the draft against the contract. |
| 40–60 | Compare options and allocate review time. |
| 60–75 | Challenge recommendations and reveal evidence. |
| 75–90 | Transfer task and exit recommendation. |
Discussion questions
- What does the client need accomplished?
- Which uncertainty is technical, organisational or educational?
- When should a senior developer take over?
- What evidence would justify a shorter review next time?
- Could a senior developer pass the same demo while misunderstanding it?
- What would change if the script updated live records?
Misconceptions to challenge
- “She did not type it, so she cannot own it.” Ask what maintenance responsibilities she can demonstrate.
- “Tests passed, so she understands it.” Ask for a fresh prediction and unfamiliar change.
- “The research proves AI harms every fresher.” Examine the population, task, outcome and limitations.
- “The demonstration saved twenty minutes.” Ask for a measured baseline and full workflow effort.
A defensible alternative recommendation can earn strong marks. Assess the quality of evidence and reasoning rather than agreement with the worked answer.
14. Assessment rubric
| Criterion | Weight | Strong evidence |
|---|---|---|
| Problem and evidence | 20% | Clear scope, exhibit use and qualified research claims. |
| Alternatives and judgment | 20% | Credible options, trade-offs and conditions for changing the decision. |
| Workflow or technical design | 20% | Feasible steps, correct contract and named responsibilities. |
| Verification and failure response | 25% | Boundary and exception tests, transfer, stop conditions and fallback. |
| Professional communication | 15% | Clear recommendation, assumptions, assistance disclosure and handover. |
Score each criterion from 1 to 4: 1 = substantially incomplete; 2 = material gaps; 3 = sound and usable; 4 = well supported and explicit about limits.
Weighted score = sum of each weight × score ÷ 4. Set pass requirements according to the teaching context. An aggregate score does not override unresolved critical defects.
15. Research and verification record
The research examined learning, task output, engineering practice and measurement limitations. Primary sources and counterevidence were used to avoid treating one study as a universal workplace rule.
- Evidence cutoff: 6 November 2026.
- Research findings are separated from fictional operational details.
- The reference passed fifteen named local contract tests.
- The sample report and threshold-change exercise were checked.
- The proposed mentoring process has not undergone a workplace trial.
- No external expert review or independent statistical replication was obtained.
- No production deployment or business saving is claimed.
The research dossier contains annotated sources, a claim register, source cautions and maintenance triggers. Review the case when central evidence is corrected, relevant follow-up research appears, the runtime changes or learners identify a defect.
16. References
- Shen, J. H., and Tamkin, A. How AI Impacts Skill Formation. arXiv:2601.20245v2. Read the paper. Companion account: How AI assistance impacts the formation of coding skills , Anthropic, 29 January 2026. Relevant sections: methods, main study, qualitative analysis and discussion.
- Cui, Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., and Salz, T. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Microsoft Research publication record . DOI: 10.1287/mnsc.2025.00535 . Used for the reported task-output finding, not a claim about learning.
- Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., and Grossman, T. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. CHI 2023. Read the paper. DOI: 10.1145/3544548.3580919. Relevant sections: user study, results, discussion and limitations.
- Lopopolo, R. Harness engineering: leveraging Codex in an agent-first world. OpenAI, 11 February 2026. Read the engineering account . First-party practice account; not a controlled learning study.
- Becker, J., Rush, N., Cunningham, T., Rein, D., and Mahamud, K. We are Changing our Developer Productivity Experiment Design. METR, 24 February 2026. Read the methodological update . Used to qualify generalisations about developer productivity.
17. Complete case and resource pack
Complete 26-page case: narrative, exhibits, practice guide, instructor notes, assessment and research dossier.
Teaching and practice pack: separate reader materials, instructor documents, worksheets, starter code, tests and reference implementation.
Reflection
What evidence would make you comfortable asking this person to handle the next change?
Answer that question for your own AI-assisted work: a program, spreadsheet, report, document workflow or automation. Identify one prediction, one failure test and one change you can explain independently.
