The Effective AI Engineer #020: The Harness Engineer Who Refused to Retry

THE EFFECTIVE AI ENGINEER · A CASE-BASED APPROACH

Case #020: The Harness Engineer Who Refused to Retry

Choose Work That Matters

By Ravindra Dastikop
AIYING — Accomplishing with AI

The agent reported a timeout. The service might already have completed the action.

Case status and evidence boundary

This is an original fictional teaching case informed by published engineering practices. Meridian Workplace Services, its people, incident, records and proposed design are author-created. No fictional event or result is attributed to a real company.

Sources consulted: 9 November 2026. No production system, code or recovery experiment was tested for this case. The exercises are proposed learning activities. Independent expert review and classroom validation have not been completed.

Contents

1. The learning challenge
2. The case: one request, an uncertain result
3. The reassuring search result
4. A contract the team had not checked
5. Exhibit A: what the team knows
6. Exhibit B: operation and attempt identity
7. What published practice contributes
8. Your decision assignment
9. Worked analysis
10. A proposed recovery-state contract
11. What the employee should see
12. Practical exercise: inject failures
13. Reusable AI prompts
14. Transfer activities and student project
15. Teaching notes and assessment
16. Research record and references
17. Author and contact

1. The learning challenge

Case #019 asked whether passing tests establish the intended behaviour. Case #020 moves into operational recovery:

What should an AI system do when an action may have succeeded, but confirmation has been lost?

After completing this case, you should be able to:

• Distinguish a failed response from a failed action.
• Explain idempotency in practical terms.
• Separate an intended operation from individual execution attempts.
• Design recovery across timeouts, restarts and concurrent workers.
• Define when to retry, reconcile or hand off.
• Report uncertainty without inventing success or failure.

Audience: AI builders, developers, loop and harness engineers, freelancers, technical leads, students and faculty.

Prerequisites: basic understanding of APIs. The main exercise can be completed without coding.

Here, a harness means the surrounding software that manages the agent’s execution: tool calls, state, permissions, recovery and records.

↑ Back to contents

2. The case: one request, an uncertain result

The normal path

Asha worked on the agent infrastructure team at the fictional Meridian Workplace Services. Her team had built an assistant that helped employees request replacement equipment.

The assistant collected details, prepared a reviewable packet and asked the employee to confirm submission. An authorised submission created a service request. It did not approve a purchase.

The pilot’s normal path was straightforward:

1. The employee reviewed the packet.
2. The employee confirmed submission.
3. The agent called create_service_request.
4. The service returned a request identifier.
5. The assistant showed that identifier to the employee.

The timeout

An employee confirmed a replacement-laptop request. The tool call ran for several seconds before the connection closed.

The tool wrapper returned:

Request timed out.
No response received.

The agent proposed trying again. That appeared reasonable: the harness already retried temporary failures when reading policy documents.

But this call created something.

Asha paused the automatic retry.

The missing distinction

The delivery lead, Vikram, wanted to know why. “If the call failed, why should we leave the employee waiting?”

Asha examined the available evidence. The wrapper knew that it had not received a response. It did not know whether the service had:

• Never received the request.
• Received it but not completed it.
• Created the request and lost the response.
• Continued processing after the client stopped waiting.

Repeating the call could create two requests if the first action had succeeded. Announcing failure could also prompt the employee to submit manually while the original operation was still running.

Asha’s concern was not that retries were inherently wrong. The system had classified an uncertain outcome as a failed action.

↑ Back to contents

3. The reassuring search result

Support engineer Dev searched the service dashboard. He found no matching request.

“Can we retry now?” Vikram asked.

Asha checked how the dashboard worked. Its results came from a search index that could lag behind the transaction system.

An empty search result was useful evidence. It did not establish that the original request had failed.

The team also discovered that the harness generated a new identifier for every tool-call attempt. Those identifiers helped trace execution. They did not identify a single business operation across retries.

If the agent restarted, it could produce another identifier and submit again. A prompt saying “avoid duplicates” would not change that behaviour.

↑ Back to contents

4. A contract the team had not checked

The service owner, Noor, explained that the API supported a caller-provided operation key.

Within a documented retention window, the same caller could repeat the same operation with the same key without creating a second request. Reusing that key with different request content produced a conflict.

However, the pilot wrapper had not exposed the key. It also had not documented:

• How long the key remained effective.
• How to handle an operation still in progress.
• Whether a status lookup was authoritative.
• What happened after a harness restart.
• Which layer owned retries.
• How the user’s confirmation related to a changed packet.

Asha now had a decision. Should the team patch the wrapper and continue the write-enabled pilot, pause all submissions, or retain preparation while routing uncertain operations to staff?

No later outcome is supplied. Decide from the evidence available.

↑ Back to contents

5. Exhibit A: what the team knows

All records below are fictional teaching material.

Evidence	What it establishes	What it does not establish
Employee confirmed the packet.	Permission for the displayed submission.	Permission for changed content or another business request.
Tool call timed out.	The client did not receive a timely response.	Whether the service created the request.
Dashboard search found nothing.	No match appeared in that search.	Authoritative absence from the transaction system.
Harness has an attempt ID.	One execution attempt can be traced.	That another attempt represents the same operation.
API supports operation keys.	A recovery mechanism may be available.	That the wrapper uses it correctly.
Agent says “submission failed.”	The model produced that statement.	That failure was confirmed.

Discussion question: Which statements may the assistant communicate as facts?

↑ Back to contents

6. Exhibit B: operation and attempt identity

Identifier	Purpose
Business operation ID	Identifies the single intended submission.
Attempt ID	Identifies one execution attempt.
Service request ID	Identifies the record created by the destination service.
Confirmation reference	Links the user’s approval to the reviewed packet.
Payload fingerprint	Helps detect changes to the submitted content.

Proposed design rules

• One intended submission keeps its operation identity during recovery.
• Each execution attempt receives a separate attempt ID.
• The operation key is scoped to the relevant caller and service contract.
• A material payload change requires renewed review and a separate decision.
• A new key must not be used merely to escape an unresolved operation.

These are proposed rules for this case. Their implementation must match the destination service.

Idempotency in practical terms

Under an applicable service contract, repeating the same intended operation does not repeat its business effect. This does not mean that every request is automatically safe to repeat, or that an operation is guaranteed to finish.

↑ Back to contents

7. What published practice contributes

AWS: a missing response creates an outcome dilemma

The Amazon Builders’ Library describes a resource-creation request whose response is lost. Retrying can create an unwanted duplicate. Its discussion covers caller-provided identifiers, parameter matching and the relationship between recording an identifier and performing an operation. This is relevant to Asha’s distinction between an uncertain response and an uncertain action. [1]

Stripe: idempotency is a documented service contract

Stripe documents keys for retrying supported operations. Its documentation distinguishes API v1 and v2 behaviour and describes parameter checks and retention considerations. The lesson is to inspect the exact endpoint and API version. Adding a header does not establish a universal guarantee across services. [2]

Anthropic: continuity needs more than conversational memory

Anthropic’s long-running coding-agent account uses progress artifacts and environment state to help later sessions understand earlier work. This supports treating continuity as an engineering concern. Its coding-harness design does not provide a transaction guarantee for Meridian’s external service. [3]

AWS: retries need operational limits

AWS guidance recommends bounded retries, backoff and jitter. It warns about retries at multiple layers and non-idempotent side effects. These controls address retry behaviour and load; they do not independently establish whether an uncertain operation succeeded. [4]

Author’s synthesis: Reliable recovery requires a service contract, durable operation identity, appropriate authority and trustworthy outcome evidence.

↑ Back to contents

8. Your decision assignment

Prepare a 600–900 word recommendation addressing:

1. What should happen to the uncertain submission now?
2. What should the employee be told?
3. Which capabilities may remain available during the investigation?
4. What must change before automatic write retries are enabled?
5. Who owns reconciliation and unresolved operations?
6. What evidence would justify resuming the pilot?

Alternative	Potential benefit	Main limitation
Continue with a wrapper patch.	Preserves pilot momentum.	Restart, concurrency or retention gaps may remain.
Pause writes; retain preparation.	Preserves some user value.	Staff still handle submission and recovery.
Pause the entire pilot.	Simplifies containment.	Removes useful capabilities that may not share the defect.

Explain your choice in terms of consequences, volume and available controls. A recommendation should identify what remains unknown.

Pause here before reading the worked analysis.

↑ Back to contents

9. Worked analysis

Open the author’s conditional recommendation

Do not blindly repeat the unresolved operation. Assign it to a named reconciliation owner. Keep preparation available if it remains clearly separated from submission.

Before restoring automatic write recovery:

1. Verify the service’s documented idempotency behaviour.
2. Preserve a stable operation key before the first dispatch.
3. Bind the operation to the reviewed payload and confirmation.
4. Persist recovery state across restarts.
5. Coordinate concurrent execution attempts.
6. Establish a bounded retry policy at the appropriate layer.
7. Confirm how the service reports in-progress and completed operations.
8. Provide a manual path when the result cannot be established.

A local ledger alone cannot prevent duplicates if the external service creates a record and the harness crashes before recording success. Recovery must address that gap through the service’s guarantees or another justified mechanism.

The unresolved original operation needs its own investigation. Adding a key to future requests does not retroactively protect an earlier unkeyed submission.

This recommendation has costs: delayed submission, engineering work and operational ownership. It is not evidence that every low-impact workflow requires a complex recovery platform.

↑ Back to contents

10. A proposed recovery-state contract

State	Meaning	Permitted next step
PREPARED	Packet exists; submission is not authorised.	Review and obtain confirmation.
AUTHORISED	Confirmation is bound to the packet.	Persist operation identity and dispatch under valid permissions.
IN_FLIGHT	An attempt has been dispatched.	Await or inspect its outcome.
OUTCOME_UNKNOWN	Available evidence cannot establish the result.	Reconcile; use only contract-safe recovery.
CONFIRMED	The destination confirms the created request.	Show the request ID.
REJECTED	The destination explicitly rejected the operation without creating the request.	Explain and correct as appropriate.
MANUAL_REVIEW	Automated recovery cannot resolve the operation safely.	Assign a named owner.

A timeout after dispatch moves to OUTCOME_UNKNOWN, unless stronger evidence establishes another state. An error message is not automatically proof of rejection.

State transitions should be enforced by the surrounding software. A model’s description of a transition does not make it true.

↑ Back to contents

11. What the employee should see

Unsupported message

Submission failed. Please submit again.

More accurate message

We have not yet confirmed whether your request was created. Please avoid submitting another copy while we check. Your reference is OP-204. The service team will update you.

The reference and follow-up promise must correspond to real operational records and ownership. If follow-up is unavailable, explain the available contact route instead of promising an update.

Prepared, submitted and confirmed are different user-visible states.

↑ Back to contents

12. Practical exercise: inject failures

Use a local fake service or a paper simulation. No real submissions, purchases or credentials are required.

Scenario	Expected property
Request is never delivered.	Recovery follows the documented contract without inventing success.
Service commits; response is lost.	Recovery does not create another request.
Harness crashes after dispatch.	Restart preserves the same intended operation.
Two workers resume together.	Concurrency does not produce duplicate business effects.
Same key, changed payload.	Conflict is surfaced; content is not silently replaced.
Search index lags.	Empty search does not become proof of failure.
Key retention has expired.	The old duplicate-prevention guarantee is not assumed.
Permission is revoked before execution.	An operation key does not bypass authorisation.
Retry limit is reached.	The unresolved operation remains visible and receives an owner.
User requests cancellation while the outcome is unknown.	Cancellation is treated as a separate, potentially uncertain action.

Procedure

1. Define the fake service’s operation-key and status-lookup contract.
2. Create a synthetic authorised operation with a fixed payload.
3. Inject one failure scenario at a time.
4. Record the attempt history and destination records.
5. Resume execution or perform reconciliation.
6. Compare the system’s reported status with the evidence.
7. Repeat with restart and concurrent-worker scenarios.

Measure

• Created requests per authorised operation.
• Correctness of reported status.
• Preservation of operation identity across restart.
• Unresolved-operation age.
• Attempts and elapsed recovery time.
• Manual intervention required.

Execution status: these are proposed tests. No code, recovery experiment or production benchmark was executed for this case. A successful local simulation would not establish a third-party service’s guarantees.

↑ Back to contents

13. Reusable AI prompts

Inspect a tool contract

Review this endpoint documentation.

Identify:
- Side effects
- Retry conditions
- Idempotency scope
- Retention
- Parameter matching
- Outcome-verification options

Mark undocumented behaviour as unknown.
Do not infer guarantees from the endpoint name.

Challenge a recovery design

Examine this workflow for crashes:
1. Before dispatch
2. After server commit
3. Before local confirmation

Include concurrent workers and delayed responses.
Identify where duplicate effects or incorrect status claims remain possible.
Distinguish local guarantees from destination-service guarantees.

Review employee messages

Compare each message with the available evidence.

Flag statements that imply confirmed success, failure or cancellation
without support.

Propose accurate wording.
Identify the operational owner needed to fulfil any follow-up promise.

Design fault scenarios

Propose a small failure-injection set for this submission workflow.

For each scenario, state:
- Failure point
- Observable evidence
- Permitted recovery action
- Property to verify
- Conditions requiring manual reconciliation

Do not perform live writes.

Prompts support analysis. They do not enforce service-side uniqueness, permissions or durable state.

↑ Back to contents

14. Transfer activities and student project

Freshers

Explain how “no response” differs from “nothing happened.” Write an accurate message for each recovery state.

Developers

Build a fake service that can commit an operation and deliberately lose its response. Compare recovery with and without a stable operation key.

Harness engineers

Demonstrate recovery after restart and concurrent resumption. Identify which layer owns retries and how other layers are configured.

Freelancers

Include endpoint guarantees, retry ownership, retention limits and reconciliation responsibilities in the integration scope and handover.

Managers

Decide who owns unresolved operations and what service commitment users receive. Compare the cost of delayed work with the consequences of duplicate actions.

Student project

Apply the case to appointment booking, notifications or support-ticket creation. Explain how duplication consequences differ across those settings.

Submit:

1. A tool contract with explicit unknowns.
2. An operation and attempt identity scheme.
3. A recovery-state table.
4. A failure-scenario set.
5. A user-message set tied to evidence.
6. A release recommendation with remaining limitations.

A paper simulation is acceptable. Clearly distinguish predicted results from actual execution results and disclose any AI assistance.

↑ Back to contents

15. Teaching notes and assessment

Assign the narrative and exhibits before revealing the worked analysis. Ask learners to state what the team knows before proposing a technical solution.

Suggested 90-minute session

1. Read the incident and classify evidence: 15 minutes.
2. Compare recovery options: 20 minutes.
3. Design states and ownership: 20 minutes.
4. Walk through failure scenarios: 25 minutes.
5. Submit a decision and user message: 10 minutes.

Criterion	Weight
Evidence and uncertainty reasoning	25%
Operation identity and service-contract understanding	25%
Recovery and concurrency design	20%
Proportionate operational decision	20%
Accurate user communication	10%

Misconceptions to challenge

• A timeout proves failure.
• A new attempt should always receive a new operation key.
• Searching before creating eliminates concurrency problems.
• A local log guarantees no duplicate external effects.
• Idempotency guarantees eventual completion.
• Human approval makes an uncertain retry technically safe.

Exit question: What would your system do if the action succeeded but every confirmation mechanism was temporarily unavailable?

↑ Back to contents

16. Research record and references

Primary sources consulted on 9 November 2026:

1. Malcolm Featonby, Amazon Builders’ Library. Making retries safe with idempotent APIs. Engineering account addressing ambiguous outcomes, caller identifiers, parameter matching and service-side consistency.
2. Stripe API documentation. Idempotent requests. Service-specific documentation; behaviour must be checked for the relevant API version and operation.
3. Anthropic, 26 November 2025. Effective harnesses for long-running agents. A coding-agent account concerning continuity, progress artifacts and testing.
4. AWS Well-Architected Framework. Control and limit retry calls. Guidance on retry limits, backoff, jitter and retry ownership.

Claim and evidence boundaries

Claim or element	Evidence status
A missing response can leave a creation outcome uncertain.	Supported by the AWS engineering discussion.
Idempotency behaviour depends on a specific service contract.	Illustrated by AWS and Stripe documentation.
Agent continuity benefits from persistent progress artifacts.	Reported in Anthropic’s coding-harness account.
Retry limits and ownership matter operationally.	Supported by AWS guidance.
Meridian’s incident, API and proposed state contract.	Author-created fictional teaching material.
The proposed design prevents duplicates in production.	Not established; no implementation or production test was performed.
The recommendation improves cost or business outcomes.	Not measured or asserted as an achieved result.

The sources support the technical distinctions and reported practices. They do not validate Meridian’s proposed architecture or establish its cost, incident frequency or return on investment.

The state contract, exercises and recommendation are original teaching designs. Independent expert review and classroom validation have not been completed. Recheck service documentation before using these ideas in a live integration.

↑ Back to contents

17. Author and contact

Ravindra Dastikop
AIYING — Accomplishing with AI
The Effective AI Engineer: A Case-Based Approach

Email: ravindra.dastikop@gmail.com
Phone: +91 84532 12131

Choose Work That Matters

Before repeating an action, establish what the system knows about the first attempt—and what the destination guarantees about another one.

← Read Case #019: The Developer Who Tested the Tests

↑ Back to contents

The AI Harness That Made the Model Less Important | The AI ThinkShift #012 |

PORABLES — FROM DEADLOCK TO POSSIBILITY

POrable #009: The Engineer Who Planned to Be Forgotten

By Ravindra Dastikop

Contents

• The story
• The PO behind the story
• Spot the lock
• Find the gateway
• Try the move
• Research note
• Further reading

The story

Every morning, Kavya explained the project again.

The AI coding agent had worked on it the previous afternoon. But after a fresh session began, it sometimes repeated completed work, overlooked an earlier decision or announced that an unfinished feature was ready.

Kavya wrote longer instructions.

Soon, the instructions needed instructions.

She worked on the agent’s harness: the software and working arrangements that supplied context, connected tools and organised its execution.

Her colleague suggested keeping the conversation going for longer.

Kavya looked at the growing history.

“What if the agent forgot me completely—and still knew exactly where to begin?”

She wrote the thought down.

Perhaps the project could introduce itself. Perhaps every stopping point could become an arrival point. Perhaps the next session could discover the state of the work without hearing the whole story.

She chose the second possibility.

Before improving the next instruction, she redesigned the handover.

The project received a short starting page linking to its requirements, decisions and current work. Each feature had an observable acceptance condition.

At the end of a work session, the agent was asked to record what changed, which checks had actually run, what remained uncertain and the next useful step. Changes were saved in version control.

The following morning, Kavya deliberately opened a fresh session.

“Continue the project.”

The agent read the starting page and inspected the latest changes.

The handover said the booking form was complete.

The acceptance check disagreed.

A booking appeared on screen, but disappeared after the page was refreshed.

Previously, Kavya might have discovered that during a demonstration.

This time, the restart routine required the agent to verify the existing application before adding another feature.

It reproduced the failure and resumed work on the booking form.

Later, another session encountered conflicting notes. It paused and asked Kavya which decision remained valid.

She corrected the record.

The handover system wasn’t infallible. It had made the uncertainty visible.

Over the next week, Kavya compared the revised harness with the earlier setup on the same tasks, using separate clean workspaces and repeated runs.

She tracked completed acceptance checks, repeated work, recovery effort, time and cost.

Some tasks benefited. Others showed little improvement. Maintaining the records also took effort.

She kept the parts that helped and simplified the rest.

At the next team meeting, someone asked whether she had finally made the agent remember everything.

“No,” Kavya said.

“I made it easier to find out what’s true.”

↑ Back to contents

The PO behind the story

“PO: The agent forgets me completely—and still knows where to begin.”

The provocation shifted Kavya’s attention from preserving every conversation to making the project’s current state discoverable and verifiable.

↑ Back to contents

Spot the lock

Kavya assumed that continuity depended on carrying more conversation forward.

What essential knowledge exists only in your chat history—or in your own memory?

↑ Back to contents

Find the gateway

She explored several possibilities before choosing structured handovers.

The practical move was to give a fresh session a reliable starting point, supported by evidence of the work already done.

The record guided the agent. Checks could challenge the record.

↑ Back to contents

Try the move

Choose a small project and prepare a handover containing:

• The intended outcome and acceptance conditions.
• The current version of the work.
• Decisions and unresolved questions.
• Checks performed and their results.
• The next useful step.

Start a fresh session and observe where it struggles. Compare its account with the actual project state.

Treat the handover as a working record that needs verification and maintenance.

↑ Back to contents

Research note

This fictional POrable draws on published harness-engineering practices. Here, “harness engineer” describes the work Kavya performs; job titles and the boundaries of that work vary.

Persistent project records, structured handovers, incremental work and executable checks are documented approaches. They do not guarantee reliable completion, and their costs and benefits depend on the task, model and implementation.

The characters, dialogue and outcomes are invented. The PO is an original modern adaptation, not a quotation from Edward de Bono.

↑ Back to contents

Further reading

• OpenAI: Harness engineering—designing environments, project knowledge and feedback loops
• Anthropic: Effective harnesses for long-running agents
• Anthropic: Harness design for long-running application development
• Anthropic: Demystifying evaluations for AI agents

↑ Back to contents

Explore more POrables

POrables — From Deadlock to Possibility

A story to enjoy. A shift to discover. A possibility to try.

By Ravindra Dastikop
AIYING — Accomplishing with AI