An agent creates a support ticket. The ticket service accepts the request, but the worker stops before saving the ticket number. The replacement worker finds an unfinished task and creates another ticket. The workspace survived. The business action happened twice.
A useful recovery design has to answer two questions: what progress can the agent recover, and what work is safe to repeat? Persistent storage addresses the first. The application and the receiving service must cooperate on the second.
TL;DR
- Save a stable operation ID and its intended input before making a state-changing tool call.
- A retry should reuse that ID; a genuinely new action should not.
- The receiving service must recognize repeats, not just the agent's checkpoint store.
- Test the gap between an accepted action and a saved result, not only a clean pause and resume.
Technical sources reviewed 15 October 2026. The examples below are application-design guidance, not a new Nirvana infrastructure benchmark.
Why can an agent recover its files but still repeat an action?
The agent and its tools may maintain separate records. An agent checkpoint can say “create ticket next” while the ticket service already holds the ticket. A timeout tells the caller it did not receive a usable response; it does not settle whether the receiver completed the operation. AWS discusses this ambiguity in its guide to safe retries.
Separate the records in your own design:
| Record | Question it answers | Question it does not answer alone |
|---|---|---|
| Workspace files | Which saved inputs and artifacts remain? | Did the external action complete? |
| Workflow checkpoint | Which graph state was saved? | Was the tool's response lost before this save? |
| Receiver's operation record | What happened for this operation ID? | Has the agent incorporated that result? |
The target is not “never retry.” It is “retries must not create a second business effect.” Some computation may repeat without causing harm. Creating a ticket, changing an account, or sending a notification needs a stronger contract.
What should be saved before the tool call?
Create an intent record before dispatch. Give each logical action a stable, non-sensitive identifier. Store its scope, validated inputs, state and eventual result reference. For example:
{
"operation_id": "op_8d2b4a",
"tenant_id": "demo-tenant",
"workflow_id": "workflow-104",
"action": "create_ticket",
"input": {"subject": "Review ingestion errors"},
"state": "prepared",
"result_id": null
}
This is an illustrative application record, not a Nirvana API schema. Generate the identifier once and persist it with the intended action. After a restart, load that record rather than generating a fresh identifier for the same work.
A payload hash is useful for detecting a changed request, but it is not enough to identify intent: two intentionally separate tickets can have identical text. AWS recommends caller-provided request identifiers and rejecting conflicting parameters under the same identifier. Keep the identifier scoped to the authenticated caller or tenant; it is not an authorization credential. Source
The practical review question is: can a replacement worker recover the same operation ID before it sends anything?
How does the receiving service make a retry safe?
For a service you control, the operation record and the business mutation should share an atomic transaction. This conceptual receiver creates a local ticket and records its result together:
BEGIN TRANSACTION
Look up (authenticated tenant, operation ID)
If already recorded:
Reject changed input; otherwise return the stored ticket ID
Else:
Create the ticket in this database
Save operation ID, input fingerprint and ticket ID
COMMIT
The concurrency control matters as much as the lookup. Two workers must not both see an absent key and create separate tickets. A unique operation key and an atomic write transaction can enforce this boundary. SQLite, for example, permits only one simultaneous writer. Choose concurrency controls that fit the production database rather than relying on an unprotected lookup. SQLite transaction documentation
Putting an HTTP call inside a local database transaction does not make the remote action atomic. If another provider performs the effect, use that provider's supported idempotency contract or a reliable reconciliation path. Otherwise, an uncertain outcome needs review rather than a blind replay.
Read the actual endpoint contract. For example, Stripe documents different retry behavior for API v1 and v2. Its v1 responses can be replayed from the first execution, and keys may be pruned after at least 24 hours; reusing a pruned key can create a new request. A generic “supports idempotency” label does not define your recovery window. Stripe documentation
What changes when the agent uses LangGraph?
LangGraph checkpointers persist graph state by thread. Its sync durability mode writes a checkpoint before the next step starts; async can continue while persistence is underway, and exit does not preserve intermediate progress for recovery from a mid-run process crash. Select that tradeoff deliberately. Checkpointers and durability modes
A synchronously saved checkpoint still cannot join an unrelated ticket service's transaction. Give the state-changing tool its stable operation ID and store the returned receipt. On recovery, an unresolved intent should lead to a safe same-ID retry or reconciliation, not a newly invented action.
Also separate recovery from deliberate historical replay. LangGraph documents that nodes after a selected historical checkpoint execute again, including API requests. A debugging replay therefore needs a read-only tool substitute, an isolated test receiver, or an explicit policy for reusing operation IDs. Replay behavior
Use a persistent checkpointer backend for restart recovery. An in-memory saver cannot survive the loss of its process. Persistence overview
How should this be tested on a persistent agent workspace?
Nirvana documents that a persistent sandbox retains its /workspace volume when paused. Resume starts a new pod that mounts the retained volume; the application reads its saved progress. Production deployments should use the documented ABS storage class rather than the development-only local-path choice. This preserves saved workspace state, not a running process's memory or an external service's transaction. Persistence and storage
Use a harmless test receiver and check the receiver's records, not only agent logs:
| Failure point | Expected recovery behavior |
|---|---|
| After intent is saved, before dispatch | The same intent is dispatched once successfully. |
| After receiver commit, before agent saves the receipt | A retry returns the existing result; there is one ticket. |
| Two replacement workers process the same intent | Both obtain the same result; one logical effect exists. |
| A retry changes the input under the same ID | It fails visibly instead of silently changing intent. |
Exercise all four cases with a harmless receiver, including a process exit after the receiver commits. Run them against your actual checkpointer, tool service and deployment before release. Passing a local simulation does not establish infrastructure crash durability, network behavior or production-scale concurrency.
The useful success condition is concrete: the replacement agent produces the correct result, each intended action has one recorded effect, and ambiguous outcomes remain visible.
FAQ
Can I solve this by checkpointing more often?
More frequent checkpoints can reduce repeated computation. They do not remove the interval between an external service accepting an action and the agent saving its receipt. That interval needs receiver-side deduplication or reconciliation.
What happens after an idempotency key expires?
Do not assume the old action is still protected. Check the provider's retention policy and reconcile the original result before issuing a new state-changing request. Stripe's API v1 documentation illustrates why key retention needs to be part of the design. Source
Should the operation record contain API credentials?
No. Keep the record about intent and outcome. Resolve credentials through your normal secret-management path, recheck authorization on resume, and avoid embedding sensitive values in identifiers or logs.
Build agents that resume useful work safely. Start with Nirvana's persistent-workspace documentation, then test the action boundary alongside the workspace boundary.
About Nirvana Labs
Nirvana Labs is a high-performance storage cloud purpose built for blockchain, AI and databases i.e. the most demanding, real-time, stateful workloads. Accelerated Block Storage (ABS) offers 20K baseline IOPS included, no over provisioning. Nirvana Kubernetes Service (NKS) with Karpenter auto-scaling, high clock-speed compute and private networking. Backed by Jump Trading, Crucible, etc with 50+ customers live in production today.
Learn more at Nirvana Labs
Nirvana Cloud | Pricing | Blog | Docs | Changelog | LinkedIn | Twitter | Telegram | YouTube
Related Posts

How to Share Persistent State Across Multiple AI Agents
Separate workspaces, checkpoints and shared records so multiple agents can coordinate safely, recover unfinished work and avoid overwriting each other.

How to Manage Cloud Infrastructure With Your AI Assistant Through MCP
Connect your assistant to the Nirvana API through MCP, start with read-only discovery, and verify infrastructure changes with a practical acceptance checklist.

How to Drain Kubernetes Nodes Running Databases Safely
Use disruption budgets, replica health, placement checks and application tests to make Kubernetes database maintenance a controlled operation.

