Designing Idempotent Tool-Calling Loops for Production LLM Agents

At 2:13 a.m., a customer support agent powered by an LLM receives a frustrated message: “I was charged twice and got three confirmation emails.” The incident channel fills up quickly. Finance sees duplicate payment captures. Support sees the same refund request reopened twice. The CRM shows conflicting notes because two tool calls wrote overlapping summaries back to the customer record. The engineering team checks traces and finds something depressingly familiar: the agent loop called a payment tool, timed out waiting for the response, reasoned that the action may not have happened, retried the tool, then proceeded to send a confirmation email on both the first and second path. A human reviewing the logs can tell what happened. The system, at runtime, could not.

This is the difference between a demo agent and a production agent.

In demos, tool calls are usually treated like function calls inside a single process: call the thing, get the result, continue. In production, tool calls are distributed systems operations. They cross process boundaries, time out, partially fail, get replayed, race with each other, and interact with external systems that may or may not support clean retry semantics. The model is not the only source of nondeterminism. Networks, databases, queues, vendors, browser sessions, and human-in-the-loop steps all introduce ambiguity.

If your agent can call tools repeatedly—and nearly every useful agent can—then idempotency is not a nice-to-have. It is the control surface that keeps retries from becoming duplicate charges, duplicate emails, duplicate tickets, and inconsistent writes across systems.

The core production lesson is simple: never let the model be the sole authority on whether an action should run again. The orchestration layer needs first-class concepts for side effects, operation identity, workflow state, duplicate suppression, and recovery. The model can decide intent. The platform must decide execution.

This article is a practical guide to designing idempotent tool-calling loops for production LLM agents. I’ll cover the recurring failure pattern, why the naive approach fails, what better architecture looks like, how to classify side effects, how to structure retries, how to design workflow state, what to log, what to evaluate, and where the real tradeoffs are.

The recurring failure pattern

The production incidents tend to look different on the surface, but they share a common shape.

A user asks the agent to do something that involves one or more external writes:

charge a card
send an email
create or update a ticket
schedule a meeting
issue a refund
update a CRM record
modify an order
trigger a deployment

The agent reasons over the request, chooses tools, and enters a loop like this:

inspect state
call tool
read tool result
reason about next step
possibly retry or call another tool

The failure starts when a tool invocation ends in an ambiguous state:

client timeout before response arrives
connection drop after the vendor received the request
orchestrator restart after dispatch but before persisting the result
model token limit truncates the tool result
queue retry replays the job
user resubmits the same request
upstream API returns 500 but actually completed the write
concurrency causes two agent runs to act on the same entity

Then the loop asks the wrong question: “Should I try again?”

That sounds reasonable, but for side-effecting operations it is incomplete. The right question is: “Do I already have an operation with this intent in flight or completed, and what is the safe recovery path for this exact operation identity?”

Without that distinction, retries become re-executions.

I’ve seen this happen in several common forms:

Pattern 1: Timeout-induced duplicate writes

The agent calls charge_customer(order_id=123, amount=49.00). The payment provider completes the charge but the network times out before the response is returned. The orchestrator reports a timeout to the model. The model says “try again.” Now you have two charges unless the payment call is protected by a stable idempotency key.

Pattern 2: Orchestrator replay after crash

The tool call was sent successfully, but the process crashed before marking the step complete. On recovery, the workflow replays from the last durable checkpoint and emits the same tool call again.

Pattern 3: Duplicate user intent across channels

A user asks in chat, then clicks a UI button, then replies to an email. Three requests express the same intent. If your system lacks request correlation and dedupe windows, three independent agent runs may send three emails or create three tickets.

Pattern 4: Multi-step inconsistency

The agent updates a CRM, then sends an email, then creates an internal ticket. The CRM write succeeds, the email succeeds, and the ticket creation fails permanently. A naive retry of the whole plan will duplicate the first two steps. A naive retry of only the failed step may leave the workflow in a state that the model can’t reliably infer.

Pattern 5: Competing agent runs

Two separate workflows operate on the same business entity. A support agent run issues a refund while a retention agent run simultaneously offers account credit. Both are locally “correct.” Together they violate policy and create accounting headaches.

All of these are idempotency problems, not just prompt problems.

Why the naive approach fails

The naive design for agent tool use usually has four assumptions baked in:

tool calls are cheap to repeat
the latest model context is enough to infer execution state
retries are a transport concern, not a business concern
successful API responses are the main source of truth

Each assumption fails in production.

Tool calls are not all equivalent

Some tools are read-only. Some are write operations but naturally idempotent. Some are side-effecting and must never be replayed without an operation key. Some have irreversible effects. Yet many agent frameworks expose all tools through the same generic interface, making “call tool again” look harmless.

A weather lookup and a payment capture should not share the same retry logic.

Model context is an unreliable execution ledger

Teams often stuff prior tool outputs into the conversation and hope the model can infer what already happened. This works until it doesn’t.

Context windows truncate. Summaries compress away critical identifiers. Tool outputs are phrased inconsistently. The model may fail to bind “the refund attempt from two turns ago” to the exact external operation ID that matters. Worse, the model may hallucinate certainty about whether a step succeeded.

Execution state must live in a deterministic workflow store, not only in the prompt.

Retries can create new business events

Infrastructure engineers know retries improve reliability. Product and finance teams know retries can multiply business actions. Both are right.

A retry of an HTTP GET is usually fine. A retry of “send money,” “email customer,” or “create order” is not just transport recovery; it is a potential duplicate event. The retry policy must be coupled to the business semantics of the operation.

Success is often ambiguous

In distributed systems, there is a notorious gap between “request sent” and “response received.” If you only record success on a clean 200 response, you have no principled way to recover from partial failures. You need to represent uncertain outcomes explicitly: dispatched, acknowledged, completed, unknown, compensating, failed-permanent.

That ambiguity is the normal case for robust systems, not an edge case.

A better approach: separate agent reasoning from operation execution

The most robust production pattern is to split the system into two layers:

Reasoning layer: the LLM plans and selects actions
Execution layer: a deterministic orchestrator validates, deduplicates, dispatches, records, retries, and reconciles actions

The model proposes. The workflow engine commits.

The architecture in practice

A production-safe tool-calling loop typically looks like this:

User request enters an agent workflow with a workflow ID and correlation metadata.
The model proposes a tool action with structured arguments and an intent label.
The orchestrator canonicalizes the action into an operation envelope.
Policy checks run: permissions, side-effect class, concurrency guardrails, rate limits, approval requirements.
An idempotency key is computed or retrieved based on the operation identity.
The orchestrator checks the operation store for an existing matching operation.
If found, it returns the prior result or current status instead of executing again.
If not found, it persists a pending operation record durably before dispatch.
The tool adapter executes the call to the external system with vendor-level idempotency fields where supported.
The result is reconciled into the operation record with external IDs, status, timestamps, and artifacts.
The model receives a normalized tool result that references the durable operation state, not just raw API text.
On timeout or failure, recovery logic runs from the operation state, not from model guesswork.

That split is the single biggest improvement most teams can make.

Operation envelope: the unit of execution

Every proposed tool call should be turned into a structured operation envelope, something like:

json
{
  "operation_id": "op_01J...",
  "workflow_id": "wf_01J...",
  "run_id": "run_01J...",
  "tool_name": "capture_payment",
  "intent": "collect_payment_for_order",
  "entity_type": "order",
  "entity_id": "order_123",
  "arguments_canonical": {
    "customer_id": "cust_456",
    "order_id": "order_123",
    "amount_cents": 4900,
    "currency": "USD"
  },
  "side_effect_class": "external_irreversible",
  "idempotency_key": "payment:order_123:capture:4900:USD:v1",
  "status": "pending_dispatch",
  "created_at": "...",
  "attempt_count": 0
}

The exact schema varies, but the principle is consistent: the system should identify and track the business operation independently of any single model turn.

Side-effect classification: not all tools deserve the same loop

One of the fastest ways to improve safety is to classify tools by side-effect semantics and tie execution policy to that class.

A practical taxonomy is:

1. Pure reads

Examples:

search knowledge base
fetch account status
query inventory
retrieve weather

Policy:

retries generally safe
no idempotency key required, though request tracing still helps
can often be executed inline with aggressive timeouts
caching may reduce cost and latency

2. Internal deterministic writes

Examples:

write workflow note to your own database
set internal status field
append audit event

Policy:

use database transaction semantics
enforce unique constraints where possible
still assign operation IDs for replay safety
easier to make exactly-once within your own boundary

3. External idempotent writes

Examples:

create payment intent with provider idempotency support
create support ticket where your API accepts a client token
update CRM object by primary key with version check

Policy:

always pass stable idempotency keys to the external API
persist the mapping of your operation ID to external request ID
retries allowed through orchestrator only

4. External non-idempotent or weakly idempotent writes

Examples:

send email through a provider without strong dedupe support
trigger webhook to a partner system
submit browser automation step against a fragile UI
place phone call

Policy:

add your own duplicate suppression layer
prefer outbox pattern and delivery records
require stronger confirmation before retry
consider approval or reconciliation before replay

5. Irreversible or regulated actions

Examples:

capture funds
issue refund
change subscription billing state
execute trade
send compliance-sensitive communication

Policy:

explicit policy gates
durable checkpoints before dispatch
strict idempotency keying
human approval or dual control where needed
post-action reconciliation jobs
richer audit logs and alerts

This classification should be part of the tool registry, not hidden in documentation. The orchestrator should know, for each tool, what retry policy, timeout, concurrency rule, and approval path apply.

Idempotency keys: design them as business identities, not transport IDs

The phrase “use idempotency keys” is common advice, but the implementation details matter.

A bad idempotency key is a random UUID generated at retry time. That only makes each request unique; it does not unify equivalent attempts.

A good idempotency key is stable across retries for the same intended business operation and different across meaningfully different operations.

What should an idempotency key represent?

It should represent the identity of the intended effect.

For example:

payment:order_123:capture:4900:USD
refund:charge_789:full
email:acct_456:template_password_reset:2026-07-07
ticket:conversation_123:create_billing_escalation

Often you also need a version suffix to handle policy or parameter changes:

payment:order_123:capture:4900:USD:v2

That prevents accidental dedupe across semantically different workflows after your business logic changes.

Canonicalize arguments before hashing

If you derive keys from arguments, canonicalization matters.

Normalize:

field ordering
units, especially money and timestamps
casing
optional/default fields
whitespace
enum names

Then either store the canonical tuple directly or hash it. If you hash, keep the unhashed canonical components available in logs for debugging.

Scope keys correctly

A common mistake is scoping too narrowly or too broadly.

Too narrow:

key based on workflow run ID
retries from another run bypass dedupe

Too broad:

key based only on customer ID
blocks valid repeated actions over time

Good scopes often include:

business entity ID
action type
semantic parameters
sometimes a time window or version

Distinguish dedupe keys from trace IDs

You still need request IDs, trace IDs, and attempt IDs.

trace ID: follows a request across services
attempt ID: uniquely identifies one dispatch try
operation ID: internal durable identifier for the logical action
idempotency key: stable business identity for duplicate suppression

Teams often conflate these and lose the ability to reason about retries.

Retry semantics: retries must be state-aware

The orchestrator should not treat all failures the same. A good retry design starts with explicit operation states.

A useful state model:

planned
pending_dispatch
dispatched
acknowledged
completed_success
completed_noop
failed_retryable
failed_permanent
unknown_outcome
compensating
compensated

Why so many states? Because production recovery depends on knowing the difference between “we never sent it,” “we sent it but don’t know what happened,” and “we know it happened.”

Retry policy by failure mode

Before dispatch persisted?

safest is to avoid this state entirely; persist first

Failed before send

safe to retry with same operation identity

Timed out after send

do not blindly resend
check external system by idempotency key or lookup API
move to unknown_outcome and schedule reconciliation

Explicit rate limit / 429

retry with backoff, same idempotency key

5xx from provider

depends on provider contract; some 5xx responses may still have committed side effects
use reconciliation endpoint where available

Validation failure / 4xx semantic error

permanent failure unless arguments change under a new operation version

Process crash after dispatch

recover from durable operation record, not from reissuing the request immediately

Backoff strategy

Use exponential backoff with jitter for transport or capacity failures. But the more important point is bounded retries plus reconciliation. Infinite retry loops on ambiguous writes are how systems produce disasters slowly.

In many systems, the right pattern is:

limited immediate retries for clearly safe cases
transition ambiguous cases to reconciliation queue
notify the workflow and possibly the user of pending status
resume once external state is confirmed

This is particularly important for payments, fulfillment, and messaging.

Duplicate suppression beyond idempotency keys

Idempotency keys are necessary, but not sufficient.

You also need duplicate suppression at the workflow and event levels.

Sources of duplicates

user sends the same request multiple times
upstream event bus delivers at least once
webhook retries from vendors
job queue replays after timeout
multi-tab UI actions
multiple agents responding to same trigger

Practical suppression patterns

1. Inbox/outbox tables

For inbound events, persist an inbox record keyed by source event ID and ignore duplicates.

For outbound side effects, use an outbox table that records the exact intended message or action before delivery. Workers deliver from the outbox and mark delivery status durably.

This pattern is boring and extremely effective.

2. Semantic dedupe windows

For actions like “send follow-up email,” exact argument matching may not be enough. You may need a business rule such as:

do not send the same template to the same user more than once in 24 hours unless explicitly overridden

That is not pure idempotency; it is policy-level duplicate suppression.

3. Per-entity concurrency locks

If only one active workflow should mutate an order or account at a time, use a lease/lock keyed by entity ID. Keep lock ownership in durable workflow state and expire leases carefully.

This prevents competing runs from racing each other into inconsistent writes.

4. State machine guards

If order state is already refunded, do not allow another refund operation to plan as executable. A lot of duplicate actions can be blocked by checking current entity state before dispatch.

Compensating actions: what to do when exactly-once is impossible

Exactly-once delivery across arbitrary external systems is not a realistic guarantee. What you can build is:

at-most-once for some actions
effectively-once through idempotency and reconciliation for many actions
compensating workflows when duplicates or partial completion happen anyway

Compensating actions are not just “undo.” Many side effects are not fully reversible.

Examples:

duplicate charge -> issue refund, add internal note, notify finance
duplicate email -> suppress future follow-ups, annotate incident, possibly send apology
CRM update conflict -> merge or restore fields from audit log
deployment triggered twice -> mark one run superseded and reconcile environment state

A good compensating design includes:

a mapping from action type to possible compensations
preconditions for running compensation
a separate operation identity for compensation itself
auditability linking original and compensating operations
policy for human escalation when compensation is risky

Do not make the model invent compensations on the fly for high-risk systems. Let the model choose among approved compensation playbooks exposed as tools or policies.

Workflow state design: keep the ledger outside the prompt

If there is one architectural takeaway to emphasize, it is this: your workflow state is the ledger of truth for side effects.

The model context should be a projection of that ledger, not the ledger itself.

What to persist

At minimum, persist:

workflow ID and run ID
user request and normalized intent
operation records for each proposed tool call
idempotency key
canonicalized arguments
side-effect class
operation state transitions with timestamps
external request IDs and external object IDs
attempt records and error categories
policy decisions and approvals
compensation links
final user-visible outcome

Why event history beats only final state

You want both current state and event history.

Current state lets the orchestrator resume quickly. Event history lets you debug incidents and compute metrics like duplicate suppression rate, unknown outcome rate, and average reconciliation delay.

If you can afford it, store state transitions as append-only events plus a materialized current-state view.

Resume behavior

On restart, the workflow should:

load current workflow and open operations
for any dispatched or unknown_outcome operations, reconcile before re-dispatch
for failed_retryable, apply policy-based retry
for completed_*, return prior results to the model if needed

The key point is that resumption logic must be deterministic and not require the model to “remember” what happened.

Tool adapter design: normalize provider weirdness

Provider APIs vary wildly in their support for idempotency, lookup, and error semantics. Your tool adapter layer should absorb that variability.

For each external system, define:

whether vendor-side idempotency keys are supported
max idempotency retention window
whether the provider offers lookup by client reference or request ID
whether 5xx can still indicate committed writes
retryable error codes
rate limit behavior
consistency model and propagation delay

Then expose a normalized contract upward to the orchestrator.

Example normalized tool result

json
{
  "operation_id": "op_01J...",
  "tool_name": "capture_payment",
  "dispatch_status": "unknown_outcome",
  "provider": "stripe",
  "provider_request_id": "req_abc",
  "provider_object_id": null,
  "retry_recommendation": "reconcile_before_retry",
  "raw_status_code": 504,
  "safe_user_message": "Payment status is being confirmed."
}

The model does not need the entire provider-specific mess. It needs a reliable summary of what it can say and what the system will do next.

Observability: if you can’t see duplicate pressure, you will ship incidents

Idempotent design is not complete without observability. Most teams log prompts and tool outputs, but not the execution semantics that matter.

You need dashboards and alerts for the operation layer.

Metrics to track

At minimum:

operations by tool and side-effect class
duplicate suppression count and rate
retries by tool and failure category
unknown outcome rate
reconciliation success rate and lag
compensation rate
time spent in each operation state
concurrent mutation conflicts by entity type
vendor idempotency hit rate
user-visible duplicate incidents

Structured logs

Every operation attempt should log:

workflow ID
operation ID
idempotency key
attempt ID
tool name
side-effect class
canonical argument fingerprint
provider request ID
state transition
latency
error category
dedupe decision reason

This is what lets you answer, quickly, “Did we send it twice, or only think we might have?”

Tracing

Distributed traces should connect:

user request
model planning span
policy evaluation span
operation store lookup/write
tool adapter dispatch
provider response or timeout
reconciliation job
compensation job

In incidents, these traces are more valuable than raw prompt logs.

Alerting

Good alerts include:

spike in unknown_outcome for a high-risk tool
duplicate suppression drops unexpectedly to zero
compensation rate rises above baseline
a provider’s idempotency lookup starts failing
many operations stuck in dispatched longer than SLA

Evaluation strategy: test failure semantics, not just answer quality

This is where many LLM teams underinvest. They evaluate whether the model chose the right tool or produced the right text, but not whether the system behaves safely under retries and ambiguity.

For tool-calling agents, evals need to cover execution semantics.

Eval dimensions that matter

1. Intent-to-operation correctness

Given a request, does the system produce the right operation identity and idempotency key?

Examples:

same request phrased differently should map to same logical operation when appropriate
materially different requests should not collide on the same key

2. Retry safety

Inject timeouts, connection resets, 429s, and ambiguous 5xx responses. Verify:

no duplicate external write
state transitions are correct
retries use same idempotency key
ambiguous outcomes go to reconciliation

3. Replay safety

Crash the orchestrator after dispatch but before commit. Replay the workflow. Verify no duplicate action occurs.

4. Concurrency safety

Run two workflows against the same entity with overlapping actions. Verify locks, guards, or state machine rules prevent bad interleavings.

5. Compensation correctness

Simulate partial failure and verify approved compensation flow runs, with linked audit trail.

6. User-facing behavior under uncertainty

Verify the assistant says “I’m confirming payment status” rather than falsely claiming success or failure.

Build a fault-injection harness

Do not wait for real vendor outages to test this.

A useful harness can simulate for each tool:

success
timeout after commit
timeout before commit
duplicate webhook delivery
429 with retry-after
500 with eventual consistency
malformed response
long propagation delay

Then run end-to-end scenarios and assert on operation counts and final external state.

Metrics for eval success

Beyond accuracy, track:

duplicate side effects per 10k runs
mean reconciliation time
percent of ambiguous outcomes resolved without human intervention
false dedupe rate
lock contention rate
extra latency added by safety controls
cost overhead from reconciliation and logging

These are the real production KPIs.

Model and tool tradeoffs

Different model and tool designs interact with idempotency in important ways.

Larger models vs smaller models

Larger models may do better at choosing when an action is needed, but they do not eliminate distributed systems ambiguity. A smarter model is not a substitute for an operation ledger.

Smaller models with stricter orchestration often outperform larger models in reliability-sensitive workflows because the safety comes from deterministic execution semantics.

My default guidance:

use the model for intent classification, slot filling, policy explanation, and conditional planning
use deterministic code for operation identity, dedupe, retries, and recovery

Free-form tool calling vs constrained schemas

Free-form tool use is flexible but dangerous for side-effecting actions.

Prefer:

typed tool schemas
canonical argument normalization
policy metadata attached to each tool
explicit “dry-run” or “plan-only” mode for risky actions

Constrained schemas make idempotency easier because you can reliably derive operation identity from normalized fields.

Direct agent-to-tool calls vs queue-backed execution

Direct inline calls minimize latency but increase coupling and reduce replay safety.

Queue-backed or workflow-engine-backed execution adds some latency, but gives you:

durable checkpoints
retries outside the model loop
better backpressure handling
replay/recovery control
clearer observability

For read-heavy low-risk tools, inline is often fine. For writes to external systems, I strongly prefer durable execution.

Cost and latency tradeoffs

All of this safety infrastructure costs something.

You are adding:

operation store reads/writes
canonicalization logic
queue or workflow engine overhead
reconciliation jobs
richer logging and tracing
lock management
policy checks

That increases p50 and p99 latency, and definitely increases engineering complexity.

But the comparison should not be “safe system vs simpler system.” It should be “safe system vs incident cost.”

A few concrete tradeoffs:

Inline execution

Pros:

lower latency
simpler stack

Cons:

weaker replay safety
ambiguous outcomes handled poorly
harder to recover from process death

Good for:

pure reads
low-risk internal operations

Durable orchestration

Pros:

strong recovery semantics
easier reconciliation
better observability

Cons:

higher latency
more infrastructure

Good for:

payments
messaging
ticketing
order modifications
any regulated or customer-visible side effect

Rich reconciliation

Pros:

prevents duplicates after ambiguous failures
improves correctness with flaky vendors

Cons:

extra API calls
slower final confirmation

In practice, users tolerate “I’m confirming status” much better than duplicate charges.

Implementation blueprint

If you are building this from scratch, here is a pragmatic sequence.

Phase 1: classify tools and add operation records

create a tool registry with side-effect class
wrap all side-effecting tools with an operation envelope
persist operation records before dispatch
add stable operation IDs and attempt IDs

Phase 2: introduce idempotency keys and dedupe store

define canonicalization per tool
generate business-level idempotency keys
enforce uniqueness in operation store where appropriate
return existing results on dedupe hits

Phase 3: add explicit state machine and retries

model operation states explicitly
separate retryable, permanent, and unknown outcomes
implement bounded retries with jitter
add reconciliation jobs for ambiguous cases

Phase 4: add concurrency controls and compensation

per-entity leases or optimistic version checks
policy guardrails for irreversible tools
compensation playbooks for common failure modes

Phase 5: observability and eval harness

metrics, traces, dashboards, alerts
failure injection in staging and CI
scenario-based regression suite

Example pseudocode for execution wrapper

python
def execute_tool_operation(workflow_id, tool_name, raw_args, intent):
    tool_meta = tool_registry[tool_name]
    args = canonicalize(tool_name, raw_args)
    entity_ref = derive_entity_ref(tool_name, args)
    idem_key = derive_idempotency_key(tool_name, intent, args, entity_ref)

    existing = operation_store.find_by_idempotency_key(idem_key)
    if existing and existing.status in TERMINAL_OR_REUSABLE_STATES:
        return existing.normalized_result

    op = existing or operation_store.create_if_absent(
        workflow_id=workflow_id,
        tool_name=tool_name,
        intent=intent,
        entity_ref=entity_ref,
        canonical_args=args,
        side_effect_class=tool_meta.side_effect_class,
        idempotency_key=idem_key,
        status="pending_dispatch",
    )

    if tool_meta.requires_entity_lock:
        acquire_lock(entity_ref, workflow_id)

    try:
        operation_store.transition(op.id, "dispatched")
        attempt_id = operation_store.new_attempt(op.id)

        response = tool_adapter.dispatch(
            tool_name=tool_name,
            args=args,
            idempotency_key=idem_key,
            attempt_id=attempt_id,
        )

        normalized = normalize_response(tool_name, response)
        operation_store.record_result(op.id, normalized)

        if normalized.status == "completed_success":
            operation_store.transition(op.id, "completed_success")
        elif normalized.status == "completed_noop":
            operation_store.transition(op.id, "completed_noop")
        elif normalized.status == "unknown_outcome":
            operation_store.transition(op.id, "unknown_outcome")
            enqueue_reconciliation(op.id)
        elif normalized.status == "failed_retryable":
            operation_store.transition(op.id, "failed_retryable")
            enqueue_retry(op.id)
        else:
            operation_store.transition(op.id, "failed_permanent")

        return operation_store.get_normalized_result(op.id)

    finally:
        if tool_meta.requires_entity_lock:
            release_lock(entity_ref, workflow_id)

The code is simplified, but the important part is the ordering:

canonicalize
dedupe lookup
durable record creation
dispatch
normalized result recording
recovery based on explicit state

Hard-earned takeaways

A few battle-tested rules are worth stating plainly.

1. Never let the model decide duplicate suppression by itself

The model should not infer “probably safe to retry” for irreversible actions. Put duplicate suppression in the execution layer.

2. Every side-effecting tool call needs a stable business identity

If you cannot define what makes two calls “the same operation,” you are not ready to automate that action.

3. Unknown outcome is a first-class state

Do not collapse ambiguity into failure. Ambiguous writes require reconciliation.

4. Persist before dispatch

If you send to an external system before you durably record the operation, recovery becomes guesswork.

5. Use provider idempotency support, but never rely on it blindly

Vendors differ in retention windows and semantics. Keep your own operation ledger.

6. Exactly-once is not the goal; controlled side effects are

You will not achieve perfect global exactly-once execution across all tools. You can achieve systems that are safe, auditable, and recoverable.

7. Evaluate failure handling as a product feature

If your eval suite only measures answer quality, it will miss the production failures that matter most.

8. Pay the latency tax where the business risk demands it

Fast duplicate charges are worse than slow confirmed charges. Users remember the former.

Production LLM agents do not fail mainly because the model picked the wrong verb. They fail because a distributed operation with side effects was treated like a local function call. Once you reframe tool execution as a workflow problem—with operation identities, idempotency keys, state machines, dedupe, reconciliation, and observability—the design becomes much more predictable.

That predictability is what separates an interesting agent from an operationally trustworthy one.

If your team is deploying agents that can write to payments, messaging systems, CRMs, ticketing platforms, or any other external system customers care about, build the idempotency layer before you need it. Doing it after the first duplicate-charge incident is always more expensive.