Tool-Call Retries That Quietly Duplicate Side Effects

# Tool-Call Retries That Quietly Duplicate Side Effects A customer got the same confirmation email four times for one order. Nothing was wrong with the email service, the order system, or the agent's reasoning. The retry logic that was supposed to make the pipeline more reliable was the thing generating the duplicates — and it took longer to find than it should have, because every individual retry looked like exactly the kind of resilience you want. ## Retries assume the failure was in transit, not in effect The standard retry pattern — call a tool, get an error or a timeout, call it again — is built on an assumption that's true for read operations and false for a lot of write operations: that a failed call means nothing happened. A timeout doesn't tell you the call failed. It tells you the response didn't arrive within the wait window. The send-email tool call in question routinely completed on the far end in nine seconds, past an eight-second client timeout, which meant the agent saw a timeout, assumed failure, and retried — sending a second email for a request the first email had already fulfilled. Under enough backpressure, this happened two or three more times before the agent moved on. Nobody had described this tool as unsafe to retry, because retry-safety isn't usually a property anyone thinks to document. The tool didn't fail in a way that made the problem obvious in a log — every individual call returned a normal success or timeout, and every retry looked like the system doing its job. ## Idempotency has to live at the tool boundary, not the agent's judgment The instinct is to fix this in the agent's reasoning — tell it to be more careful about retrying, or to check whether an action already succeeded before trying again. That's fragile, because it asks a probabilistic reasoning step to enforce a guarantee that should be structural. The actual fix was to make the tool itself idempotent: every write-side tool call now takes a client-generated idempotency key derived from the agent's call parameters, and the underlying service deduplicates on that key server-side. A retried call with the same key is a no-op against an action that already completed, regardless of what the agent believed happened. This moves the safety property to the one place that can actually enforce it. The agent can retry as aggressively as it wants — timeouts, transient errors, even a full agent restart mid-task — and the side effect only happens once, because the service holding the side effect is the one doing the deduplication, not the caller deciding whether to call again. ## The audit question that actually finds this Looking at agent transcripts for this class of bug doesn't work well, because a transcript shows individual tool calls succeeding or timing out, not the downstream effect count. The check that actually surfaces it is auditing the *external system's* record of side effects against the *agent's* count of intended actions — how many emails did the mail provider actually send for this conversation, versus how many times the agent's plan called for one email. A mismatch there is a duplicate-effect bug regardless of what the agent's own logs claim happened. Any tool an agent can call more than once needs an answer to one question before it ships: what happens if this gets called twice with the same intent. If the answer is "it does the thing twice," that's not a hypothetical — it's a matter of when the timeout window and the true response time cross, which for anything running under real network conditions is a when, not an if.