The chokepoint fix: why one retry wrapper beats twenty try/catch blocks

# The chokepoint fix: why one retry wrapper beats twenty try/catch blocks Every production system that calls an LLM API eventually hits a 429 (rate-limited) or 529 (overloaded) response. The question isn't whether it happens — it's what your system does the moment it does. ## The failure mode In one production pipeline we reviewed, the LLM call had no retry logic anywhere in the codebase. A single transient "Overloaded" response from the API — not a bug, not bad input, just the provider momentarily out of capacity — was enough to kill an entire batch run. Four consecutive full-run failures happened before anyone traced the cause: not a logic bug, not a data problem, just an unhandled transient error propagating straight up through the call stack and taking the whole job down with it. The system had multiple call sites — separate code paths for parsing, qualifying, and batch-processing across several integrations. Each one called the LLM independently. None of them retried anything. ## Why "just add try/catch everywhere" is the wrong fix The instinctive fix is to wrap each call site in its own try/catch and retry logic. That's the wrong level to fix it at. Scattering retry logic across every caller means: - Inconsistent behavior — one call site retries 3 times, another retries 5, a third forgets to retry at all - Retrying errors that can't be fixed by retrying — a malformed request or bad credential will fail identically on attempt two as it did on attempt one, so blindly retrying everything just burns time reproducing the same deterministic failure - No single place to reason about backoff strategy when it needs to change The fix belongs at the chokepoint every call already passes through, not at each call site. ## What the fix actually looks like 1. **Classify errors as retryable or not, at the source.** Only 429 (rate-limited) and 529 (overloaded) are the API's own signal to back off and retry. Everything else — bad auth, bad model name, malformed request, a CLI spawn failure — is retried never, because retrying a deterministic failure just wastes time reproducing it. 2. **Fix it once, at the one function every caller already funnels through**, not in each caller. If your LLM integration has a single low-level "call the model" function underneath every higher-level use, that's the only place this needs to live. 3. **Exponential backoff with jitter, capped attempts.** A fixed short delay just hammers an already-overloaded API again immediately; unbounded retries can hang a pipeline indefinitely. Growing delay + a few hundred milliseconds of randomness (so parallel callers don't all retry in lockstep) + a hard attempt ceiling is the standard shape, and it's a handful of lines. ## Did it actually work Verified against a real re-run, not a synthetic test: the same pipeline that had failed four consecutive times recovered from **6 real 529 responses across 23 batches** and completed the full run cleanly — 275 jobs processed, zero manual intervention. ## The broader lesson An LLM API is a network dependency like any other, and it fails the way network dependencies fail — transiently, under load, in ways that have nothing to do with your code being wrong. If you wouldn't ship a database client with no retry logic, don't ship an LLM integration with none either. The fix is small. The failure mode it prevents — an entire batch job dying because a provider was briefly at capacity — is not.