Why Agent Evals Pass in CI and Fail in Production

# Why Agent Evals Pass in CI and Fail in Production An eval suite for a tool-using agent went green for three straight weeks. Then it started failing in production twice a day, on requests that looked nothing like edge cases. The eval suite hadn't regressed. It had just never been testing the thing that was actually breaking. ## The gap is in what the eval fixes and what production varies Most agent evals fix everything except the model's output: same tool responses, same retrieved context, same conversation history, replayed from a recorded fixture. That's reasonable for catching prompt regressions — did this change make the model phrase things worse, pick the wrong tool, or drop a step it used to take. It is not reasonable for catching the failure modes that only show up when the *environment* around the model is live: a search API that returns results in a different order today, a database that's slower under real concurrent load and times out mid-tool-call, a upstream service that changed its error-response shape without a version bump. The agent in question had a retrieval step that, in the fixture, always returned results in the same three-item order. In production, the underlying index reranked based on live traffic signals, and item order shifted day to day. The prompt template referenced "the first result" in a way that assumed stability. The eval never had a chance to catch it, because the eval's retrieval step was frozen in amber. ## Replaying inputs isn't the same as replaying conditions The instinct to fix this is usually "record more fixtures" — capture the actual production traffic and replay it. That helps, but it's still replaying *inputs*, not *conditions*. A replayed fixture from a slow day doesn't reproduce the timeout that only happens under concurrent load. A replayed fixture from a normal day doesn't reproduce the malformed response from a dependency's brief outage last Tuesday. The fixture is a snapshot; production is a distribution. What actually closed the gap was splitting the eval suite into two tiers with different jobs. The fixture-based tier stayed as-is, running on every PR, catching prompt and reasoning regressions fast and cheap. A second tier ran against the real staging environment on a schedule — same tools, same live (non-prod) data sources, same network conditions — and was allowed to be flaky, because flakiness in that tier is signal, not noise. A test that fails intermittently against a live dependency is telling you something a deterministic fixture never could. ## The metric that mattered wasn't pass rate The team's first move was to chase 100% pass rate on the fixture suite, which is the wrong target once you understand what the suite can and can't see. A fixture suite at 100% tells you the agent hasn't regressed against known scenarios. It says nothing about scenarios the fixture never encoded. The more useful number turned out to be the gap between fixture-suite pass rate and live-tier pass rate — when that gap widened, it meant production conditions had drifted further from what the fixtures assumed, which was itself worth investigating independent of any single failure. Evals that only replay recorded inputs will always plateau at "no regressions I already knew to check for." The failures that actually reach users are the ones nobody encoded into a fixture, because by definition nobody knew to write that test yet. The fix isn't a better fixture. It's admitting the fixture is testing a narrower thing than "does this agent work," and building a second check for the part it can't reach.