A Feature Flag That Skipped a Channel Instead of Failing Red

# A Feature Flag That Skipped a Channel Instead of Failing Red A multi-channel automation pipeline hit a wall on one of its channels: the platform on the other end had a hard capacity ceiling, and the pipeline was already at it. The obvious failure mode for a system like this is loud and unhelpful — retry forever, throw on every run, page someone at 3am about a rate limit that isn't going to lift itself. That's not what happened, and the reason it didn't is worth being deliberate about. ## Designing for "not yet" instead of "broken" The channel had a flag governing whether it participated in each run, defaulting off once the capacity ceiling was hit, rather than defaulting on and failing on every attempt. The pipeline's other channels kept running normally. The blocked channel simply sat out, cleanly, with the state of *why* it was sitting out recorded somewhere a human would actually look, rather than buried in a stack trace from the hundredth identical failed attempt. This matters because "at capacity" isn't the same category of problem as "broken." A channel that's broken needs someone to go fix code. A channel that's at capacity needs someone to either wait, or make a decision about priority — and a pipeline that treats both the same way, by failing loudly and identically, forces a human to re-diagnose "is this actually broken" every single time, which trains people to ignore the alert once they've done that diagnosis enough times and confirmed it's always the same non-issue. ## The flag has to be a decision, not a symptom The easy version of this pattern is dangerous: silently skipping a failing step and calling it "graceful" is how real failures go unnoticed for months, because "skipped" and "silently broken and skipped" look identical from the outside if nothing distinguishes them. The difference here was that the skip was gated on a specific, checkable condition — a known capacity ceiling, confirmed by the platform's own reporting, not inferred from a request failing — and the skip state was visible and re-evaluated automatically once the condition changed, not manually flipped back on and forgotten in an "off" position for a boundary case that resolved itself weeks ago. A flag that defaults to skip-on-failure without that distinction isn't a resilience feature. It's a way to convert every real outage into a quiet, permanent degradation nobody follows up on, because nothing about "the flag is off" tells you whether that's still the right call. ## What I check for now in any multi-channel pipeline When I look at automation that fans out across several independent channels or integrations, I specifically look for whether a failure in one is contained to that one, and whether the containment mechanism carries enough context to be re-evaluated later rather than just suppressing the symptom. "One channel is down and the others kept running" is the correct shape of resilience. "One channel is down and nothing tells you it's still down three weeks later" is the same mechanism with the one piece that actually matters left out.