A tagline match isn't a role match: what our own qualifier bug actually cost us

# A tagline match isn't a role match: what our own qualifier bug actually cost us I built an agent to decide, on its own, whether a job posting was worth applying to. It ran for weeks looking healthy. Then I read a batch of its actual decisions line by line, and found it had been confidently wrong in two different ways at once. ## The Setup The qualifier's job was simple on paper: read a posting, decide fit or no-fit, move on. Two checks did most of the work. One matched the company's own tagline text against an expected role type — if the tagline read like the kind of company that hires for this role, treat it as a plausible match. The other was a hard filter on compensation: if the posting didn't show a clean USD salary figure, reject it. Both checks were reasonable-sounding shortcuts. Neither one was actually checking what it claimed to check. ## What Was Actually Happening The tagline match was doing type inference by vibes. A company whose one-line description happened to contain the right keywords got treated as a role-type match, even when the actual open position was a completely different function at that company. The tagline told you what the company does in general. It told you nothing about which role this specific posting was for. The qualifier conflated the two anyway, because checking the tagline was easy and checking the actual role type required one more real comparison it wasn't making. The compensation filter had the opposite problem: it was too blunt in the other direction. Any posting listing equity instead of, or alongside, a salary got rejected outright. Any posting quoting comp in a non-USD currency got rejected outright. Neither of those is actually a disqualifying signal on its own. An equity component doesn't mean a role is unpaid or informal. A non-USD figure doesn't mean a role isn't remote-eligible or well-compensated — it just means the number needs to be read in the currency it's actually denominated in. The filter had turned "I don't know how to parse this compensation format" into "reject this role," and those are not the same decision. ## Finding It Nothing about this looked broken from the outside. No errors. No crashes. The agent applied to a steady stream of postings and reported each decision as confident. The only way to catch it was to stop trusting the aggregate success rate and actually read a sample of individual decisions against the real posting text. That's where it showed up. A role at a company whose tagline mentioned a keyword the qualifier keyed on, but whose actual open position was for a completely different function — approved anyway, because the tagline check never looked past the keyword. A remote role with a clearly strong package quoted partly in equity — rejected outright, with the equity treated as if it meant "unpaid." A well-compensated role priced in a non-USD currency — rejected, with the currency mismatch treated as if it meant "not eligible," when it just meant the parser hadn't been taught to read that number. Two different bugs. Same root shape: a proxy signal standing in for the real question, without ever being checked against what it was actually a proxy for. ## The Fix Both fixes followed the same principle — stop trusting the proxy, check the actual thing. For the tagline match, that meant adding an explicit role-type comparison: does the specific posting's role type match what we're looking for, not just does the company's general description contain the right words. The tagline is now one input into that comparison, not the whole decision. For compensation, that meant replacing the blanket USD-only filter with actual handling for the cases it was rejecting by default: parse equity as a component of comp rather than a disqualifier, and parse non-USD figures instead of discarding anything that wasn't already in the expected format. The filter's job changed from "reject anything I can't read" to "read it, then decide." ## The Broader Lesson A heuristic that's right most of the time will look right in aggregate for a long time. Success-rate dashboards and pass/fail counts don't distinguish between "correctly matched" and "matched for the wrong reason that happened not to matter yet." The only way to catch a proxy signal quietly standing in for a real check is to periodically stop trusting the aggregate and read individual decisions against ground truth. The other pattern worth naming: "I can't parse this" and "this should be rejected" feel like the same outcome from inside a filter, because both end in the item getting dropped. They are not the same claim, and collapsing them means every format your parser doesn't yet handle silently becomes a rejection rule instead of a parsing gap you'd actually want to know about and fix. If an automated system is making yes/no decisions at any real volume, the question worth asking regularly isn't "what's the pass rate." It's "if I read twenty of these decisions by hand right now, would I sign off on every one of them." That's the check that actually catches this.