Same code, same machine, same network — the only difference was a visible window

We had a scraper that returned zero results, every time, in every environment we tried — sandboxed CI, cloud dev boxes, even a fresh VM on the same cloud provider our production jobs run on. The working theory, reasonable enough at first, was network-level: something about the sandbox's outbound IP range was getting blocked or rate-limited by the target site. We spent real time chasing that theory: rotating egress IPs, checking for CAPTCHA challenge headers, comparing request fingerprints across environments. All roads led back to the same page: a bot-detection challenge screen, title unchanged, body full of the vendor's own branding. The candidate list wasn't unreasonable. But it also wasn't right — and the process for finding out was checking the assumption directly. ## Testing the actual variable, not the plausible one The break in the theory came from finally running the exact same script, from the exact same machine, on the exact same network — a real workstation, not a sandbox — with the headless flag left in. Same failure. Then the same script again with headless off. It cleared instantly: real page title, real listing rows, no challenge screen in sight. That ruled out network and IP entirely. The environments only ever *looked* network-related because every environment tested so far happened to also be running headless. It was a single confound hiding behind a plausible-sounding one. ## What's actually being fingerprinted Modern bot-detection vendors don't only look at request headers or IP reputation anymore — they profile the browser's own runtime signature: how many CPU cores it reports, whether `navigator.webdriver` is set, timing characteristics of canvas/WebGL rendering, even how mouse and scroll events arrive (or don't). A headless Chromium instance, even a well-configured one with common stealth patches applied, still diverges from a real, windowed browser session on enough of those signals to get flagged — while a headed instance, automated identically, does not. That's an uncomfortable property for infrastructure that's supposed to run unattended: it means the "correct," lower-overhead mode — headless, on a server, no display — is the one variant guaranteed to fail, and the fix is closer to "keep a real window open" than "add a stealth flag." ## The fix, and its cost The actual fix was almost embarrassingly simple once isolated: run headed by default, but push the browser window off-screen — a fixed, arbitrary display coordinate far outside any monitor's bounds — so it never visibly appears. The automation gets a real, windowed Chromium instance, with all the runtime signals that implies, without needing a monitor pointed at it. Headless mode stays available for local debugging only, with the explicit expectation that it'll still trip the same wall. The cost is a real constraint, not a cosmetic one: a headed browser needs an actual GUI session to attach to. That rules out the class of infrastructure most scraping jobs default to — a headless server, a minimal container, a CI runner with no display server at all. The job now has to run somewhere with a real login session present, which is a meaningfully different, and more expensive, operational requirement than the one it started with. ## The lesson The instinct to reach for the plausible explanation — network, IP reputation, rate limiting — instead of the actual one cost real time. Every environment where the bug reproduced also happened to be headless, which made "it's the network" look confirmed when it was actually just correlated. The fix was to change exactly one variable at a time on the real target, on the real machine, until the confound broke: identical script, identical network, identical machine — headless fails, headed succeeds, every time, confirmed live. If your automation degrades to nothing "in the cloud" but works "on your laptop," check whether the actual difference is the network, or whether it's just the one property your laptop and the cloud box don't share: a window.