ToolScore
Data & Files 4 min read

How We Beat Cloudflare With a Python Library, Not a Browser Farm

1 library replaced a browser farm the number that matters here

By Stan K.

Real lessons from scraping 1,500+ live data channels: TLS fingerprinting blocks plain requests before a single header lands, curl_cffi impersonation fixes layer 1–2 for free, and browser automation is the expensive last resort, not the default.

We maintain scrapers for well over a thousand live data channels — county records portals, auction sites, registries. Over that volume you meet every flavor of blocking, and the single most useful thing we learned is cheap to copy: most “Cloudflare blocked us” problems are not Cloudflare problems. They are TLS problems.

The three layers of bot detection

Modern anti-bot systems check you in order, and the cheapest check runs first:

  1. TLS fingerprint. Your handshake happens before any HTTP header. Python’s requests, httpx and aiohttp all present the same fixed OpenSSL/JA3/JA4 signature that no real browser has ever produced. You can be rejected here with a perfect User-Agent and impeccable headers — nothing you set matters because the server never reads it.
  2. HTTP/2 and header order. Real browsers order headers and frame HTTP/2 in specific ways; libraries don’t.
  3. JavaScript and behavior. Turnstile challenges, DataDome interstitials, mouse-movement scoring. This is the expensive layer — the one that actually needs a browser.

The insight: a huge share of blocks fire at layers 1–2, and layers 1–2 can be fixed without a browser, with a ~$0 tool.

curl_cffi: impersonation for the first two layers

curl_cffi (MIT license) wraps curl-impersonate and replicates the exact TLS signature, cipher ordering and HTTP/2 framing of real Chrome/Firefox/Safari builds. impersonate="chrome" selects the current bundled profile. It cannot execute JavaScript — by design it will not pass interactive Turnstile or DataDome walls. It covers layers 1–2 completely, for free.

Where this worked for us

County recorder and auction portals behind Cloudflare TLS checks that returned challenge pages to plain requests started returning real HTML the moment the same call went through curl_cffi with a browser profile. No proxy, no headless Chrome, no captcha budget — the site was only ever checking the handshake. We now treat curl_cffi as the default client and escalate only when a target proves it needs more.

For the rare genuinely guarded portal — JS-rendered grids, widget-based navigation — we escalate to Playwright, but through the site’s own preset widgets rather than fighting raw URLs, which is a lesson in itself: some portals lie to direct GET requests and only behave when driven the way their frontend drives them.

The pacing lesson browsers can’t teach you

The second thing volume teaches you: you get blocked by rate patterns long before content checks. Our search backends rotate seven search engines in three groups precisely because a single endpoint asked at full speed soft-blocks you with junk results — the most dangerous block, because it looks like success.

The same discipline applies to scraping: light pacing with jitter, ramp-up on a fresh target, and a stop rule after consecutive failures. A scraping stack that respects the target’s rate runs for months; one that hammers runs for hours and burns the IP for everyone.

When you genuinely need the browser

Escalate to Playwright/Camoufox only when the data physically isn’t in the HTML:

  • Content rendered client-side from authenticated APIs
  • Multi-step wizards (accept terms → address → results) where the flow itself is the payload
  • Interactive challenges after layers 1–2 pass

And even then: Camoufox (MPL-2.0, actively developed) exists precisely because vanilla Playwright announces its automation flags. The full tool-selection guide — Scrapy for bulk static, Crawlee for mixed HTTP+browser, Playwright for journeys — is in the open-source table of our scraping hub.

The decision tree we actually use

  1. Is the data published in bulk or via API already? → Don’t scrape at all
  2. Plain fetch fails? → curl_cffi with browser impersonation — still $0
  3. Data not in HTML? → Playwright through the site’s own widgets, paced
  4. Interactive challenge walls at volume? → This is the only case where paid scraper APIs and unblockers earn their fee

Most projects we see stop at step 2. The full cost math for each branch is in The Real Cost of Scraping — the no-JS path runs ~$125/month at a million pages, versus ~$1,023 for the same volume through a browser-based DIY stack.

Numbers change — the math doesn't
Token prices and plan limits shift monthly; the structural arguments in this guide age slowly. For current per-tool numbers see the live rankings.