URL: https://webscraping.spaceBest viewed at 1024×768

Anti-Bot Published Jul 9, 2026 · 42 min read · 9,162 words

Bypassing anti-bot protections: TLS, fingerprints, and Cloudflare

A frank guide to anti-bot defenses and how scrapers get past them: TLS/JA3 fingerprinting, Cloudflare's challenge, PerimeterX, CAPTCHAs, residential proxies, and the libraries (curl_cffi, Camoufox) that actually work in 2026.

Anti-bot has gotten good. The era when a custom User-Agent was enough is over. Modern systems like Cloudflare, PerimeterX/HUMAN, DataDome, and Akamai Bot Manager fingerprint your traffic at the TLS and HTTP/2 layers, where no header can save you. This guide explains what they actually look at and which tools work in 2026.

If you've been scraping for more than a year, you've watched the same arc I have. First the target blocks you on the User-Agent, you change the string and you're in. Then it starts checking header order, you copy the exact browser header set and you're in. Then one day you send a perfect-looking request and get a 403 with a challenge page, or a 200 that's an empty HTML shell, or a response that renders nothing at all. That's the moment most people discover the game moved to a layer their HTTP client can't touch.

This post is the map of that layer. I've spent the last two years running production scrapers — for my own sites and under contract for targets I was explicitly allowed to hit — and the details below are the ones that actually mattered in the field. I'll be honest about which parts are solved problems, which parts are an arms race, and which parts are a waste of your money. A lot of what gets written about anti-bot bypass is either vendor marketing or advice from people who have never run a crawl at scale. I'm not going to do that to you. Where I give numbers, they are numbers I have seen on real targets, and I'll flag the ones that vary by site.

Before we go any further, the framing. Everything here assumes you are scraping your own properties, targets you have permission to scrape, or data that is clearly public. Bypassing access controls can violate Terms of Service and, in some places, anti-circumvention law. I'll come back to this at the end. The engineering is the useful part, so let's get to it.

Why browsers work but `requests` doesn't

The short answer: a browser is a program that can be verified, and requests is a program that cannot lie about what it is. When Chrome connects to a server, everything it sends is mutually consistent. The TLS handshake matches Chrome's, the HTTP/2 settings match Chrome's, the headers arrive in Chrome's order, the IP is a residential one, and if the server runs JavaScript, the response actually executes and sends results back. A bot detector isn't looking for any one signal. It's looking for a request that is consistent with being a real browser, end to end.

requests fails that test at the very first layer. Python's ssl module produces a ClientHello that no browser produces: the cipher order is OpenSSL's, not Chrome's, the extensions are a Python-native set, and version negotiation follows a completely different path. That alone is enough for a detector that keeps a database of known TLS fingerprints. When Cloudflare or DataDome sees a handshake that matches neither Chrome, nor Firefox, nor Safari, nor Edge, nor any of the mobile builds, the request is scored as suspicious before a single byte of HTTP arrives. You can send the most beautiful headers in the world and it will not matter.

The second failure is the HTTP/2 layer. When you negotiate HTTP/2, the client sends a SETTINGS frame with window sizes, stream concurrency limits, and header-compression preferences. Browsers pick specific values and send them in a specific order. requests, httpx, and most other Python clients advertise values that no browser uses. That's a second independent fingerprint, scored separately from the TLS one.

And then there's behavior. A real page load triggers dozens of subresource requests in a specific order — fonts, images, analytics, the challenge script itself. It runs JavaScript. It moves a mouse. A scraper that only fetches the HTML looks nothing like a page load, and the detector knows. This is why "I got the HTML with requests, why do they block me" conversations go the way they do. Getting the HTML is not the same as loading the page.

The detection stack, layer by layer

Let me go through each signal in the stack — what it measures, how easy it is to fix, and where it sits in importance. A detector layers these on top of each other and scores the whole request, not any one signal in isolation.

Relative discriminating power of the signals in a modern bot-detection stack. The TLS handshake and HTTP/2 settings dominate; the User-Agent string is nearly worthless to a detector in 2026.

Here's the same stack as a table, because the "fix" column is where most people go wrong — they patch the easiest layer and skip the ones that matter.

Signal	How it is detected	What fixes it
TLS handshake	JA3/JA4 hash of the ClientHello matches no known browser	curl_cffi impersonate, tls-client, a real browser
HTTP/2 SETTINGS	Settings values and order differ from Chrome or Firefox	curl_cffi impersonate, a browser's own stack
Header order	Requests sends headers in a canonical, non-browser order	Copy the exact browser header order and casing
IP reputation	ASN is a cloud provider or the IP has a bad history	Residential or mobile proxy
JS challenges	Behavioral JavaScript runs in the page and reports back	A real browser, or a managed solver that runs one
TCP/IP	TTL, window size, and other OS defaults in the SYN packet	Weak on its own; usually fixed by using a real browser

IP reputation

The IP address is the first thing a detector sees, and it's the cheapest signal to get right. Every major provider keeps a constantly updated list of which netblocks belong to cloud providers — AWS, DigitalOcean, GCP, Hetzner, OVH, all of them — and assigns a risk score to requests that come from those ranges. If you're crawling from a $5 VPS, that's a signal before you've even made a request.

IP reputation also includes the history of the ASN. An IP that has made thousands of rapid requests in the past has a bad reputation. An IP from a residential broadband or mobile ASN starts with a much better baseline, because that's where normal users come from. Detectors also watch for entropy: a single IP that requests many different domains, or a pool of IPs that all behave identically, which is the signature of a proxy farm.

The fix for IP reputation is proxies, which I cover in detail later. The important thing is what proxies do not fix: they don't change your TLS fingerprint, your HTTP/2 settings, or your behavior. A datacenter IP with a perfect Chrome fingerprint still has a bad IP. A residential IP with a requests fingerprint still has a dead-giveaway TLS handshake. The layers compound, and you have to fix the right ones for the right reasons.

Header order and casing

HTTP/2 pseudo-headers and regular headers have a defined order that browsers follow. Chrome sends :method, :scheme, :authority, :path in that order, then accept, accept-encoding, accept-language, cache-control, and so on, preserving the casing Chrome chose like Sec-Fetch-Mode and Upgrade-Insecure-Requests. requests sends headers in a different order and lowercases them by default, which is itself suspicious — no real Chrome build sends a lowercase user-agent.

This layer is the easiest to fake. You can build a dict in the exact browser order and httpx will send it in that order if you don't let it normalize. But it's also the layer where most people stop, which is why so many tutorials promise "the perfect header set" and still get 403s. Headers are necessary but not sufficient. They get you nothing if the TLS handshake above them already failed.

The TLS handshake

I'll do a full deep dive in the next section, but the short version: the ClientHello is the strongest single signal in the stack because it's emitted before anything else and because very few HTTP libraries can control it. JA3 and its successor JA4 hash the ClientHello into a compact identifier, and the detector simply checks whether that identifier is a known browser value. If it isn't, everything after it is already tainted.

The HTTP/2 layer

On top of the TLS handshake, the HTTP/2 SETTINGS frame carries its own fingerprint: header table size, initial window size, max concurrent streams, max frame size, and the order the settings are sent in. Chrome advertises one specific set, Firefox another, Go's net/http a third, Python's h2 a fourth. curl_cffi's impersonate mode reproduces the browser's settings along with its TLS handshake, which is exactly why it beats hand-rolled clients that nail the TLS but write their own HTTP/2 layer.

There's also fingerprinting at the frame level — priority order, stream-creation patterns, whether the client uses server push. These are secondary signals, but they're cheap for the detector to collect and they compound with everything else.

TCP fingerprints

Below TLS is the TCP layer. The kernel that sends your packets leaves traces: the initial window size, the maximum segment size, the TTL, the timestamps option, the selective ACK settings. A Linux VPS, a macOS laptop, and Windows Chrome all produce slightly different TCP option sequences. Some detectors score these, though in practice the TCP fingerprint is a weak signal compared to TLS — it's hard to attribute reliably at the load balancer level and NAT makes it noisy.

I mention it because it explains a common failure mode: you randomize the things you can control but leave the TCP stack exactly as the kernel shipped it, so you're still emitting a consistent signal. And if you try to randomize the TCP stack too, you risk breaking connections and looking worse. Randomization at any layer is a trap, as I'll argue in the TLS section.

Browser behavioral signals

The deepest layer is JavaScript instrumentation. Cloudflare's challenge, DataDome's interstitial, and HUMAN's script all run in a real page context and collect information a plain HTTP client never produces: how long the JavaScript took to execute, what the rendering metrics look like, whether the challenge-script request happened in the right order, and what the JavaScript environment reports about its own authenticity.

The key trick in this layer is consistency. A real Chrome reports navigator.webdriver as false or absent, a real plugin list, a real language array, coherent values for screen.width and window.devicePixelRatio, and a window.chrome object. A headless browser leaks on several of these: navigator.webdriver is true, the plugin list is empty, the User-Agent contains "HeadlessChrome", and window.chrome is missing. The fix for these leaks is init scripts and patched browser builds, which is what the stealth tools in this guide do.

Canvas and WebGL

Canvas fingerprinting draws text and shapes to an offscreen canvas and hashes the pixels. Different GPU drivers, font rendering, and antialiasing settings produce measurably different hashes. WebGL fingerprinting does the same with the GL renderer string, the extension list, and shader behavior. Headless browsers and VMs often report a software renderer like SwiftShader, which is a strong bot signal because almost no real user has one.

The detector doesn't need your canvas hash to match a specific known device. It needs the hash to be stable across visits and consistent with the rest of your fingerprint. If your canvas hash changes between every request, or reports a software renderer, or is identical to every other visitor from the same datacenter, that's a signal.

Mouse movement and input

If a site runs behavioral scripts, they record mousemove events, scroll events, click timing, and key timing, and they analyze the trajectories. A human moves in curves with acceleration and small jitter. Automated mouse movement is linear, or instant, or follows a perfect bezier — which is itself a detectable pattern. Some detectors also measure the time between page load and first interaction, and the time between keystrokes. Bots interact too fast and too regularly.

There are two honest approaches here. One is to use a real browser and let a human, or a service that simulates a human, drive it. The other is to not fight this layer at all: choose targets whose detection doesn't require you to simulate a mouse, or accept that some sites are off-limits for automation. Trying to fake human mouse movement deterministically is a rabbit hole that consumes months and rarely pays off.

Request timing and concurrency

Finally, the meta-signal: how your requests are spaced. A human opens a page and pauses to read. A scraper sends requests in perfect clockwork rhythm. Detectors model this — they measure the distribution of inter-request gaps and flag anything that looks periodic. Aggressive proxy rotation is itself a pattern: if one IP makes a single request and is never seen again while a different IP from the same provider starts exactly when the first one stops, that's a rotation signature, and it's detectable.

This is why the worst thing you can do is hammer a target with 100 concurrent requests from a rotating pool and hope the diversity hides you. The diversity is the signal. A detector that sees the same pool, the same client, and the same timing across a thousand IPs has just learned the shape of your entire operation.

TLS fingerprinting, deep dive

The TLS handshake is where the modern detection game is won and lost, so let's spend real time on it. When your client connects to an HTTPS server, the first thing it sends is the ClientHello: the TLS version it wants, the cipher suites it supports, a list of extensions, and within those, the elliptic curves and point formats it can use. The exact bytes, and the order they appear in, are determined by the client library — not by you, not by the server, and not by any HTTP header you can set.

What JA3 actually computes

JA3 takes five fields from the ClientHello and concatenates them: the SSL/TLS version, the accepted cipher suites in order, the list of extensions in order, the elliptic curves, and the elliptic-curve point formats. That string is then hashed with MD5 to produce a 32-character value. Chrome 122 on Linux produces a specific JA3. Firefox produces a different one. Python's ssl produces a third that no real browser has ever produced.

The detector doesn't need to understand the handshake deeply. It has a lookup table of known client fingerprints, and it checks whether the JA3 it just saw is in that table. If your JA3 matches Chrome, you're in. If it matches nothing, or matches OpenSSL's default, you're scored as suspicious before you send a single HTTP request.

The order matters more than the set. Two clients could support the same twenty cipher suites, but if they list them in different orders, their JA3 differs. Browsers order ciphers by preference, with TLS 1.3 suites listed in a particular way. OpenSSL's default order is different. This is why you can't fix a fingerprint by simply adding or removing ciphers — you have to match the exact order, byte for byte.

JA4 and what it fixes

JA3 has two practical problems. The first is that it's an MD5 hash, so you can't read anything from it — two different handshakes can only be compared for equality, never for similarity. The second is that it ignores some handshake fields that turn out to matter, like the ALPN extension, which tells the server whether the client can speak HTTP/2, and the raw number of extensions.

JA4, published in late 2023 and now widely deployed, fixes both. It isn't a hash; it's a readable, sortable string like t13d1517h2_8daaf6152771_02713d6af862. The parts encode the TLS version, whether SNI is present, the cipher count, the ALPN protocols, then a truncated hash of the cipher suites and a truncated hash of the extensions. Because it's readable, a detector can say "this client is in the Chrome family but has an unusual extension set" rather than just "unknown."

For scraping, the practical difference is that JA4 makes near-misses visible. A client that fakes Chrome's JA3 but forgets ALPN, or adds an extension Chrome never sends, now produces a JA4 that clearly isn't Chrome's. The impersonation libraries keep up, but the bar is higher than it was in the JA3-only era.

Why OpenSSL and Go look different from Chrome

Python's ssl and Go's crypto/tls both build on the same idea — a portable TLS implementation — but neither reproduces Chrome's handshake. A few concrete differences:

Cipher order. OpenSSL's default order is optimized for OpenSSL's notion of security and performance. Chrome's order reflects its own preferences, including putting TLS 1.3 suites first and inserting GREASE values into the list.
GREASE. RFC 8701 defines reserved values that clients insert randomly to make sure servers don't hardcode assumptions. Chrome inserts GREASE into its cipher list and extensions. OpenSSL and Go don't. A handshake with no GREASE isn't necessarily a bot, but a handshake with no GREASE, from a datacenter IP, with OpenSSL's cipher order, almost certainly is.
Extension set. Chrome sends extensions like extended_master_secret, session_ticket, key_share, supported_versions, psk_key_exchange_modes, and application_settings, the last one mattering more every year because it advertises HTTP/3 support. Go sends a leaner set. The presence or absence of application_settings is one of the strongest signals in the HTTP/3 era.

The takeaway: you can't hand-roll this. By the time you've matched Chrome's cipher order, extension list, GREASE values, and TLS version list, you've essentially reimplemented a browser's TLS stack. That's exactly what the impersonation libraries do for you.

What a "random" TLS fingerprint looks like

There's a tempting idea floating around: if detection is about known fingerprints, just randomize your TLS handshake so nobody can match you. It doesn't work, and here's why. A random fingerprint is still a fingerprint — it just matches no known client. Detectors score you on how far you are from the known-good set, not on whether you match something. If your ClientHello advertises thirty cipher suites in a random order, with a random extension list, and no GREASE, that's more suspicious, not less, because the detector's model of a real user never includes such a handshake.

Worse, randomizing the handshake breaks real-world TLS: some servers reject handshakes they don't understand, some middleboxes choke on unusual cipher lists, and you'll spend your debugging time on connection failures rather than blocks. The correct approach is not randomization. It's replication. Match a real, known client exactly, and stay matched.

How often each client's TLS handshake matches a real Chrome build. The standard library clients sit near zero; impersonation libraries and real browsers sit near one hundred.

curl_cffi and the impersonate parameter

The practical fix for all of this is curl_cffi, a Python library that wraps a patched build of curl whose TLS library is configured to produce a browser's handshake. The impersonate parameter picks which browser profile to copy, and it covers the JA3, the JA4, the HTTP/2 settings, and the header order in one argument:

# pip install curl_cffi
from curl_cffi import requests

resp = requests.get(
    "https://protected.example.com/",
    impersonate="chrome",   # reproduces Chrome's JA3/JA4 + HTTP/2 settings
    timeout=15,
)
print(resp.status_code, resp.text[:200])

This one line fixes the first three layers of the detection stack. For a huge class of targets — the "I get 403 with requests but my browser works" tier — this is the entire solution. It's worth knowing the versioned identifiers too: chrome124, chrome131, safari17, edge99, firefox130. The version matters because Cloudflare keeps a model of what current browsers look like, and impersonating an obsolete Chrome is a weaker match. Impersonate the newest stable build you can.

You can verify what your client actually looks like on a service like tls.browserleaks.com or Cloudflare's TLS test page. This is genuinely worth doing once, because it replaces folklore with data:

# pip install tls-client
import tls_client

session = tls_client.Session(client_identifier="chrome_124")
r = session.get("https://tls.browserleaks.com/json")
print(r.json()["ja3_hash"])   # compare against a real Chrome's JA3

If your client's JA3 hash falls in the same family as Chrome's known values, you've done the job. If it's something no browser produces, no amount of User-Agent rotation will save you.

HTTP/2 fingerprinting

A correct TLS handshake gets you through the door; the HTTP/2 SETTINGS frame is the second check. When a client negotiates HTTP/2, it immediately sends a SETTINGS frame announcing its parameters: the header table size, the initial flow-control window, the maximum number of concurrent streams, the maximum frame size, and whether it supports header compression. Browsers send these in a specific order with specific values.

Chrome, for example, sends a header table size of 65536, an initial window size of 6291456, max concurrent streams of 100, and a max frame size of 16384. Go's http2 sends a different window size. Python's h2 sends yet another set. A detector that sees an HTTP/2 connection can fingerprint the settings and check them against known client profiles, independently of the TLS handshake. This is why a hand-rolled client that nails JA3 but writes its own HTTP/2 stack still gets flagged.

curl_cffi's impersonate mode covers this too — it reproduces the browser's SETTINGS values and order along with the handshake. If you're building a custom client, you can check what your settings look like with the h2 library:

# pip install h2 httpx[http2]
import h2.config
import h2.connection

c = h2.connection.H2Connection(
    config=h2.config.H2Configuration(client_side=True)
)
c.initiate_connection()
for k, v in c.local_settings.items():
    print(k.name, v)

Compare that output to what Chrome actually sends and you'll see why raw clients get caught. On top of SETTINGS, detectors look at the order of your WINDOW_UPDATE frames, whether you send priority information the way browsers do, and whether you open the number of parallel streams a real page load would. These are weaker signals on their own, but they're free to collect and they tighten the scoring around an already-suspicious client. A quick check that your client negotiates HTTP/2 at all is worth doing first:

# pip install httpx[http2]
import httpx

with httpx.Client(http2=True) as client:
    r = client.get("https://www.cloudflare.com/")
    print(r.http_version)   # "HTTP/2" if the server negotiated it

How Cloudflare's challenge works

Cloudflare is the most common wall you'll hit, so it deserves its own section. When a request is scored as suspicious, Cloudflare responds with an interstitial page. On a browser, you see the spinner and the text "Checking your browser before accessing..." or, on many sites, "Just a moment...". If the challenge passes, the page continues and Cloudflare sets a cookie called cf_clearance. If it fails, the interstitial loops.

What "Just a moment..." actually is

The interstitial is a JavaScript challenge, not a delay. The page ships a script that performs several checks before it lets you through. Modern variants do proof-of-work: the script runs a computation designed to take a few hundred milliseconds to a few seconds on a real browser. The "5-second interstitial" people talk about is the proof-of-work plus a few validation round-trips, not an actual sleep. The script also probes the browser environment: it checks that JavaScript APIs exist and behave, that the canvas and WebGL renderers are real, that the User-Agent and platform are consistent, and that the request for the challenge script came from a real page load rather than a direct fetch.

The result of the computation is sent to Cloudflare's edge, and if everything checks out, the edge returns the cf_clearance cookie with a TTL — commonly 15 to 60 minutes, though sites can configure it. The cookie is bound to the IP address, the User-Agent, and the ASN that solved the challenge. Replaying it from a different IP or User-Agent invalidates it.

Managed Challenge vs Turnstile

There are two things people call "Cloudflare challenges" and they're different. Managed Challenge is the interstitial flow I just described — the proof-of-work page that shows up on a suspicious request. Turnstile is Cloudflare's CAPTCHA product: a widget, often invisible, that runs in the page and issues a token, meant to be used in your own forms and login flows. Turnstile isn't Cloudflare-exclusive — any site can embed it. So when a target uses Turnstile, you're solving Cloudflare's challenge in a form context, and the token is submitted with the form. Managed Challenge, by contrast, gates page loads on Cloudflare-proxied sites.

The practical difference for a scraper: a Turnstile token is often enough to submit a form or an API request, and the CAPTCHA services can solve it. A Managed Challenge is a full page-level gate, and the cf_clearance cookie is what you need.

If you solve the challenge in a browser, you can harvest the cf_clearance cookie and replay it in a fast client for the bulk of your crawl. The constraints: same IP, same User-Agent, same ASN, and within the TTL. If you rotate IPs, you have to solve per IP. If your IP's reputation is bad enough that Cloudflare re-challenges on every request, harvesting is pointless — you'll spend all your time solving.

There are three workable approaches, roughly in order of robustness:

Don't trigger it. A correct TLS fingerprint, a residential IP, and a sane rate often mean Cloudflare never challenges you at all. This is the cheapest solution and the one I reach for first.
Solve it in a browser, then replay. Use Playwright or Camoufox to solve the challenge, save the cf_clearance cookie, and replay it in curl_cffi for the volume of requests. Watch the TTL and re-solve before it expires.
Use a managed solver. Services like ScrapingBee, ZenRows, or the CAPTCHA providers' Cloudflare modules run the challenge for you. You pay per successful response. This is the right call when you need scale and reliability more than you need to save money.

Share of Cloudflare challenges completed on the first attempt by setup. The jump from the requests client to a browser-accurate TLS client is larger than the jump to a real browser — the fingerprint is the bigger lever.

What I see people get wrong: they treat Cloudflare as a single binary "blocked or not." It's a scored system with multiple tiers. The same site will challenge a requests client from a datacenter IP immediately, let a curl_cffi client from a residential IP through without a challenge, and re-challenge a previously validated session only when behavior degrades. The cheapest win is almost always to look more like a browser at the layers you control, so the challenge never appears.

The commercial detectors: DataDome, HUMAN, Akamai, Kasada

Cloudflare is the most common, but the harder targets use the commercial bot-management vendors. Each has a different fingerprinting emphasis, and knowing which one you're facing tells you which layer to fix. You can usually identify the vendor by response headers or by the script names in the challenge page.

DataDome

DataDome sits in front of a site or API and scores every request. It's known for an aggressive JavaScript challenge that appears when you fail the risk score, and for how heavily it leans on IP reputation and behavioral signals in addition to TLS. DataDome maintains a large fingerprint database and is particularly good at catching headless browsers: it checks for missing plugin lists, inconsistent navigator properties, and CDP artifacts. It also fingerprints the HTTP/2 layer carefully, so a correct TLS handshake with default h2 settings still scores badly.

The adaptation for DataDome-protected targets is usually the full stack: a real browser with a stealth patch, a residential IP, and human-like timing. The curl_cffi-only approach works for the easiest DataDome configurations and fails on the hardened ones. If you're facing DataDome on a target you genuinely need, expect to run a browser, not a library.

PerimeterX / HUMAN

PerimeterX, now HUMAN, takes a different approach: heavy client-side instrumentation. The site loads a large obfuscated script that collects a wide behavioral profile — mouse movement, touch events, DOM interactions, canvas and WebGL hashes, and dozens of browser-environment probes — then posts that profile to HUMAN's scoring API and gets back a token that gates the request. The token is short-lived and bound to the session.

What this means in practice: a plain HTTP client cannot get HUMAN's token, because the token is the output of a page that must actually run. The options are a real browser (let the instrumentation run, grab the token) or a managed service that runs the page for you. The good news is that HUMAN's instrumentation is mostly a page-load event — if you load the page in a browser and then use the session, you can often crawl with browser sessions rather than raw HTTP. The bad news is that HUMAN's models are behavioral, so a headless browser that doesn't move a mouse or scroll can still be scored as a bot even with the token.

Akamai Bot Manager

Akamai scores traffic on a very large feature set — hundreds of signals, including TLS and HTTP/2 fingerprints, header order, TCP characteristics, and a JavaScript sensor. The sensor script runs in the page and reports environment data back to Akamai, which returns a score and decides whether to challenge, throttle, or allow. Akamai is heavily deployed on commerce and ticketing sites, and it's one of the vendors most likely to use risk-based responses: slow down suspicious users rather than block them outright, which is harder to detect because you get 200s that are just slower and occasionally wrong.

Adapting to Akamai usually means matching the browser fingerprint at every layer and passing the sensor. That's real-browser territory again. Akamai's sensor is also known for detecting automation itself — some of the stealth patches that work against Cloudflare are caught by Akamai because the sensor checks for their specific artifacts.

Kasada

Kasada is the hardest of the four, and it's worth understanding why. It serves an extremely obfuscated JavaScript challenge with heavy proof-of-work — the challenge is designed so that the only economical way to solve it is to run it in a real browser. Kasada is aggressive about detecting headless environments: WebGL and canvas are probed, GPU renderers are checked, and even subtle things like the number of cores and the exact performance timing of the challenge execution are scored. There are no tokens to reuse the way cf_clearance works — the solution is tied tightly to the session and the environment.

For a legitimate project, the Kasada response is usually: use a real browser with strong stealth, a clean residential IP, and a low rate, or use a managed solver that has built Kasada integration. Some of the CAPTCHA services and scraping APIs support Kasada precisely because it's too expensive to fight yourself. My honest advice: if a target you're allowed to scrape runs Kasada, budget for a managed service or for substantial browser-engineering time. This is the tier where a lot of people realize the target isn't worth it.

The common thread across all four vendors: they all fingerprint TLS, HTTP/2, and headers, and they all add a layer on top that requires a real browser. The libraries get you past the first two layers; the browser tools get you past the third. Nothing gets you past all of them for free.

Browser-grade tools and what they actually fix

When the detection stack requires a real browser, the game shifts to automation with an anti-detection layer. Let me be precise about what each tool fixes, because the marketing around these is worse than useless — it's misleading.

Playwright plus stealth patches

Playwright is the standard for browser automation, and out of the box it is trivially detectable. It sets navigator.webdriver to true, runs Chromium with a headless flag that leaks in the User-Agent, has an empty plugin list, and exposes a CDP surface that detectors probe. Stealth patches close the most obvious leaks with init scripts that run before the page's own JavaScript:

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright

STEALTH = """
// Remove the biggest tells before any page JS runs
Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
Object.defineProperty(navigator, 'plugins', {
  get: () => [1, 2, 3, 4, 5]          // pretend we have plugins
});
Object.defineProperty(navigator, 'languages', {
  get: () => ['en-US', 'en']
});
window.chrome = { runtime: {} };      // headless Chrome lacks window.chrome
"""

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    ctx = browser.new_context(locale="en-US")
    ctx.add_init_script(STEALTH)
    page = ctx.new_page()
    page.goto("https://bot.sannysoft.com/")
    page.screenshot(path="fingerprint-check.png")

Run that and look at the screenshot: you'll see which rows are red. A stealthed Playwright fixes navigator.webdriver, the plugin list, the languages array, and a few other basics. It does not fix the WebGL renderer, it does not fix canvas noise, and it does not fix timing or mouse behavior. Stealth is a layer, not a guarantee.

Camoufox

Camoufox is the more serious option. It's a build of Firefox modified specifically for anti-detection: it ships with fingerprint randomization, canvas and audio noise by default, a proper navigator surface, and no headless-mode tells because it's a real browser window. It works with Playwright via a special driver, and it's the tool I reach for when a target's challenge checks WebGL and canvas rather than just navigator.webdriver.

The tradeoff: it's Firefox, so sites that assume Chrome can still flag it, and it's a smaller project than Playwright, so some site-specific quirks surface that the big vendors have already solved. But for a hardened Cloudflare or DataDome target, Camoufox plus a residential proxy is the combination that gets through most of the time.

Puppeteer-extra-stealth

If you're in the Node ecosystem, puppeteer-extra-stealth is the equivalent of the Playwright init-script approach: a plugin that patches dozens of browser properties before page scripts run. It fixes the same first-layer leaks and has a similar ceiling — it doesn't hide the fact that a real browser process is under automation control when a detector probes deep enough. It's fine for moderate targets and much less effective against the vendors that check for the patch itself.

Patchright

Patchright is a patched build of Playwright that removes automation artifacts from Chromium itself — the parts of the binary and the CDP surface that expose that Playwright is driving it. Where init scripts are surface-level fixes, Patchright modifies the browser, which is strictly better against detectors that check for CDP properties or Playwright-specific markers. If Playwright plus an init script gets you sixty percent of the way on a hardened target, Patchright gets you closer to ninety, and it costs nothing but a change of import.

The honest summary of the browser tier is this table. Each tool fixes a layer; none of them fix everything:

Tool	What it fixes	What it doesn't
curl_cffi	TLS, HTTP/2 settings, header order	JS challenges, behavioral detection, CAPTCHAs
tls-client	TLS impersonation	HTTP/2 settings if not configured
Playwright + stealth	JS challenges, real browser runtime	WebGL/canvas tells, headless quirks, IP reputation
Patchright	Playwright-specific detection leaks	Behavioral timing, IP reputation
Camoufox	Canvas/WebGL noise, real Firefox runtime	Nothing is 100 percent; Chrome-only sites may flag it
Residential proxies	IP reputation, geo-blocks	Bad TLS or HTTP/2 fingerprints

The gap that no browser tool closes

Here's the uncomfortable truth: every automation tool leaks, and the best detectors are built to find the leaks that specific tools introduce. This is why the arms race doesn't end. A detector that knows Patchright's build hash can flag it. A detector that knows Camoufox's fingerprint profile can flag it. A detector that scores mouse trajectories will flag any automation that doesn't simulate human motion. The winning move for most projects isn't a better stealth tool. It's picking targets and rates where the target's detection tier is lower than the effort you're willing to spend. More on that in the workflow section.

CAPTCHA handling

A CAPTCHA means something upstream already failed. The detector decided you were suspicious enough to require a proof of humanness, and no amount of TLS spoofing is going to undo that decision for the current session. So the first question to ask when you see a CAPTCHA isn't "how do I solve it" — it's "what made the detector suspicious." Usually the answer is one of the layers above: a datacenter IP, a headless browser, or an impossible request rate.

When you do need to solve one, the options are:

In-browser solving. If you're driving a real browser and a CAPTCHA appears, a human can solve it once and the session's cookies and tokens carry forward. For low-volume work this is completely fine. It's also the only option that handles brand-new CAPTCHA variants, because a service can't solve a CAPTCHA it hasn't seen.
CAPTCHA-solving services. 2Captcha, CapSolver, Anti-Captcha, and DeathByCaptcha accept an image or a site-key-plus-page-url, return a token, and charge per solve. They support reCAPTCHA v2 and v3, hCaptcha, and Cloudflare Turnstile. Solve times run 5 to 30 seconds, and they're a per-request cost, so they're viable for occasional solves and expensive for volume.
Token-based integration. For reCAPTCHA and Turnstile, the site doesn't see the CAPTCHA as an image; it sees a token in the form. You call the service with the site key and page URL, get the token, and submit it. That's the integration you'll actually write:

# pip install requests
import time
import requests

def solve_turnstile(site_key: str, page_url: str, api_key: str) -> str:
    submit = requests.post("https://2captcha.com/in.php", data={
        "key": api_key,
        "method": "turnstile",
        "sitekey": site_key,
        "pageurl": page_url,
        "json": 1,
    }).json()
    task_id = submit.get("request")
    for _ in range(60):
        result = requests.get("https://2captcha.com/res.php", params={
            "key": api_key, "action": "get", "id": task_id, "json": 1,
        }).json()
        if result.get("status") == 1:
            return result["request"]   # the token to submit
        time.sleep(5)
    raise TimeoutError("CAPTCHA solve timed out")

The economics matter. At a few cents per solve, a site that CAPTCHAs every request costs you both money and latency. That's usually the signal that the target is not worth an automated crawl — a real user would have been CAPTCHA'd once, not on every page. When a target challenges every request, I treat it as a design decision by the operator: they don't want automated access. Respect that and move on, or scope the project around it deliberately.

Proxies: what each type actually fixes

Proxies fix exactly one thing: IP reputation. They do not fix a bad TLS fingerprint, they do not fix a headless browser, and they do not fix an impossible request rate. The detector reads your fingerprint and behavior before it cares much about your IP, and a residential IP on a requests client is still a residential IP on a dead-giveaway TLS handshake. Pair them — never substitute.

The proxy market breaks into four tiers:

Datacenter. A few dollars per gigabyte, or a flat VPS rate. Comes from cloud ASNs, which reputation systems flag. Good for targets that don't filter on IP at all, or for testing. This is where almost everyone starts, and where most "why am I blocked" complaints come from.
ISP / static residential. IPs that belong to real broadband providers but are sold as fixed addresses. More expensive than datacenter, much better reputation, and the IP is stable — which matters for login sessions and for cf_clearance reuse, since the clearance cookie is bound to a stable IP.
Residential rotating. A large pool of real user IPs, rotated per request or per session. The best IP reputation money can buy at scale, but rotation itself is a pattern. If you rotate per request, every request looks like a brand-new user who instantly does exactly one thing — that's detectable. If you rotate per session and keep each session's requests human-shaped, you look like many users, which is the point.
Mobile. IPs from mobile carriers, the rarest and most expensive tier. Mobile ASNs have the most permissive reputation, which matters against the strictest targets. The cost is often high enough that it's only worth it for the last five percent of targets that nothing else gets through.

Proxy tiers by cost per gigabyte against anti-detection effectiveness. The effectiveness axis is IP reputation alone — no proxy type fixes a bad fingerprint, and the tiers differ on nothing else.

A few things I've learned the hard way about proxy hygiene:

Match your rate to the proxy tier. A residential pool that supports 20 concurrent requests is fine; 200 concurrent requests from the same pool will get the whole pool flagged, and pool providers terminate accounts for that.
Keep sessions sticky when it matters. For any target that issues a token bound to a session — Cloudflare, DataDome, anything with a login — a rotating IP per request is fatal.
Stagger your rotation. If you must rotate, rotate between sessions, not between requests, and add human-sized pauses.
Measure your IP's reputation before you build on it. Hit a free reputation checker once with each candidate proxy and record the score. Proxy quality varies wildly between providers and even within a provider's pool.

The pairing is what makes the whole thing work. curl_cffi for the fingerprint, residential for the IP, and a rate that doesn't look automated:

from curl_cffi import requests

resp = requests.get(
    url,
    impersonate="chrome",
    proxies={"https": "http://user:pass@residential.proxy:4444"},
    timeout=15,
)

Rotate responsibly. One request per IP per second is a sane ceiling for most targets; one IP per request is a pattern. Aggressive rotation is itself a fingerprint.

Is the target worth the fight? A realistic workflow

Every time I start a new scrape, I run the same decision procedure. It saves more hours than any stealth tool.

Step 1 — Probe from a clean browser. Open the site in a regular browser on a residential connection and see what happens. If it loads normally, you have a baseline: the site doesn't challenge everyone. Save the exact request headers and cookies a browser sends.

Step 2 — Try the cheap client. Run the same request through curl_cffi with impersonate set to your browser's version, from the same network. If it returns the same HTML, you're done — the site's detection tier is low, and you can build the whole scraper on curl_cffi plus a modest rate limit.

Step 3 — Add a proxy tier. If the datacenter path fails on IP reputation but the residential path works, decide whether residential is worth the cost for the data you're getting. Most targets live here.

Step 4 — Escalate to a browser. If a curl_cffi plus residential combination still gets challenged, run the page in a stealthed Playwright or Camoufox and see whether the challenge auto-solves and whether the session stays valid. If it does, you're in the browser tier: higher cost per request, higher success rate, and a real engineering budget.

Step 5 — Decide. If the target requires a managed CAPTCHA service on every request, or a mobile proxy pool, or a custom browser patch to stay alive, do the math. Take the cost per successful response and multiply it by the number of pages you actually need. Nine times out of ten, the number comes out higher than the data is worth, and the right engineering decision is to not scrape that target — or to use a managed scraping API that has already solved the problem at a lower effective price.

The trap is sunk cost. You've spent two days on a target, so the "obvious" next step is to spend two more days on the next stealth layer. It isn't. The layers stack additively, and so does the cost. There is no configuration that defeats a well-run bot-management system at scale; there are only configurations that are cheaper than the data they protect. Choose your targets with that in mind and you'll never be surprised by a block again.

A note on ethics and law

I said I'd come back to this, so here it is plainly. Bypassing access controls can violate a site's Terms of Service and, in some places, anti-circumvention law such as the DMCA's anti-circumvention provisions or computer-fraud statutes. Whether something is legal depends on what data you're accessing, whether it's public, the jurisdiction you're in, and the jurisdiction the target is in. None of this guide changes that.

Everything here is for scraping your own properties, targets you have permission to scrape, or data that is clearly public. If a site is behind a login or a paywall, you don't get to bypass the wall because you want the data — you get to work within the site's terms. The ethics and robots.txt guide covers this in more detail. Read it before you point any of this at someone else's site.

The professional frame I use: rate limits, caching, politeness, and consent. Those are the things that keep scraping sustainable for everyone. The scraping at scale guide and the Playwright guide are the right places to build the responsible version of everything in this post.

Key takeaways

Detection happens at the TLS and HTTP/2 layers now, not the User-Agent. Headers are the easiest layer to fix and the least important one.
JA3/JA4 fingerprint the TLS ClientHello; JA4 is readable and catches near-misses that JA3 missed. Randomizing a fingerprint is worse than matching one.
curl_cffi with impersonate="chrome" is the single highest-leverage change for most 403s. Verify your JA3 against a real browser once.
Cloudflare's challenge is proof-of-work plus browser checks, not a delay. The cf_clearance cookie is bound to IP, User-Agent, and ASN, and has a short TTL.
DataDome, HUMAN, Akamai, and Kasada all fingerprint TLS and HTTP/2 and then add a browser layer. The harder vendors require a real browser, not a library.
Playwright, Camoufox, and Patchright each fix a specific set of leaks; none of them fix everything. The arms race is real.
Proxies fix IP reputation and nothing else. Pair them with a correct fingerprint; never substitute them for one.
The best CAPTCHA strategy is not triggering one. Solving CAPTCHAs per-request is a cost signal that the target isn't worth crawling.
Run the five-step workflow before you build anything. Most targets are worth the cheap client and nothing more.

Frequently Asked Questions

Is bypassing anti-bot protection legal?

Bypassing access controls can violate a site's Terms of Service and, in some places, anti-circumvention law. It depends on what data you access, whether it's public, and where you are. This guide is for education and for scraping your own or clearly-permitted targets. It's not legal advice.

Why does my scraper get blocked but my browser works fine?

Anti-bot systems fingerprint more than your User-Agent. They inspect your TLS handshake (JA3/JA4), HTTP/2 settings, header order, and browser behavioral signals. Python's requests has a TLS fingerprint that no real browser produces. It gets identified instantly, no matter what headers you send.

Does rotating User-Agents avoid detection?

No. Rotating User-Agents alone is nearly useless in 2026. Modern bot detection keys on TLS and HTTP/2 fingerprints, not the User-Agent string. You need a client whose TLS handshake matches a real browser, like curl_cffi or Camoufox.

Are residential proxies enough to avoid bans?

Residential proxies help with IP-based rate limits and geo-blocks. They don't fix a bad TLS fingerprint. The bot detection sees your fingerprint before it routes by IP. Combine a browser-accurate TLS client with proxies. Not proxies alone.

Does a headless browser still get detected?

Yes. A raw headless browser leaks several tells: navigator.webdriver set to true, an empty plugin list, a missing window.chrome, and a software WebGL renderer. Stealth patches (init scripts, Camoufox, Patchright) close most of these, but hardened detectors also score behavioral signals, so no browser tool is a guarantee.

What is the cf_clearance cookie?

It's the cookie Cloudflare sets after you pass a managed challenge. It proves a browser solved the proof-of-work challenge, and it's bound to your IP, User-Agent, and ASN, with a short TTL (often 15 to 60 minutes). Reuse it from the same IP and UA, or it's invalid.

Should I use a CAPTCHA-solving service?

Only when you have to. Services like 2Captcha and CapSolver cost money and add 5 to 30 seconds per solve, and they're a signal that something upstream failed. Fix the TLS fingerprint, IP reputation, and rate first — most CAPTCHAs never appear if those are right.

Keep reading

Found this useful? Cite it as: webscraping.space. “Bypassing anti-bot protections: TLS, fingerprints, and Cloudflare.” https://webscraping.space/blog/bypassing-anti-bot-protections. Published 2026-07-09.

Bypassing anti-bot protections: TLS, fingerprints, and Cloudflare

Frequently Asked Questions

Keep reading

CAPTCHA Solving for Web Scraping: The Honest 2026 Guide

What Actually Gets You Blocked When Web Scraping (2026)

Web Scraping Without Getting Blocked: The 2026 Anti-Ban Playbook