URL: https://webscraping.spaceBest viewed at 1024×768

Anti-Bot Published Aug 16, 2026 · 24 min read · 5,192 words

What Actually Gets You Blocked When Web Scraping (2026)

We run a web scraping API and see millions of requests a day. Here's what actually gets you blocked in 2026 — and what doesn't, signal by signal.

We run a web scraping API. Every day, millions of requests pass through our infrastructure — from a lone requests script someone wrote at 2am to stealthed browsers running in data centers on three continents. We see the full spectrum of what gets blocked and what doesn't, because when it goes wrong, we're the ones who get the support ticket.

Here's the honest version of what actually gets you blocked in 2026, in rough order of how often it's the real cause. It is not what the proxy marketing says. It is not what the anti-bot vendors say either. It's what the request logs say.

The short version, for people who want the answer before the argument: most blocks are dumb rate-limit triggers, not sophisticated fingerprinting. Rate and cadence cause roughly 45% of the blocks we see. IP reputation causes another 25%. TLS and HTTP/2 fingerprinting — the thing the entire anti-bot industry writes about — causes about 15%. The rest is behavior and headers. If you fix the boring stuff first, the expensive stuff mostly stops mattering.

Key takeaways

Most blocks are rate-limit triggers, not forensics. A naive requests script gets blocked after a median of about 40 requests on protected sites. The cause is usually a WAF counter, not an AI.
Fingerprinting is a multiplier, not a replacement. A browser-accurate TLS/HTTP-2 client lasts 10 to 50 times longer than a naive one — but only if the rate is human.
IP reputation is checked before your fingerprint. A flagged datacenter IP can be blocked before the site ever looks at how you speak TLS.
CAPTCHAs are the last gate, not the first. reCAPTCHA v3 and Turnstile score you invisibly; by the time you see a challenge, upstream signals already failed.
The cost of a block is time, not requests. One block can cost you thousands of pages you never got, and no proxy bill buys that back.
The fix order is: rate, IP, fingerprint, behavior. In that order. Most people do it backwards.

The honest truth: most blocks are dumb rate-limit triggers

The single most important thing I can tell you about getting blocked in 2026 is that it is usually not sophisticated. The image of an anti-bot system doing real-time TLS forensics on every request is mostly marketing. What actually happens, in the majority of cases, is that you trip a rate counter that a developer configured in ten minutes, and the site's edge returns 403 until the counter resets.

We see this in our own logs constantly. A customer points a naive requests script at a protected site, gets a 403 after about 40 requests, and assumes the site has deployed some terrifying fingerprinting AI. In reality, the site's WAF counted 40 requests in 90 seconds from one IP with a Python TLS fingerprint and a python-requests User-Agent, and applied a rule that says "this is not a browser, slow down or go away."

The numbers are stark. Across the protected sites we crawl daily, a naive requests script gets blocked after a median of about 40 requests. The same script with browser-accurate TLS and HTTP/2 fingerprinting — curl_cffi with impersonate="chrome" — lasts 10 to 50 times longer before the first block. A stealthed browser with human-like pacing can run for weeks on the same IP.

Fingerprinting is a 10 to 50x multiplier on top of behavior, not a replacement for it. A perfect fingerprint at 20 requests per second still gets blocked, because no human reads 20 pages a second.

The point of that chart is not that fingerprinting doesn't matter. It's that fingerprinting is a multiplier on top of behavior, not a replacement for it. A perfect fingerprint at 20 requests per second still gets blocked, because no human reads 20 pages a second. And a naive fingerprint at one request every three seconds often survives for a long time, because the site never bothers to look closely at a client that isn't causing load.

How a request actually gets scored

Before we go signal by signal, it helps to see the whole pipeline, because the order matters as much as the signals themselves. Every request that hits a protected site passes through roughly the same stages, and most requests die at the first or second stage, long before any fingerprint analysis runs.

The order matters: IP reputation is scored before the TLS fingerprint, and the fingerprint before behavior. Most requests die at stage 1 or 2, which is why fixing rate and IP reputation first is the highest-ROI move.

The order is the part people get wrong. IP reputation is checked before your TLS fingerprint, and your TLS fingerprint is checked before your behavior. That means a request from a flagged datacenter IP can be blocked before the site ever looks at how you speak TLS. It also means that if your rate is obviously inhuman, the site can block you on that alone, without ever running the expensive fingerprint checks.

This ordering is why the fixes stack the way they do. Fix the cheap, early signals first — IP reputation and rate — and you'll find that the later signals (TLS, HTTP/2, behavior) rarely get a chance to matter.

IP reputation: datacenter vs residential

IP reputation is the first gate, and it's the one most people misunderstand. The anti-bot industry has done an excellent job of convincing scrapers that every datacenter IP is radioactive and every residential IP is gold. The truth is more boring: IP reputation is a spectrum, and most of it is about who used the IP before you.

A datacenter IP from a major cloud provider is not automatically blocked. It's flagged — the ASN is known to host bots, so the site's risk score starts a few points higher — but a clean, quiet datacenter IP can scrape a huge amount of data before anything happens. The problem is that most datacenter IPs are not quiet. They're shared, and the previous tenant was probably also a scraper, possibly an abusive one. You inherit their history.

Residential IPs fix that by being boring. A residential IP looks like a home broadband connection, which is the most common and least suspicious thing on the internet. But residential proxies cost real money — roughly $4 per GB from Bright Data, $5 per GB from Oxylabs — and they're not magic. A residential IP with a naive requests fingerprint and a 20-requests-per-second cadence still gets blocked, because the behavior is still inhuman.

The honest rule of thumb: use a clean datacenter IP for most targets, and reach for residential only when the target specifically blocks datacenter ranges or when you need volume that a single IP can't hold. The complete proxy guide has the full tier-by-tier breakdown, including when residential is worth the money and when it's a waste of it.

There's a second, subtler point about IP reputation that most guides miss: warm-up. A fresh IP that immediately starts hammering a site is more suspicious than one that has been making polite requests for a few days. The reputation systems that track this are crude — they mostly count requests and errors per IP over time — but they exist, and they reward starting slow. If you're going to rotate IPs, rotate slowly and let each one warm up before you push it.

TLS fingerprinting: curl vs real browsers

TLS fingerprinting is real, and it's the most over-hyped signal in the whole space. Here's what actually happens: when your client opens a TLS connection, the exact sequence of messages it sends — the cipher suites it offers, the order it offers them in, the extensions it includes — forms a fingerprint. The classic version is JA3; the modern version is JA4. Python's requests (via urllib3 and OpenSSL) produces a TLS fingerprint that no browser on earth produces, and a site that checks TLS fingerprints can identify it in one handshake.

The same is true for HTTP/2. Browsers send a specific SETTINGS frame with specific values in a specific order, and they use a specific header compression scheme. Python's requests doesn't even speak HTTP/2 by default. A site that checks the HTTP/2 settings frame can tell "this is not a browser" before the first byte of the response. Header order matters too — browsers send headers in a fixed order, and a client that sends them in a different order is detectable even if every value is correct.

This is all real, and it's all fixable. The curl_cffi library impersonates a real Chrome TLS and HTTP/2 fingerprint at the C level, and it's the single highest-ROI fix for a Python scraper. The bypassing anti-bot protections post goes deep into the JA3/JA4 territory and the libraries that actually work, including Camoufox for the browser side.

But here's the honest part: TLS fingerprinting is rarely the first thing that gets you blocked. It's the thing that gets you blocked after your rate or your IP reputation has already made the site suspicious. A site that's deciding whether to challenge you will check your fingerprint; a site that's already decided to block you because you're hammering it at 10 requests per second doesn't need to. Fix the rate first, and the fingerprint matters less.

This is also where the "stealth" extensions and random-header tools fail. Rotating User-Agent strings does nothing in 2026, because detection reads the TLS handshake and the HTTP/2 settings frame, not the User-Agent. A tool that randomizes headers while leaving the TLS fingerprint untouched is rearranging deck chairs. The fingerprint is the handshake, and the handshake is the part you can't fake with a header.

Behavioral signals: rate, timing, mouse, scroll

Behavioral signals are the ones that actually separate humans from bots, and they're the ones most scrapers ignore. The biggest is request cadence. A human reading a site leaves gaps of 3 to 8 seconds between requests, and the gaps are irregular — sometimes you pause to read, sometimes you scroll, sometimes you get distracted. A naive scraper fires a request every 400 milliseconds, on a metronome, for hours. The inter-arrival time distribution is the single most distinguishing feature between a human session and a bot session, and every serious detection system models it.

The second behavioral signal is dwell time. A human spends tens of seconds looking at a page before the next request. A scraper downloads and moves on in 200 milliseconds. Combined with the gap between requests, dwell time is cheap for you to simulate and trivial for them to measure: just look at the distribution of time-on-page. Even a modest fake dwell of 1 to 3 seconds changes the distribution a lot.

The third is the stuff that only exists in a real browser: mouse movement, scroll events, focus changes, JS execution. A headless browser that never moves the mouse and never scrolls is detectable by that absence alone. This is where stealthed browsers like Camoufox earn their keep, and it's also where most scrapers don't need to go — because if your rate and cadence look human, the site rarely bothers to check whether you're moving a virtual mouse.

There's also the referer flow, which is cheap and underrated. A scraper requesting the product page directly with no referer and no history looks like a crawler even at a polite rate. Sending the right Referer that matches your crawl path — the listing page, then the product page it linked to — is a free signal that you're following the site's own structure.

The web scraping without getting blocked post is the full operations playbook for this layer: rate shaping, jitter, caching, ban detection, and recovery. It's the post I'd hand to anyone who's getting blocked and doesn't know why.

CAPTCHAs: the symptom, not the disease

CAPTCHAs are the most misunderstood part of the whole pipeline, and the 2026 landscape makes them worse. The classic image — a wobbly word, a grid of traffic lights — is a decade out of date. The market is now dominated by three families, and two of them are invisible.

reCAPTCHA v3 doesn't show a puzzle at all. It scores every page interaction from 0.0 to 1.0, and the site decides what to do with the score — typically blocking or challenging below 0.5, letting through above 0.7, and grey-zone in between. You cannot "solve" a v3 challenge, because there's nothing to solve. You can only make the score better by looking more like a human.

Cloudflare Turnstile is the fastest-growing type in 2026, because Cloudflare sits in front of a huge share of the web and Turnstile is free. It's non-interactive: the widget runs in the background, checks the browser environment and behavior, and issues a token in one to three seconds. There's no checkbox and usually no image grid, which makes it the most annoying type for scrapers, because the token is bound to a real browser environment.

hCaptcha is the main non-Google alternative, structurally similar to reCAPTCHA v2 — a checkbox that escalates to image grids — but the images are harder and the labeling is more varied. It's the one you'll hit on privacy-focused and EU sites that dropped Google.

Here's the thing that matters for this post: a CAPTCHA is never the first line of defense. It's the last gate in a pipeline that has already scored your request as suspicious. By the time you see a challenge, the detector has already flagged your IP, your fingerprint, or your behavior. Fix those and most challenges never appear.

The trigger rates tell the story: a naive client hits a challenge on about a third of requests, a fingerprinted client on under 10%, and a stealthed browser on residential on about 1%. The CAPTCHA is the meter, not the disease.

The trigger rates tell the story. A naive requests script hits a challenge on about 35% of requests to protected sites. A curl_cffi client with a real Chrome fingerprint drops that to about 8%. A stealthed browser on a residential IP sees challenges on about 1% of requests. The CAPTCHA isn't the problem — it's the meter that tells you how bad your upstream signals are. The captcha solving guide covers the full landscape, including when a solving service is worth the money (rarely) and when it isn't (almost always).

What actually matters, weighted

If you've read this far, you can probably guess the shape of the answer. Let me put numbers on it, because this is the chart I wish someone had shown me when I started.

Rate and cadence cause about 45% of blocks, IP reputation another 25%, and TLS/HTTP-2 fingerprinting only about 15%. The sophisticated stuff is a minority of the problem.

Across the block events we see in our fleet, rate and cadence are the root cause of roughly 45% of blocks. IP reputation is another 25%. TLS and HTTP/2 fingerprinting is 15%. Behavioral signals beyond rate — mouse, scroll, JS — are about 10%. Headers and miscellaneous is the last 5%.

Read that again, because it contradicts almost everything written about anti-bot evasion. The sophisticated stuff — fingerprinting, behavioral analysis, the arms race — accounts for a quarter of blocks. The boring stuff — requesting too fast, too regularly, from an IP with a bad history — accounts for 70%.

This is why the standard advice is backwards. Most people reach for residential proxies and CAPTCHA solvers first, because that's what the marketing sells. The people who actually survive start with a token bucket, a browser-accurate fingerprint, and a clean IP, and they don't need the expensive stuff until the target is genuinely hard.

The cost of getting blocked

Let me make the cost concrete, because "getting blocked" sounds like a minor inconvenience and it isn't. A block isn't a single failed request. It's a cascade: the request fails, your retry logic hammers the block, the block gets worse, you rotate the IP, the new IP has to warm up, and meanwhile your crawl is stalled and your pipeline is empty.

Same target, same single IP, first hour. The naive script gets blocked at about 40 pages and loses 28 minutes waiting for the block to expire; the polite client finishes the hour at 3,600 pages. The shaded gap is the real cost.

Here's the same target, same single IP, first hour. The naive script gets blocked at about 40 pages and loses 28 minutes waiting for the block to expire. The fingerprinted, polite client does one request per second and finishes the hour at 3,600 pages. The shaded area between the two lines is the cost of getting blocked — and it's not the 40 pages you got, it's the 3,500 you didn't.

That's the real math. A block costs you time, and time is the thing you can't buy back with a bigger proxy bill. The cheapest anti-block measure in existence is to not get blocked in the first place, and that's a behavior problem, not a procurement problem.

A working polite crawler

Let me put all of this into a crawler you can actually run. This is the shape of the thing we run in production — it's not the whole thing, but it's the skeleton, and it's honest about what matters:

import random
import time
from urllib.robotparser import RobotFileParser
from curl_cffi import requests

# 1. Respect robots.txt before anything else.
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
if not rp.can_fetch("*", "https://example.com/products"):
    raise SystemExit("robots.txt says no")

# 2. Browser-accurate TLS and HTTP/2 fingerprint.
session = requests.Session(impersonate="chrome")

# 3. Polite rate: 1 req/s sustained, irregular gaps, small bursts.
last_request = 0.0

def polite_get(url):
    global last_request
    gap = random.uniform(1.0, 3.0)
    time.sleep(max(0.0, gap - (time.monotonic() - last_request)))
    last_request = time.monotonic()
    return session.get(url, timeout=15)

# 4. Retry only what's transient; never retry a 403.
for attempt in range(4):
    r = polite_get("https://example.com/products?page=1")
    if r.status_code == 200:
        break
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", 2 ** attempt)))
    elif r.status_code >= 500:
        time.sleep(2 ** attempt + random.random())
    else:
        raise SystemExit(f"permanent refusal: {r.status_code}")

That's the whole strategy in about thirty lines: robots.txt respected, a real browser fingerprint, one request per second with irregular gaps, and retries that only touch transient errors. Run that against most protected sites and you will not get blocked. Run a naive requests script against the same sites and you'll be gone in 40 requests.

Two details are worth calling out. First, the sleep is gap - elapsed, not gap — if another thread already waited recently for this domain, we don't stack delays. Second, random.uniform(1.0, 3.0) produces a flat distribution, which is not exactly the human shape from the histogram, but it's close enough to defeat fixed-interval detection, which is the common case. If you're being specifically profiled, fit a lognormal distribution instead; for everyone else, uniform jitter is 90% of the value.

When to just use an API

There's a point where doing this yourself stops being worth it, and I want to be honest about where that point is, because we run a scraping API and the honest answer is "sometimes you should, sometimes you shouldn't."

Do it yourself when: the target is easy, the volume is low, and your time is cheap. A polite crawler with curl_cffi and a token bucket will handle a huge amount of the web for free.

Use a managed API when: the target is genuinely hard (Cloudflare, DataDome, heavy JS), the volume is large, the pages change shape constantly, or your time is worth more than the bill. A managed service handles rotation, fingerprint maintenance, and challenge-solving as a job, and it passes the liability problem along too.

And if the data you're after is really the search and retrieval side — you want answers, not HTML — that's a different problem entirely, and the complete AI search API list is the honest map of that territory.

FAQ

Why do I get blocked when scraping?

In 2026, most blocks are dumb rate-limit triggers, not sophisticated fingerprinting. A naive requests script gets blocked after roughly 40 requests on most protected sites because it fires too fast, too regularly, from an IP with a bad history. Rate and cadence cause about 45% of blocks, IP reputation another 25%, and TLS/HTTP-2 fingerprinting only about 15%. Fix the boring stuff first: slow down, add jitter, use a browser-accurate client, and use a clean IP.

Is TLS fingerprinting real?

Yes. The exact sequence of messages in your TLS handshake (JA3/JA4) and your HTTP/2 SETTINGS frame form a fingerprint, and Python's requests produces one no browser on earth produces. A site that checks fingerprints can identify it in one handshake. But it's rarely the first thing that gets you blocked — it's what confirms the suspicion your rate or IP reputation already created. curl_cffi with impersonate="chrome" fixes it in one line.

Do I need residential proxies?

Usually not. A clean, quiet datacenter IP can scrape a huge amount of data before anything happens; residential is only worth the $4-5 per GB when the target specifically blocks datacenter ranges or you need volume a single IP can't hold. A residential IP with a naive fingerprint and an inhuman cadence still gets blocked. Fix behavior first, then decide if the IP is actually the problem.

Why do I keep getting CAPTCHAs?

A CAPTCHA is a symptom, not the disease. reCAPTCHA v3 and Cloudflare Turnstile score you invisibly — there's nothing to solve, only a score to improve. By the time you see a challenge, the detector has already flagged your IP, fingerprint, or behavior. A naive requests script hits a challenge on about 35% of requests to protected sites; a curl_cffi client drops that to about 8%; a stealthed browser on residential sees about 1%.

How many requests before I get blocked?

It depends almost entirely on the client and the site. A naive requests script gets blocked after a median of about 40 requests on protected sites. The same script with browser-accurate TLS and HTTP/2 fingerprinting lasts 10 to 50 times longer. A stealthed browser with human-like pacing can run for weeks on the same IP. The multiplier is fingerprint plus behavior, not fingerprint alone.

Frequently Asked Questions

Why do I get blocked when scraping?

In 2026, most blocks are dumb rate-limit triggers, not sophisticated fingerprinting. A naive requests script gets blocked after roughly 40 requests on most protected sites because it fires too fast, too regularly, from an IP with a bad history. Rate and cadence cause about 45% of blocks, IP reputation another 25%, and TLS/HTTP-2 fingerprinting only about 15%. Fix the boring stuff first: slow down, add jitter, use a browser-accurate client, and use a clean IP.

Is TLS fingerprinting real?

Yes. The exact sequence of messages in your TLS handshake (JA3/JA4) and your HTTP/2 SETTINGS frame form a fingerprint, and Python's requests produces one no browser on earth produces. A site that checks fingerprints can identify it in one handshake. But it's rarely the first thing that gets you blocked — it's what confirms the suspicion your rate or IP reputation already created. curl_cffi with impersonate='chrome' fixes it in one line.

Do I need residential proxies?

Why do I keep getting CAPTCHAs?

A CAPTCHA is a symptom, not the disease. reCAPTCHA v3 and Cloudflare Turnstile score you invisibly — there's nothing to solve, only a score to improve. By the time you see a challenge, the detector has already flagged your IP, fingerprint, or behavior. A naive requests script hits a challenge on about 35% of requests to protected sites; a curl_cffi client drops that to about 8%; a stealthed browser on residential sees about 1%.

How many requests before I get blocked?

It depends almost entirely on the client and the site. A naive requests script gets blocked after a median of about 40 requests on protected sites. The same script with browser-accurate TLS and HTTP/2 fingerprinting lasts 10 to 50 times longer. A stealthed browser with human-like pacing can run for weeks on the same IP. The multiplier is fingerprint plus behavior, not fingerprint alone.

Keep reading

Found this useful? Cite it as: webscraping.space. “What Actually Gets You Blocked When Web Scraping (2026).” https://webscraping.space/blog/what-actually-gets-you-blocked. Published 2026-08-16.