Scaling Published Jul 12, 2026 · 2 min read · 508 words

Scraping at scale: queues, caching, and not getting banned

How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.

A scraper that works on ten pages is a script. A scraper that works on ten million pages is a system. The difference isn't parsing. It's scheduling, caching, and politeness. This guide is the architecture between "it ran once" and "it runs every night without getting banned."

The four problems of scale

1. Throughput without melting the target.
2. Reliability. One 500 shouldn't kill the crawl.
3. Politeness. You're a guest on someone else's server.
4. Dedup and resumability. Never request the same URL twice. Never lose progress.

Use a work queue

Never crawl with a bare for url in urls: loop. Put URLs in a queue and pull from it. A queue gives you resumability, bounded concurrency, and a place to put newly discovered URLs.

import queue, threading

work = queue.Queue()
for url in seed_urls:
    work.put(url)

seen = set()

def worker():
    while True:
        url = work.get()
        if url in seen:
            work.task_done()
            continue
        seen.add(url)
        try:
            for new_url in crawl(url):
                work.put(new_url)
        except Exception as e:
            log(e)
        finally:
            work.task_done()

threads = [threading.Thread(target=worker, daemon=True) for _ in range(4)]
for t in threads: t.start()
work.join()

For anything beyond a laptop, use a real queue. Redis, RabbitMQ, or a database-backed queue. Scrapy ships its own scheduler and is the easiest path for most teams.

Bounded concurrency

More concurrency is not more throughput. Past a point it's just more 429s and more load on a server that didn't ask to be crawled. Two rules:

• Cap concurrent connections per domain (2 to 5), not just globally.
• Cap requests per second per domain, with jitter.

# asyncio example: per-domain semaphore + rate limit
import asyncio, time

sem = asyncio.Semaphore(3)
last = {}

async def fetch_one(session, url, domain):
    async with sem:
        now = time.monotonic()
        wait = max(0, last.get(domain, 0) + 1.0 - now)  # 1 req/sec/domain
        await asyncio.sleep(wait)
        last[domain] = time.monotonic()
        async with session.get(url) as resp:
            return await resp.text()

Cache responses to disk

Caching is the highest-ROI thing you can do. It makes you faster, cheaper, and more polite all at once. You never re-request a page you already have.

import hashlib, pathlib, json

CACHE = pathlib.Path("cache")
CACHE.mkdir(exist_ok=True)

def cache_key(url):
    return hashlib.sha256(url.encode()).hexdigest()

def get_cached(url):
    p = CACHE / f"{cache_key(url)}.json"
    if p.exists():
        return json.loads(p.read_text())["body"]
    return None

def set_cached(url, body):
    p = CACHE / f"{cache_key(url)}.json"
    p.write_text(json.dumps({"url": url, "body": body}))

During development this is a lifesaver. You run the parser fifty times against the same cached HTML instead of hitting the site fifty times.

Retries with backoff

Networks fail. Retry, but retry well:

• Retry only on transient errors (timeouts, 5xx, 429). Never retry a 404.
• Use exponential backoff with jitter.
• Cap the attempt count.

async def fetch(session, url, attempts=4):
    for i in range(attempts):
        try:
            resp = await session.get(url)
            if resp.status == 200:
                return await resp.text()
            if resp.status in (404, 410):
                return None
            if resp.status == 429:
                wait = int(resp.headers.get("Retry-After", 2 ** i))
                await asyncio.sleep(wait)
                continue
            if resp.status >= 500:
                await asyncio.sleep(2 ** i + random.random())
                continue
            return None
        except (asyncio.TimeoutError, ClientError):
            await asyncio.sleep(2 ** i + random.random())
    return None

Dedup

Normalize URLs before you check seen:

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def normalize(url):
    s = urlsplit(url)
    scheme = s.scheme.lower()
    netloc = s.netloc.lower()
    if netloc.endswith(":80"): netloc = netloc[:-3]
    if netloc.endswith(":443"): netloc = netloc[:-4]
    query = urlencode(sorted(parse_qsl(s.query)))
    return urlunsplit((scheme, netloc, s.path.rstrip("/") or "/", query, ""))

For crawls bigger than memory, swap the set for a Bloom filter (pybloom_live) or a Redis set.

Be polite

• Set a per-domain crawl delay. 1 to 3 seconds is a reasonable default.
• Honor Retry-After and 429.
• Identify yourself with a custom User-Agent that includes a contact URL when scraping permitted targets.
• Respect robots.txt. Scrapy does this by default with ROBOTSTXT_OBEY = True.

Scrapy vs roll-your-own

Scrapy gives you the scheduler, dedup, retries, throttling, and item pipelines you'd otherwise write. And scrapy-playwright adds the browser layer when you need it.

Key takeaways

• Crawl from a queue, not a loop.
• Cap concurrency per domain. Add a rate limit with jitter.
• Cache every response. Never re-request.
• Retry transient errors with exponential backoff and jitter. Never retry 4xx.
• Normalize and dedup URLs. Use a Bloom filter when the set outgrows memory.
• Prefer Scrapy once you have a real crawl.

Found this useful? Cite it as: webscraping.space. “Scraping at scale: queues, caching, and not getting banned.” https://webscraping.space/blog/scraping-at-scale. Published 2026-07-12.