URL: https://webscraping.spaceBest viewed at 1024×768

Python Published Jun 28, 2026 · Updated Jul 10, 2026 · 35 min read · 7,664 words

Web scraping with Python requests: a practical starter

A hands-on guide to web scraping with Python's requests library: downloading pages, setting headers, handling redirects, parsing with BeautifulSoup, and avoiding the most common beginner mistakes.

Web scraping with requests is how most people start scraping in Python. It's small. It's fast. It handles the majority of pages that ship data in the initial HTML. And it's the layer underneath almost every other Python scraping stack — Scrapy's downloader, aiohttp wrappers, and most "scraper frameworks" are all doing what requests does, just with more machinery around it.

This guide is the requests deep dive. It walks through the library the way a production engineer actually uses it: the request lifecycle, headers, sessions, timeouts, redirects, proxies, TLS, retries, streaming, compression, encoding, and error handling. There's a complete scraper at the end that ties it all together. If you want the from-scratch full walkthrough — picking a target, reading robots.txt, pagination, saving to SQLite — that's the companion post. This one owns the library itself.

A note on scope before we start. requests fetches HTML. It does not run JavaScript. If the data you need is injected by a script after the page loads, requests will hand you an empty <div id="root"> and nothing else. That's not a bug in requests; it's a boundary. Knowing where that boundary is — and what to do on each side of it — is half of what this guide is about.

What you need

Two packages. Nothing else.

pip install requests beautifulsoup4

requests downloads the page. BeautifulSoup turns the HTML into something you can query. The rest is standard-library Python.

If you're on a fresh machine, create a virtualenv first: python3 -m venv venv && source venv/bin/activate on Linux and macOS, venv\Scripts\activate on Windows. Then run the pip install above. You want requests 2.32 or newer — the 2.x line has been stable for years, and the API this guide uses hasn't changed since 2.0.

The request lifecycle

Before you write a single scraper, it's worth knowing what requests.get(url) actually does, because every knob in this guide — headers, sessions, timeouts, proxies, retries — is a lever on one of these steps.

Resolve the hostname. The URL's host is looked up via DNS. This is a network round-trip you don't control, and it's why the first request to a domain is always slower than the ones after it.
Open a connection. requests opens a TCP socket to the resolved IP, on port 80 for http or 443 for https.
TLS handshake (https only). For https URLs, client and server negotiate encryption. This is several round-trips of cryptographic negotiation, and it's where the TLS fingerprint problem lives (more on that later).
Send the request. The method, path, headers, and body are serialized and written to the socket.
Wait for the response. The server processes and responds. The time between "request sent" and "first byte received" is time-to-first-byte (TTFB).
Read the body. The response headers arrive first, then the body streams in.
Close or reuse the connection. If you're using a Session, the connection goes back into the pool for reuse. If you called the module-level requests.get(), it's torn down.

Each of these steps is a place where things fail, and each has a corresponding knob in requests. DNS can hang (timeouts). The connection can be refused (ConnectionError). The TLS handshake can be fingerprinted (403s). The server can stall after accepting (read timeout). The connection can die mid-body (ChunkedEncodingError). The rest of this guide is, in a sense, just the map of those failure points and what to do about each one.

GET, POST, and params

The two verbs you'll use 99% of the time are GET and POST. GET fetches a resource; POST submits data. requests makes both trivial:

import requests

# GET with query parameters
params = {"q": "python requests", "page": 2, "sort": "recent"}
resp = requests.get("https://example.com/search", params=params, timeout=10)
print(resp.url)  # https://example.com/search?q=python+requests&page=2&sort=recent

# POST with form data
data = {"username": "alice", "password": "hunter2"}
resp = requests.post("https://example.com/login", data=data, timeout=10)

Two things are worth noticing here. First, params does the URL-encoding for you — spaces become + or %20, special characters get escaped, and the query string is built correctly. If you hand-build query strings with f-strings, you will eventually produce a URL that breaks on an ampersand or a non-ASCII character. Let requests do it. Second, data= sends application/x-www-form-urlencoded (form encoding), while json= sends application/json and serializes a dict for you. For a login form, use data. For a JSON API, use json=.

For scraping, GET is the workhorse. POST matters when you're submitting search forms, paginating through a POST-based API, or logging in. One rule that saves time: if a site's search form uses GET (check the Network tab in DevTools), you can often just build the URL with params and skip the form entirely.

Headers are not optional

A bare requests.get() sends a User-Agent like python-requests/2.32.0. A lot of servers reject that on sight. Send a real browser User-Agent and a Referer:

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Referer": "https://www.google.com/",
}

resp = requests.get(url, headers=HEADERS, timeout=10)

A Referer of https://www.google.com/ makes the request look like a click from search. Many sites treat that better than a direct hit.

Why do these three headers matter so much?

User-Agent is the first thing a server's bot detection reads. python-requests/2.32.0 is a neon sign. A real Chrome UA is the difference between "looks like a browser" and "looks like a script" at the cheapest possible cost.
Accept-Language is the header that most scrapers forget, and it's the one that leaks. A browser always sends it; a default requests call doesn't. A server that sees a Chrome UA but no Accept-Language knows something is off. It also controls which language variant of a page you get — scrape without it and you may get the default locale instead of the one you want.
Accept tells the server what content types you can handle. Browsers send a long, specific list. The default requests value is */*, which is technically valid but reads as "I'll take anything," which is what a script says.

There's a subtlety about header order that bites people who copy browser headers from DevTools. HTTP header order is not semantically meaningful — servers are not supposed to care, and most don't. But some bot-detection systems fingerprint the exact byte sequence of your request, including header order, and a request that has the right headers in the wrong order can still score as suspicious. The practical takeaway: don't obsess over order, but do send the full set of headers a browser sends, in roughly the order a browser sends them. If a site is still suspicious after you've set User-Agent, Accept, Accept-Language, and Referer, the problem is almost never header order — it's the TLS fingerprint, which we'll get to.

One more header habit: set headers once on a Session, not per request. You'll see why in the next section.

Sessions and connection pooling

Every requests.get() opens a new connection. For a scraper that hits dozens of pages on one domain, that's wasteful and noisy. A Session reuses connections and keeps cookies. Faster and less conspicuous:

session = requests.Session()
session.headers.update(HEADERS)

# Fake "I arrived from search" hit
session.get("https://example.com/", timeout=10)

# Now crawl
for path in ["/page1", "/page2", "/page3"]:
    resp = session.get(f"https://example.com{path}", timeout=10)
    resp.raise_for_status()
    process(resp.text)

The difference between module-level requests.get() and session.get() is the difference between renting a car for every errand and owning one. The module-level functions create a fresh Session, do the request, and throw it away — connection, TLS state, cookies, and all. A Session you create yourself keeps all of that alive.

The connection reuse is the big win. Opening a TCP connection and doing a TLS handshake costs several round-trips — on a typical connection that's two to four RTTs of pure overhead before your request even leaves. With keep-alive, the second request on the same connection skips all of that. On a crawl of a hundred pages on one domain, that's the difference between a hundred handshakes and one. The chart below shows the cumulative effect on a typical 300-millisecond round-trip connection:

Fetching 100 pages on one domain: a fresh connection per request costs roughly 45 seconds of handshake overhead, while one Session with keep-alive finishes in about 15 seconds. The gap only widens as the crawl grows.

The cookie persistence is the second win. A Session stores cookies from every response and sends them back on subsequent requests. That's how login works (more in the cookies section), and it's also how a site's "set a cookie, then check for it" bot detection works — a fresh connection that doesn't send back the cookie it was just given looks like a bot.

A Session also gives you a place to configure things once: headers, proxies, auth, a default timeout. Set them on the session and every request inherits them. That's not just convenience; it's consistency, and consistency is what makes your traffic look like one client instead of a swarm.

One caveat: a Session is not thread-safe. If you're going to use threads (and you will, eventually), give each thread its own Session, or use a lock. Sharing one Session across threads produces corrupted requests and confusing bugs. The scaling guide covers this properly.

Timeouts: the most skipped argument

The single most common bug in beginner scrapers is omitting timeout. Here's what happens: requests.get(url) with no timeout waits forever. Not "a long time" — forever. If the server accepts your connection and then stalls, your script hangs on that line indefinitely, and a scraper that hangs on page 47 of 10,000 is a scraper that never finishes.

# Connect timeout: 3s to establish the connection.
# Read timeout: 10s between bytes of the response.
resp = session.get(url, timeout=(3, 10))

The tuple form is the one to use. The first number is the connect timeout — how long to wait for the TCP connection to be established. The second is the read timeout — how long to wait between chunks of the response body. A page that starts streaming and then stalls mid-body will trip the read timeout; a page that never accepts the connection will trip the connect timeout.

Why two numbers? Because they fail for different reasons. A connect timeout fires when the server is down, the IP is unreachable, or the port is filtered. A read timeout fires when the server accepted your request but is taking too long to produce the body — a slow query, a busy worker, a server that's throttling you. The fix for each is different: a connect timeout means "this host is unreachable, move on"; a read timeout means "this request is too slow, retry or skip."

With timeout=(3, 10), a healthy page finishes in under a second, a slow-but-fine page squeaks through, a stalled server trips the read timeout at 10 seconds, and an unreachable host trips the connect timeout at 3. Without a timeout, the stalled-server case hangs forever.

What values should you use? For scraping, (3, 10) is a sane default: three seconds to connect, ten seconds between reads. If a page takes longer than ten seconds to produce its next byte, it's not going to finish in a useful time anyway. For large file downloads, raise the read timeout — a slow-but-steady stream of a big file can legitimately take longer than ten seconds between chunks. The point of a timeout is not to be generous; it's to fail loudly instead of hanging silently.

A note on retry semantics: a timeout is not a retry. requests does not retry anything by default. If you want a timed-out request to be retried, you have to build that yourself — which is exactly what the retries section does. And when you do retry, remember that a read timeout on a POST is ambiguous: the server may have processed your request and just been slow to answer. Retrying a POST can double-submit. For scraping, where almost everything is a GET, this matters less, but it's worth knowing.

Redirects: follow them, but know where you are

requests follows redirects by default. That's usually what you want — a URL that 301s to a canonical location should just work. But "follow by default" hides information you sometimes need.

resp = session.get(url, timeout=10, allow_redirects=True)
print(resp.url)          # the FINAL url after all redirects
print(resp.history)      # list of Response objects for each hop
print(resp.status_code)  # status of the FINAL response

resp.url is the URL you actually ended up on. If you're scraping a site that redirects /product/123 to /product/123/slug-here, the final URL is the canonical one — and that's the URL you should store, because it's the one that won't change. resp.history is the chain of redirects, each with its own status code and headers. The Location header on the last hop tells you where you ended up.

Two redirect behaviors to know. First, allow_redirects=False gives you the raw 301/302 response without following it. That's useful when you want to see the redirect target yourself, or when a site uses a redirect as a signal — some sites redirect bots to a challenge page, and seeing the redirect is how you detect it. Second, requests follows a maximum of 30 redirects by default, then raises TooManyRedirects. A redirect loop — /a to /b to /a — will hit that limit and raise. Catch it, log the URL, move on.

The redirect-related bug that bites most people: a 301/302 that lands on a login page. You scrape a URL, it redirects to /login?next=..., and you parse the login page as if it were the content you wanted. The fix is to check resp.url after the request and verify it's still on the domain and path you expect. If a site starts redirecting you to a challenge or login page, that's a signal — stop and look at what changed.

Proxies

A proxy is a middleman. Your request goes to the proxy, and the proxy forwards it to the target. The target sees the proxy's IP, not yours. That's the entire point: when your IP gets rate-limited or blocked, you switch to a different proxy and keep going.

# Per-request
resp = session.get(url, proxies={"http": "http://proxy.example.com:8080",
                                "https": "http://proxy.example.com:8080"},
                   timeout=10)

# Per-session (set once, applies to every request)
session.proxies.update({"http": "http://proxy.example.com:8080",
                        "https": "http://proxy.example.com:8080"})

The dict maps scheme to proxy URL. You usually want both http and https entries, because an https URL goes through the https entry. A common beginner mistake is setting only http and wondering why https requests still come from your IP.

Proxies come in two flavors for scraping. Datacenter proxies are cheap, fast, and hosted in data centers — they work fine for sites that don't check IP reputation. Residential proxies route through real home IPs and are much harder to block, but they cost more and are slower. For a beginner, the honest advice is: you probably don't need proxies yet. On a sandbox site at one or two requests per second with a real User-Agent, you will not be blocked. Add proxies when a legitimate target is rate-limiting you and politeness has failed.

When you do need them, rotation is the pattern: a pool of proxies, and each request (or each N requests) uses a different one. The simplest rotation is a list and an index:

PROXIES = [
    {"http": "http://p1.example.com:8080", "https": "http://p1.example.com:8080"},
    {"http": "http://p2.example.com:8080", "https": "http://p2.example.com:8080"},
    {"http": "http://p3.example.com:8080", "https": "http://p3.example.com:8080"},
]

for i, url in enumerate(urls):
    proxy = PROXIES[i % len(PROXIES)]
    resp = session.get(url, proxies=proxy, timeout=10)

That's a round-robin, and it's fine for a few proxies. Real rotation services handle the harder parts: health-checking proxies, dropping dead ones, and making sure a session's cookies stay on the same IP (some sites log you out if your IP changes mid-session).

Two proxy gotchas. First, proxy environment variables: requests honors HTTP_PROXY and HTTPS_PROXY from the environment, and if you've ever set those globally (or a corporate proxy set them for you), your "direct" requests are actually going through a proxy without you knowing. session.trust_env = False disables that. Second, proxies and TLS: when you use an https proxy, the proxy sees the domain you're connecting to (via the CONNECT method) even though it can't read the encrypted traffic. If you need the proxy to not know the target, that's a different, much harder problem.

TLS: why requests gets 403 and a browser doesn't

This is the section that saves people days. You set the perfect headers. You use a Session. You're polite. And the site still returns 403 on every request. Meanwhile, the same URL opens fine in your browser. What's different?

The answer is the TLS fingerprint. When a client connects over https, the first thing it sends is a TLS ClientHello — a message that negotiates the encryption. That message contains a list of cipher suites, TLS versions, extensions, and their order. Every TLS implementation produces a slightly different ClientHello, and the exact shape of yours is a fingerprint. requests uses Python's ssl module, which produces a ClientHello that is instantly recognizable as "not a browser." A server running bot detection can look at the ClientHello alone and decide to block you before you've sent a single HTTP header.

This is why the "add headers" fix stops working at some point. Headers are easy to fake; the TLS handshake is not. A server that fingerprints TLS will block requests no matter how perfect your User-Agent is. The chart below is illustrative of what I've seen on real fingerprinting targets — the same URL, the same headers, different clients:

On a TLS-fingerprinting site, headers and a Session barely move the needle — the 403 rate stays high because the block happens at the handshake, before any HTTP header is seen. curl_cffi impersonating Chrome's ClientHello drops the rate to near zero.

What are your options, in order of increasing effort?

Check whether the site actually fingerprints TLS. If a plain requests call with good headers works, it doesn't. Most sites don't fingerprint TLS; the ones that do are usually the ones worth scraping.
Use curl_cffi. This is the modern fix. curl_cffi bundles a patched version of curl that can impersonate a browser's TLS handshake — Chrome, Safari, Firefox — at the ClientHello level. It's a drop-in replacement for requests:

from curl_cffi import requests as crequests

resp = crequests.get(url, impersonate="chrome", timeout=10)
print(resp.status_code)  # 200, where requests got 403

The impersonate="chrome" argument makes the TLS handshake look like a real Chrome browser. Same API as requests — get, post, Session, params, headers — so the migration is usually a one-line import change. This is the first thing to try when headers fail. 3. Use a headless browser. Playwright or Puppeteer runs a real browser, which has a real TLS stack and real JavaScript. This defeats TLS fingerprinting completely, but it costs roughly 10x the CPU and memory of an HTTP client, and it's far easier to detect as a browser automation tool. Use it when you need JavaScript rendering anyway, not just to fix a 403.

The honest framing: TLS fingerprinting is an arms race, and requests is on the losing side of it by default. For the majority of sites, requests with good headers is fine. For the sites that matter — the ones with real bot detection — you'll want curl_cffi in your toolkit. It's the difference between "requests gets 403" and "the same code gets 200."

Retries: urllib3 Retry and HTTPAdapter

requests doesn't retry. If a request fails, it fails, and the exception propagates to you. For a scraper, that's the wrong default: networks are flaky, servers have bad days, and a single dropped connection shouldn't kill a run. The fix is to configure retries at the urllib3 layer, which is what actually does the networking under requests.

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,                      # max 5 retries
    backoff_factor=1,             # sleep 1s, 2s, 4s, 8s between attempts
    status_forcelist=(429, 500, 502, 503, 504),
    allowed_methods=("GET",),     # never auto-retry POSTs
    respect_retry_after_header=True,
)

adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)

Let me unpack each argument, because this is where the real decisions live.

total is the maximum number of retries. Five is a reasonable ceiling — beyond that, the request is probably failing for a reason that retrying won't fix.
backoff_factor controls the sleep between attempts. urllib3 sleeps backoff_factor * (2 ** (attempt - 1)) seconds: with a factor of 1, that's 1s, 2s, 4s, 8s. This is exponential backoff, and it's the polite way to retry — you hammer less as you fail more.
status_forcelist is the list of status codes that trigger a retry. The critical decision: retry only transient failures. 429 (rate limited), 500, 502, 503, 504 (server hiccups) are transient. 404, 403, 401, 400 are permanent — retrying them is hostile and pointless. This is the single most important thing to get right in a retry policy.
allowed_methods — never auto-retry POSTs. A POST that times out may have been processed; retrying it can double-submit. For scraping, where almost everything is a GET, this is mostly about not shooting yourself in the foot later.
respect_retry_after_header=True — when the server sends Retry-After with a 429, urllib3 honors it instead of following the backoff schedule. The server is telling you exactly when to come back; listen to it.

The urllib3 backoff schedule with backoff_factor=1: the sleep doubles each attempt — 1s, 2s, 4s, 8s, 16s. The point of the doubling is that you hammer less as you fail more, giving a struggling server room to recover.

The session.mount() calls are how you attach the adapter. mount("https://", adapter) means "use this adapter for all https requests." You can mount different adapters for different hosts — a strict one for a fragile site, a lenient one for a robust one.

One thing to know: HTTPAdapter retries connection errors and the status codes in status_forcelist, but it does not retry read timeouts by default. If you want read timeouts retried, add ReadTimeout to the Retry object's retry_on_exception — or handle it in your own loop. And when retries are exhausted, urllib3 raises RetryError, which you'll catch in the error-handling section.

Rate limiting and politeness

The highest-leverage thing you can do is slow down. A scraper that hammers a server is the scraper that gets banned first.

import time
import random

for path in paths:
    resp = session.get(f"https://example.com{path}", timeout=10)
    process(resp.text)
    time.sleep(random.uniform(1.5, 3.5))  # jittered delay

Jitter matters. A fixed 2-second interval is obviously a bot. A random interval between 1.5 and 3.5 seconds looks human. It also spreads your load.

The politeness rules I actually follow:

One to three seconds between requests to the same domain. If the site's robots.txt declares a Crawl-delay, use that instead. This is the single politeness knob that matters most, and it's the one people skip because it feels slow. It's supposed to feel slow. That's the point.
Honor 429 and Retry-After. A site telling you to slow down is a gift. Treat it as one.
Respect robots.txt. It's the site's statement about what automated crawlers may access. The urllib.robotparser module parses it in a few lines, and there's rarely a good reason to ignore it. The legal and ethical line-work is worth reading in the ethics and robots.txt guide before you point anything at a site you don't own.
Identify yourself. Set a User-Agent that includes a contact URL when you're scraping permitted targets, so an operator who sees your traffic can reach you instead of blocking you.
If the site offers an API or a data dump, use that instead. Always. No amount of politeness beats not crawling at all.

The "one fast human" ceiling is the number I design against: a fast human skimming a page takes two to five seconds per page and doesn't do it for eight hours straight. If your crawler runs at one request per second per domain, you're already faster than that human over the long haul. Sites that run bot defense build their traffic models from aggregate behavior — requests per second per IP, the ratio of HTML to assets, the shape of the request timeline — and the polite crawler sits comfortably inside those models.

Some data lives behind a login. requests handles this in two ways, and knowing which one you need is most of the battle.

Session cookies. If you log in through a form, the server sets a session cookie, and a Session stores it and sends it back automatically:

session = requests.Session()
session.headers.update(HEADERS)

# 1. GET the login page (some sites set a CSRF token cookie here)
session.get("https://example.com/login", timeout=10)

# 2. POST the credentials
resp = session.post(
    "https://example.com/login",
    data={"username": "alice", "password": "hunter2"},
    timeout=10,
)

# 3. Now the session carries the auth cookie; scrape away
resp = session.get("https://example.com/account", timeout=10)

The key line is the last one: because the login POST's response set a cookie, and the Session stored it, the subsequent GET to /account sends that cookie automatically. This is the same mechanism as a browser's "stay logged in" — the cookie is the credential, and the Session is the cookie jar.

Manual Cookie headers. Sometimes you already have a cookie value — from your own browser, from a previous run, from a response you inspected. You can set it directly:

session.headers["Cookie"] = "sessionid=abc123; csrftoken=xyz789"

This works, but it's fragile: the cookie header is a single string, and if the server sets new cookies, your manual header and the session's jar can disagree. The cleaner way is session.cookies.set("sessionid", "abc123"), which puts the cookie in the jar where the Session manages it properly.

The honest advice about login scraping: if the site has real authentication, consider whether you should be scraping it at all. Scraping behind a login is where the legal and ethical lines get sharp — the terms of service almost always cover it, and the data behind a login is usually not "public data." If you're scraping your own account on a service you use, that's a different story, but be careful. And if the login involves CAPTCHA, two-factor auth, or a JavaScript-rendered login form, requests is the wrong tool — that's when you reach for a browser, because you're no longer doing HTTP, you're doing browser automation.

Streaming large files

resp.text and resp.content load the entire response into memory. For a 50 KB HTML page, that's nothing. For a 2 GB file, that's a problem — your scraper will use 2 GB of RAM to download something it could have written to disk in chunks.

with session.get(url, stream=True, timeout=(3, 30)) as resp:
    resp.raise_for_status()
    with open("big_file.bin", "wb") as f:
        for chunk in resp.iter_content(chunk_size=65536):
            f.write(chunk)

Two things are going on. First, stream=True tells requests not to read the whole body into memory — the response is read lazily, chunk by chunk. Second, iter_content(chunk_size=65536) yields the body in 64 KB pieces, and each piece is written to disk immediately. Peak memory is one chunk, not the whole file.

A few details worth knowing. iter_content is the right method for binary downloads; iter_lines is for text files where you want to process line by line (logs, CSVs, JSONL). The chunk size is a tradeoff: too small (1 KB) and you spend all your time on I/O overhead; too large (10 MB) and you're back to using lots of memory. 64 KB is a good default. And when you use stream=True, you must either consume the body or close the response — with handles that for you, which is why the example uses it. A streamed response that's never read and never closed leaks the connection back to the pool in a broken state.

Download time scales linearly with size on a log-log plot. A 100 MB file takes 80 seconds at 10 Mbps and 800 seconds at 1 Mbps — which is why streaming to disk matters, and why compression (shrinking the wire size) is worth more than any other optimization on a big crawl.

For scraping, streaming matters in two cases: downloading files (PDFs, images, datasets) and scraping pages that are unexpectedly huge. A page that's supposed to be 50 KB but is actually 50 MB is a signal — either it's a data dump you should handle differently, or something is wrong. Streaming with a size check catches that:

with session.get(url, stream=True, timeout=(3, 30)) as resp:
    resp.raise_for_status()
    total = int(resp.headers.get("Content-Length", 0))
    if total > 10_000_000:  # 10 MB
        raise ValueError(f"page too large: {total} bytes")
    data = resp.content

Compression

Most servers compress responses. The two encodings you'll see are gzip and brotli (br). requests handles gzip and deflate automatically — it sends Accept-Encoding: gzip, deflate by default and decompresses the response transparently. resp.content is always the decompressed bytes; you never see the compressed stream.

Brotli is the wrinkle. Modern browsers send Accept-Encoding: gzip, deflate, br, and many servers will happily serve brotli to a client that asks for it. But requests does not decompress brotli by default — if you send br in Accept-Encoding and the server responds with a brotli body, resp.content will be compressed garbage. The fix is to install the brotli package (pip install brotli), after which requests decompresses it automatically.

The practical rule: don't send br in Accept-Encoding unless you've installed brotli. If you copy a browser's full header set including br and you haven't installed the package, you'll get unreadable bodies. Either install brotli or drop br from the header. And if you're scraping a site that serves huge pages, compression is your friend — a 200 KB HTML page often compresses to 30 KB on the wire, which is the difference between a fast crawl and a slow one.

One more compression note: Content-Encoding is about the transfer, not the content. A server can send Content-Encoding: gzip (compressed on the wire, decompressed by requests) and Content-Type: text/html (the actual format). Don't confuse the two. The first is handled for you; the second is what you parse.

Encoding: the .encoding dance

HTML pages declare their character encoding in a <meta charset> tag or in the Content-Type header. When they disagree — or when the header is missing — you get mojibake: cafÃ© instead of café, â€™ instead of '.

requests handles this with a two-step process. resp.encoding is the encoding requests guessed from the headers. resp.text decodes the body using resp.encoding. If the guess is wrong, you get garbage — and the fix is to set resp.encoding before reading .text:

resp = session.get(url, timeout=10)
if resp.encoding is None or resp.encoding.lower() not in ("utf-8", "utf8"):
    resp.encoding = resp.apparent_encoding
text = resp.text

resp.apparent_encoding is requests' best guess from the actual bytes, using the charset_normalizer library. It's slower than trusting the header, but it's usually right. The dance is: check what encoding requests thinks it is, and if it's not what the page actually uses, override it before you read .text.

The bug that bites people: reading resp.text once, getting mojibake, and then setting resp.encoding — too late, because .text is cached after the first read. Set resp.encoding before the first .text access. If you've already read it, you need a fresh response or resp.content.decode(resp.apparent_encoding).

For most modern sites, UTF-8 is the answer and you'll never touch this. But legacy sites, and sites that serve different encodings per locale, will hand you mojibake at some point, and knowing the dance is the difference between a five-second fix and an hour of confusion.

JSON vs HTML

requests doesn't care what the body is. It's bytes. The question is what you do with them, and the answer splits cleanly in two.

JSON. If the response is JSON — a modern API, or a site that embeds its data as a JSON blob — use resp.json():

resp = session.get("https://api.example.com/items", timeout=10)
data = resp.json()  # raises requests.exceptions.JSONDecodeError on bad JSON
for item in data["items"]:
    print(item["title"], item["price"])

resp.json() parses the body as JSON and returns a Python object. It raises JSONDecodeError if the body isn't valid JSON — which is how you detect that an API changed shape or that you're actually looking at an HTML error page. A common pattern: check resp.headers.get("Content-Type") starts with application/json before calling .json(), so a surprise HTML page fails loudly instead of confusingly.

HTML. If the response is HTML, parse it with BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(resp.text, "html.parser")
cards = soup.select("article.product-card")

The choice between html.parser and lxml as the parser is worth one sentence: lxml is faster and more forgiving of broken HTML, but it's a compiled dependency. html.parser is built in and fine for most jobs. Install lxml when parsing speed matters or when a page's broken markup trips up the built-in parser.

The deeper point: a lot of "scraping" is really finding the JSON. Modern sites increasingly ship their data as a JSON blob inside the HTML — a <script type="application/json"> tag, or a window.__DATA__ variable. When that's the case, you don't parse HTML at all; you extract the JSON and parse that. It's faster, more robust, and less likely to break when the site's CSS classes change. The pattern:

import json, re

match = re.search(r'<script type="application/json"[^>]*>(.*?)</script>',
                  resp.text, re.S)
if match:
    data = json.loads(match.group(1))

This is the single most underrated technique in scraping. Before you write a BeautifulSoup selector for a field, check whether the page already contains the data as JSON. If it does, you've just made your scraper dramatically more robust.

Error handling

A scraper that crashes on the first error is a scraper that doesn't finish. The discipline is: catch the right exceptions, at the right level, and decide what each one means.

import requests
from requests.exceptions import (ConnectionError, ReadTimeout,
                                 RetryError, TooManyRedirects)

def fetch(session, url):
    try:
        resp = session.get(url, timeout=(3, 10))
    except (ConnectionError, ReadTimeout) as exc:
        log.warning("transient failure %s: %s", url, exc)
        return None          # retryable; caller decides
    except TooManyRedirects:
        log.warning("redirect loop %s", url)
        return None
    except RetryError:
        log.warning("retries exhausted %s", url)
        return None
    if resp.status_code == 429:
        retry_after = int(resp.headers.get("Retry-After", "30"))
        time.sleep(retry_after)
        return fetch(session, url)
    if resp.status_code in (404, 410):
        return None
    resp.raise_for_status()
    return resp.text

The exception hierarchy is worth knowing, because catching the right one is the difference between handling a real failure and swallowing everything:

requests.RequestException is the base class for all requests errors. Catching it catches everything — which is sometimes what you want at the top level, and sometimes a way to hide bugs.
ConnectionError means the connection couldn't be established — DNS failure, refused connection, network down. Usually transient; retry.
ReadTimeout and ConnectTimeout are the timeout exceptions. Transient; retry with backoff.
TooManyRedirects means the redirect chain exceeded 30 hops. Almost always a bug or a hostile site; log and move on.
RetryError means urllib3's retry policy was exhausted. The request failed all its attempts; record the URL for a later pass.
JSONDecodeError (from resp.json()) means the body wasn't JSON. Often means the site changed or you're looking at an error page.

The pattern above is the one I use everywhere: catch the transient failures and return None, let the caller decide whether to retry; handle 429 explicitly with Retry-After; treat 404/410 as "gone, record and move on"; and let raise_for_status() handle the rest. A crawl that finishes with a list of a hundred URLs that failed is infinitely more valuable than a crawl that crashes on the hundredth failure.

The status-code table is the map:

Status code	Meaning	Scraper action
200	OK	Parse it.
301 / 302	Redirect	Follow it; store `resp.url` as the canonical URL.
401 / 403	Unauthorized / Forbidden	Headers, auth, or bot detection. Don't retry; fix the cause.
404 / 410	Gone	Record and move on.
408	Request timeout	Retry with backoff.
429	Too many requests	Honor Retry-After, then back off.
5xx	Server error	Transient; retry with backoff.

A complete scraper

Here's everything tied together: a polite, resumable scraper that fetches a list of product pages, parses them with BeautifulSoup, cleans the data, and writes a CSV.

import csv, random, time
import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/126.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

def make_session():
    s = requests.Session()
    s.headers.update(HEADERS)
    retry = Retry(total=4, backoff_factor=1,
                  status_forcelist=(429, 500, 502, 503, 504),
                  allowed_methods=("GET",))
    adapter = HTTPAdapter(max_retries=retry)
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    return s

def parse_product(html):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.select_one("h1.product-title")
    price = soup.select_one(".price")
    if not title or not price:
        return None
    return {
        "title": title.get_text(strip=True),
        "price": price.get_text(strip=True).replace("$", ""),
    }

def main():
    session = make_session()
    urls = [f"https://example.com/products?page={i}" for i in range(1, 11)]
    with open("products.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["url", "title", "price"])
        writer.writeheader()
        for url in urls:
            try:
                resp = session.get(url, timeout=(3, 10))
                if resp.status_code != 200:
                    continue
                record = parse_product(resp.text)
                if record:
                    writer.writerow({"url": resp.url, **record})
            except requests.RequestException as exc:
                print(f"failed {url}: {exc}")
            time.sleep(random.uniform(1.5, 3.5))

if __name__ == "__main__":
    main()

Every piece of this guide is in that file: realistic headers, a Session, retries with backoff, timeouts, status-code checks, null-checked selectors, get_text(strip=True) for clean text, CSV output, and a jittered delay. That's 90% of a production scraper. The other 10% — caching, queues, dedup, monitoring — is what the scaling guide covers.

Key takeaways

Send realistic headers. User-Agent, Accept, and Accept-Language matter; Referer helps.
Reuse a Session. Don't open a new connection per request.
Always set a timeout — (3, 10) is a sane default. No timeout means a hang forever.
Follow redirects, but check resp.url and resp.history so you know where you ended up.
Retry only transient failures (429, 5xx, timeouts) with exponential backoff. Never retry 404, 403, 401.
If headers don't fix a 403, the problem is the TLS fingerprint — try curl_cffi.
Jitter your delays. Politeness is a defense, not just courtesy.
Handle 429 and Retry-After explicitly.
Stream large downloads with iter_content; don't load 2 GB into memory.
Set resp.encoding before reading .text when the charset is wrong.
Null-check every selector before you use it.
Only reach for a browser when the data is genuinely not in the HTML.

Frequently Asked Questions

Is web scraping with Python requests legal?

Scraping public data is usually legal, but it depends on what you scrape and where you live. Check the site's robots.txt and Terms of Service. Don't scrape behind a login or paywall. Don't grab personal data you don't need. Use common sense. This blog's ethics guide goes deeper.

Do I need a browser to scrape a website?

No. If the data is in the HTML, plain requests is faster, cheaper, and harder to detect than a headless browser. You only need a browser when the content shows up after JavaScript runs.

Why does requests.get() return 403 Forbidden?

The server thinks you're a bot. Send a real User-Agent and a Referer. If that fails, the site is probably running bot detection that checks your TLS fingerprint. Try curl_cffi or a headless browser.

How do I avoid overloading the target server?

Slow down. Add a delay between requests. Reuse one Session. Cache to disk so you never ask twice. Cap your concurrency. Be more polite than a human visitor.

Why should I use a Session instead of requests.get()?

A Session reuses TCP connections (keep-alive) and persists cookies across requests. The module-level requests.get() creates a fresh connection and throws it away every time, which is slower and noisier on a multi-page crawl.

What timeout should I set in requests?

Always set one. A tuple like timeout=(3, 10) means three seconds to connect and ten seconds between bytes of the response. Without a timeout, a stalled server hangs your scraper forever.

How do I retry failed requests with requests?

requests doesn't retry by default. Mount an HTTPAdapter with a urllib3 Retry policy: exponential backoff, retry only transient statuses like 429 and 5xx, and never auto-retry POSTs.

Do I need proxies to scrape as a beginner?

No. Proxies solve IP reputation problems you shouldn't have yet. On a sandbox site at one or two requests per second with a real User-Agent, you will not be blocked. Add proxies when a legitimate target rate-limits you and politeness has failed.

Keep reading

Found this useful? Cite it as: webscraping.space. “Web scraping with Python requests: a practical starter.” https://webscraping.space/blog/web-scraping-with-python-requests. Published 2026-06-28.