Python Published Jun 28, 2026 · Updated Jul 10, 2026 · 3 min read · 608 words

Web scraping with Python requests: a practical starter

A hands-on guide to web scraping with Python's requests library: downloading pages, setting headers, handling redirects, parsing with BeautifulSoup, and avoiding the most common beginner mistakes.

Web scraping with requests is how most people start scraping in Python. It's small. It's fast. It handles the majority of pages that ship data in the initial HTML.

This guide walks through a real scraper end to end. The parts most tutorials skip: headers, sessions, redirects, polite delays, and knowing when requests is the wrong tool.

What you need

Two packages. Nothing else.

pip install requests beautifulsoup4

requests downloads the page. BeautifulSoup turns the HTML into something you can query. The rest is standard-library Python.

Your first request

The smallest useful scraper downloads one page and prints the title:

import requests
from bs4 import BeautifulSoup

url = "https://example.com"
resp = requests.get(url, timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.string)

This works on example.com. It will not work on most real sites. Most real sites block traffic that looks like a default requests call. The fix is almost always the same: send realistic headers.

Headers are not optional

A bare requests.get() sends a User-Agent like python-requests/2.32.0. A lot of servers reject that on sight. Send a real browser User-Agent and a Referer:

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Referer": "https://www.google.com/",
}

resp = requests.get(url, headers=HEADERS, timeout=10)

A Referer of https://www.google.com/ makes the request look like a click from search. Many sites treat that better than a direct hit.

Reuse a Session

Every requests.get() opens a new connection. For a scraper that hits dozens of pages on one domain, that's wasteful and noisy. A Session reuses connections and keeps cookies. Faster and less conspicuous:

session = requests.Session()
session.headers.update(HEADERS)

# Fake "I arrived from search" hit
session.get("https://example.com/", timeout=10)

# Now crawl
for path in ["/page1", "/page2", "/page3"]:
    resp = session.get(f"https://example.com{path}", timeout=10)
    resp.raise_for_status()
    process(resp.text)

Politeness: the part that saves your IP

The highest-leverage thing you can do is slow down. A scraper that hammers a server is the scraper that gets banned first.

import time
import random

for path in paths:
    resp = session.get(f"https://example.com{path}", timeout=10)
    process(resp.text)
    time.sleep(random.uniform(1.5, 3.5))  # jittered delay

Jitter matters. A fixed 2-second interval is obviously a bot. A random interval between 1.5 and 3.5 seconds looks human. It also spreads your load.

Handling redirects and status codes

requests follows redirects by default. That's usually what you want. But handle status codes explicitly so a quiet 404 or 429 doesn't poison your data:

def fetch(session, url):
    resp = session.get(url, timeout=10, allow_redirects=True)
    if resp.status_code == 429:
        retry_after = int(resp.headers.get("Retry-After", "30"))
        time.sleep(retry_after)
        return fetch(session, url)
    if resp.status_code in (404, 410):
        return None
    resp.raise_for_status()
    return resp.text

429 is the server telling you to back off. Honor Retry-After and you'll usually be unblocked in minutes.

Parsing with BeautifulSoup

Once you have HTML, parsing is the easy part. Find all article cards. Pull the title and link from each:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
cards = soup.select("article.product-card")

for card in cards:
    title = card.select_one("h2 a")
    price = card.select_one(".price")
    if not title or not price:
        continue
    yield {
        "title": title.get_text(strip=True),
        "url": title["href"],
        "price": price.get_text(strip=True),
    }

Two habits that pay off forever:

1. Always null-check select_one before you touch it. Pages change. Selectors break.
2. Use get_text(strip=True) instead of slicing strings. It collapses whitespace for free.

When requests is the wrong tool

requests fetches HTML. It does not run JavaScript. If the data is injected by a script after load, you get an empty <div id="root"></div> and nothing else.

For the JS-rendered case, read the guide on headless browser scraping with Playwright.

Putting it together

A polite, reusable scraper template:

import time, random, json, pathlib
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 ... Chrome/126.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

def crawl(urls, out_path="out.jsonl"):
    cache = pathlib.Path(out_path)
    session = requests.Session()
    session.headers.update(HEADERS)
    with cache.open("a", encoding="utf-8") as f:
        for url in urls:
            try:
                resp = session.get(url, timeout=10)
                if resp.status_code != 200:
                    continue
                soup = BeautifulSoup(resp.text, "html.parser")
                record = {
                    "url": url,
                    "title": soup.title.string if soup.title else None,
                }
                f.write(json.dumps(record) + "\n")
            except requests.RequestException:
                continue
            time.sleep(random.uniform(1.5, 3.5))

That's 90% of a production scraper. The other 10% is caching, retries, queues, and dedup. That's what the scaling guide covers.

Key takeaways

• Send realistic headers. User-Agent and Referer matter.
• Reuse a Session. Don't open a new connection per request.
• Jitter your delays. Politeness is a defense, not just courtesy.
• Handle 429 and Retry-After explicitly.
• Null-check every selector before you use it.
• Only reach for a browser when the data is genuinely not in the HTML.

Found this useful? Cite it as: webscraping.space. “Web scraping with Python requests: a practical starter.” https://webscraping.space/blog/web-scraping-with-python-requests. Published 2026-06-28.