URL: https://webscraping.spaceBest viewed at 1024×768

Python Published Aug 8, 2026 · 36 min read · 8,007 words

Web Scraping with Python: The Complete 2026 Tutorial

A complete from-scratch web scraping tutorial in Python: picking a target, reading robots.txt, fetching with requests, parsing with BeautifulSoup and lxml, handling JavaScript pages, pagination, saving clean data to CSV/SQLite, and doing it all politely without getting blocked. Zero-to-working in one guide.

Every scraper starts the same way: a URL, a requests.get(), and a hope that the HTML contains what you need. Most tutorials stop there. They show you one page, one parser, one print(), and call it a tutorial. Then you go scrape something real and it breaks on page two, or gets a 403 on request thirty-seven, or saves the same record fourteen times.

This guide is the version I wish someone had handed me. It's the complete from-scratch pipeline: pick a legal practice target, read its robots.txt, fetch pages with requests, parse them with BeautifulSoup, paginate through the whole site, clean the fields into usable data, store them in CSV and SQLite, and do all of it politely enough that the site never blocks you. Every step has runnable code, real reasoning, and real numbers. When you're done, you'll have a scraper you can point at a real project — not a script that worked once.

A few things to expect before we start. This tutorial assumes you can write a for-loop and install a Python package, but nothing else. I explain every choice as we make it — why the Session, why the jitter, why the upsert — because the "why" is what survives when the site changes and your first script stops working. I also keep the numbers honest: a two-second delay per page means a 500-page site takes about fifteen minutes, and if that's too slow for your use case, the polite answer is concurrency and caching, not hammering faster. Everything here is reproducible on the sandbox sites in this guide, so every example is something you can run today.

One note before we start. There's a companion post that goes deep on the requests library itself: sessions, headers, redirects, the works. I'm not going to re-do that here. This post assumes you want the whole path, not the one library, and it links out where the deep dive already exists.

What you're building

Here is the entire discipline of web scraping, in one picture:

The whole job in one diagram: fetch, parse, extract, store — with politeness (rate limits, retries, caching) wrapped around every single request.

Four steps, one loop, one policy. Everything in this tutorial is just building that loop correctly and making it not fall over.

Setup: Python, a virtualenv, and three packages

You need Python 3.11 or newer. If you're on a Mac or Linux box, it's almost certainly already installed — check with python3 --version. Windows users: install from python.org and tick "Add Python to PATH" during setup, or use the Windows Store version. Any 3.11+ will do; the code here uses nothing exotic.

The number-one beginner mistake is installing packages into your system Python and then wondering why a script can't find them. Don't do that. Create a virtualenv — a small isolated folder per project — so each project gets its own packages and you never break a dependency for another project.

# one-time project setup, from inside your project folder
python3 -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install requests beautifulsoup4 lxml

Three packages, that's all you need to start:

requests — sends HTTP GET requests and returns the page. It's the fetch step.
beautifulsoup4 — parses HTML into a tree you can query. It's the parse step.
lxml — an XML/HTML parser that BeautifulSoup uses as its engine. It's an order of magnitude faster than the pure-Python html.parser, and it handles malformed real-world HTML better. You pass it in as the parser: BeautifulSoup(html, "lxml").

A sane project structure for a scraper that will grow:

scraper/
  .venv/
  scraper.py        # the fetch + parse + extract logic
  store.py          # CSV and SQLite writers
  cache.py          # disk cache (add it early; see Step 6)
  data/             # your CSVs and SQLite files land here

You don't need to split files on day one — the final project in this guide is a single script. But keep the boundaries in your head: fetching, parsing, cleaning, and storing are different concerns, and when one of them breaks, you want to fix it without touching the others.

Choose a legal, scrape-friendly target

Before you write a single line of fetch code, decide what you're going to scrape. This matters more than any library choice, because it determines whether you're learning or getting your IP blacklisted at hour one.

The smart move as a beginner is to practice on a site that exists for exactly this purpose. There are two excellent public sandboxes:

books.toscrape.com — an online bookstore with categories, pagination, prices, and ratings. It's the site I'll use throughout this guide because it has a realistic structure and it's explicitly meant to be scraped.
quotes.toscrape.com — a quote site with authors and tags, plus some paginated and JavaScript-rendered variants. Great for a second exercise.

Second-best: your own website, or a local HTML file. You can literally write page.html to disk and parse it with BeautifulSoup — that's the cheapest, most polite rehearsal of the parsing half of the pipeline, and you can do it completely offline.

Read robots.txt first — every time

robots.txt is the site's way of saying which parts it prefers you not to crawl. It is not a law; it's a convention, and in the US it generally isn't legally binding. But ignoring it is how you become the person whose IP is blocked by Cloudflare at the firewall level, and it's a bad habit that scales into worse ones. Read it before you scrape anything.

Here's what a typical robots.txt looks like:

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Crawl-delay: 2

User-agent: Googlebot
Disallow:

Reading it: User-agent: * means "for everyone". Disallow: /admin/ means "don't request paths under /admin/". Crawl-delay: 2 means "wait at least 2 seconds between requests" — it's a direct instruction about your request rate. The second block grants Googlebot everything, because Google is the reason the site exists.

books.toscrape.com's real robots.txt is even more permissive — it disallows only /cdn-cgi/, a Cloudflare system path — and the site is explicitly built to be scraped. That's why it's the right first target.

Fetch robots.txt with requests the same way you'll fetch everything else:

import requests
resp = requests.get("https://books.toscrape.com/robots.txt", timeout=10)
print(resp.text)

Two rules that will keep you out of trouble: (1) don't crawl the parts Disallow names, and (2) if a site's robots.txt has no Crawl-delay, pick your own conservative delay. We'll do the numbers in Step 6.

Step 1: Fetch a page with requests

Time to download a page. The smallest correct version:

import requests

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

url = "https://books.toscrape.com/catalogue/page-1.html"
resp = session.get(url, timeout=10)
resp.raise_for_status()
html = resp.text
print(len(html))

Three things here that most one-liner tutorials get wrong, and all three are the difference between working once and working reliably:

A Session, not bare requests.get(). A Session reuses TCP connections and keeps cookies across requests. Every bare requests.get() opens a brand-new connection — for a scraper hitting dozens of pages on one domain, that's wasteful, slower, and noisier on the server side. One session, many requests. The deep dive on the requests library goes further into headers and sessions.

Real headers. A default requests call advertises itself as python-requests/2.32.0 and a lot of servers reject that on sight. A normal browser User-Agent and an Accept-Language header makes your traffic look like exactly what it is trying to be: a browser. This is not deception to bypass a block — it's not sending a "please block me" banner. Nearly every 403 you'll ever hit as a beginner is fixed by real headers.

A timeout. requests.get() without a timeout will wait forever on a hung server. Your scraper stalls silently at request 23 and you lose an afternoon. timeout=10 says "give up after 10 seconds" — on connect and on read. Always set it. When a page times out, you'll retry it later (Step 6).

resp.raise_for_status() is a cheap assert: if the server returns 404 or 500, it raises instead of letting you parse an error page as if it were content. For 404, 410, and 429 you want custom handling (Step 6), but for now: if it's not a 200, don't proceed.

Step 2: Parse with BeautifulSoup

You have HTML. Now you need to ask questions of it. BeautifulSoup turns the page into a tree, and you query that tree with CSS selectors — the same selector language you already know from CSS.

To see how the querying works, here's the shape of one book card on books.toscrape.com, condensed:

<article class="product_pod">
  <h3>
    <a href="a-light-in-the-attic_1000/index.html" title="A Light in the Attic">A Light in the Attic</a>
  </h3>
  <div class="product_price">
    <p class="price_color">£51.77</p>
    <p class="instock availability">In stock</p>
    <p class="star-rating Four"></p>
  </div>
</article>

Parse it and pull out the three fields we care about:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")

for card in soup.select("article.product_pod"):
    link = card.select_one("h3 a")
    title = link.get("title") or link.get_text(strip=True)
    price = card.select_one(".price_color")
    rating = card.select_one(".star-rating")
    if not link or not price:
        continue
    print(title, price.get_text(strip=True), rating.get("class"))

This is the core of the entire tutorial, so it's worth getting the habits right on day one:

soup.select() returns a list of everything matching the selector. select_one() returns the first match or None. Use select for loops over repeated elements (cards, rows, posts) and select_one for one-off fields inside each.
Null-check every select_one before touching it. If the page changes and .price_color disappears, price is None and price.get_text() crashes your whole run. The if not link or not price: continue line is what turns a crash into a skipped card. Pages change. Selectors break. Null-checks are your seatbelt.
get_text(strip=True) collapses whitespace. It gives you "A Light in the Attic" instead of "\n A Light in the Attic\n". Never slice strings when you can let the parser strip them for you.
get() on a tag reads an attribute. link.get("title") reads the title attribute; if it's absent, the or falls back to the text content. Real sites use title attributes like this all the time and it's usually cleaner than the text.

One more thing worth knowing: BeautifulSoup with html.parser is fine, but lxml is faster and more forgiving of real-world HTML. That's why we installed it. There's a dedicated guide on parsing HTML with BeautifulSoup that goes deeper into selectors, parent/sibling traversal, and the parser trade-offs — this tutorial only needs the above.

The five selectors that cover 90% of pages

You do not need the full CSS selector spec to scrape. Five patterns handle the overwhelming majority of real pages:

tag matches elements by name — h3, a, p. Coarse but a good first anchor.
.class matches by class — .price_color, .next. Classes are the most common hook in scraped pages because front-end developers use them for styling.
#id matches by id. Ids are unique per page, so they're the strongest hook when they exist — but many sites don't use them on the elements you want.
tag.class combines both — article.product_pod means "an article element that also has class product_pod." Combining a tag with a class is dramatically more specific than either alone.
parent child (descendant selector, space-separated) — h3 a means "an a inside an h3, at any depth." This is how you navigate structure without needing parent/child traversal.

A few practical notes. a[href^="/catalogue/"] matches anchors whose href starts with a prefix — useful for filtering navigation links from footer links. ul li (descendant) is usually safer than ul > li (child) because real HTML nests unexpectedly. And when two widgets share a class name, scope the query: article.product_pod h3 a restricts matches to inside the card. If your select returns more than the page shows, that's this problem, and scoping is the fix.

Step 3: Pagination — the part everyone forgets

The first page is a warm-up. Real scrapers loop. books.toscrape.com paginates with a "next" link at the bottom of each listing page. The pattern is universal: find the "next" link, follow it, repeat, stop when it's gone.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def listing_urls(session, start_url, max_pages=5):
    url = start_url
    seen = 0
    while url and seen < max_pages:
        resp = session.get(url, timeout=10)
        resp.raise_for_status()
        soup = BeautifulSoup(resp.text, "lxml")

        for card in soup.select("article.product_pod"):
            yield url, card

        nxt = soup.select_one("li.next a")
        url = urljoin(url, nxt.get("href")) if nxt else None
        seen += 1

Three patterns to copy, not just here but in every scraper you ever write:

urljoin for relative links. The next link is almost always relative — page-2.html — and urljoin(base, relative) resolves it correctly against the current page. String concatenation with f"{url}{href}" silently breaks the day the site changes its URL structure. Use urljoin always.

A cap on pages. max_pages=5 is the difference between "test my pagination logic" and "accidentally crawl 10,000 pages because my while-loop condition was wrong." Every pagination loop gets a hard cap during development. Raise it when you've verified the loop terminates.

The loop stops when next disappears. soup.select_one("li.next a") returns None on the last page, so the loop ends on its own. That's the cleanest possible termination condition — no guessing at page counts, no math on total items divided by page size.

There's a subtle point hiding in that while url condition: it guards against a "next" link that loops back to the current page. That happens more than you'd think on real sites — a pagination widget that renders a "next" arrow even when there's nowhere to go. If the site hands you a link back to the page you're on, while url alone won't save you; you need the seen count or a visited set. For a sandbox site the cap is enough.

The delay between page requests goes here too — one time.sleep(random.uniform(...)) at the end of each loop iteration. That's Step 6's concern, but the hook is in this loop, right after yield. Build it into the loop from the start rather than adding it later, because "later" is when you forget.

Step 4: Extract and clean the data

Parsing gives you strings from HTML. Strings are not data. "£51.77", "In stock", and "Four stars" are display formatting; what you actually want is 51.77 (a float), True (a boolean), and 4 (an integer). Cleaning is where the scraped text becomes usable — and it's where the "scraping is easy" crowd skips the work.

import re
from datetime import datetime

def clean_price(raw):
    # "£51.77" -> 51.77
    digits = re.sub(r"[^\d.]", "", raw)
    return float(digits) if digits else None

def clean_rating(raw_classes):
    # ["star-rating", "Four"] -> 4
    words = [c for c in raw_classes if c != "star-rating"]
    stars = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
    return stars.get(words[0]) if words else None

def clean_date(raw):
    # "Aug 8, 2026" -> "2026-08-08"
    for fmt in ("%b %d, %Y", "%Y-%m-%d", "%d/%m/%Y"):
        try:
            return datetime.strptime(raw.strip(), fmt).date().isoformat()
        except ValueError:
            continue
    return None

def clean_whitespace(raw):
    return " ".join(raw.split())

record = {
    "title": clean_whitespace("  A Light  in the Attic  "),
    "price": clean_price("£51.77"),
    "rating": clean_rating(["star-rating", "Four"]),
    "stock": "In stock" in "In stock",
}
print(record)  # {'title': 'A Light in the Attic', 'price': 51.77, 'rating': 4, 'stock': True}

Notes on the choices here:

Currency symbols are formatting, not data. Strip them. re.sub(r"[^\d.]", "", "£51.77") keeps only digits and the decimal point. (If you ever scrape a locale where thousands separators use commas — 1,234.56 or 1.234,56 — this regex needs a smarter sibling. For the sandbox, digits-plus-dot is right.)

Normalize whitespace with .split()/.join(), not regex. " ".join(raw.split()) collapses runs of spaces, tabs, and newlines into single spaces, and strips the ends. This one idiom handles 90% of the messy-HTML whitespace problems you'll meet. The regexes are for structural changes (currency, punctuation); .split() is for whitespace.

Ratings-as-words need a lookup table. The site encodes a four-star rating as a CSS class Four. That's display, not data; map the word to the number.

Dates get normalized to ISO 8601 (YYYY-MM-DD). If you store "Aug 8, 2026" you can't sort on it or compare it without re-parsing it every time. Store a sortable, unambiguous string once, at write time. The multi-format loop handles the reality that real sites rarely use the format you expect — and returns None rather than crashing when none of your formats match, which surfaces the problem in your data instead of killing the run.

Clean as close to the extraction as possible, field by field. Don't collect raw strings and "fix them later" — later you'll have 40,000 dirty rows and no way to know which came from which page. Clean once, at the edge, and everything downstream (CSV, SQLite, analysis, API) gets clean input for free.

Validate before you store

Cleaning turns strings into types. Validation checks the values — and it's the step that catches a changed page before it poisons your dataset. The principle is cheap and brutal: a record with a missing or impossible field is worth None and a log line, not a row in your database.

def is_valid(record):
    checks = [
        bool(record["title"]) and len(record["title"]) < 300,
        record["price"] is not None and record["price"] > 0,
        record["rating"] in (1, 2, 3, 4, 5),
        record["url"].startswith("https://"),
    ]
    return all(checks)

records = [r for r in raw_records if is_valid(r)]

What this buys you: if the site redesigns and the price selector starts matching a promo badge — or stops matching at all — the price > 0 check quietly drops the bad rows and you notice a count drop instead of discovering six months later that your analysis used a column half full of None. Bounds like "rating is 1 through 5" and "price is positive" are the ones worth asserting; don't go overboard and write a schema validator, just catch the impossible. What happens to a bad record is a policy choice — skip it, store it with a None flag, or fail the run. For a scraper you're still developing, skipping with a log line keeps the run alive so you can see which URLs failed, which is exactly the signal you need to fix the selector.

Step 5: Save the data — CSV first, then SQLite

Cleaned data that lives only in memory is data you will lose. Save it, and save it in two forms for two different jobs.

CSV with the stdlib `csv` module

CSV is for handing data to a human: open it in a spreadsheet, send it to a colleague, load it into pandas. The stdlib csv module handles quoting and escaping, which is more than you'd think — the naive ",".join(record) approach breaks on any field containing a comma or a newline, and book titles love commas.

import csv

def write_csv(records, path):
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["title", "price", "rating", "stock", "url"])
        writer.writeheader()
        writer.writerows(records)

Three details that each prevent a specific bug:

newline="" stops the csv module from writing the blank lines between rows that Windows line endings otherwise inject. Your CSV will be corrupted-looking in Excel without it.
encoding="utf-8" is non-negotiable. Without it, non-ASCII characters — the accented é in "René Descartes", the £ sign — get mangled or crash the writer on some platforms.
csv.DictWriter takes dicts with named fields. It writes a header row and handles escaping of commas and quotes inside fields. You keep your record dicts; the writer does the formatting.

CSV's weakness is that it's a snapshot. Re-run your scraper and you'll get a second file with duplicates, or you'll overwrite the first. That's fine for a one-off report. For anything you'll re-run, you want a database.

SQLite with the stdlib `sqlite3` module

SQLite is a database that lives in one file. No server, no setup, no credentials — it's the perfect persistence layer for a scraper. The feature that makes it the right tool for this job is idempotent upserts: you can insert a record knowing that if it already exists, it gets updated instead of duplicated.

import sqlite3

def init_db(path="data/books.db"):
    conn = sqlite3.connect(path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS books (
            title TEXT PRIMARY KEY,
            price REAL,
            rating INTEGER,
            stock INTEGER,
            url TEXT UNIQUE
        )
    """)
    return conn

def upsert(conn, record):
    conn.execute("""
        INSERT INTO books (title, price, rating, stock, url)
        VALUES (:title, :price, :rating, :stock, :url)
        ON CONFLICT(title) DO UPDATE SET
            price = excluded.price,
            rating = excluded.rating,
            stock = excluded.stock,
            url = excluded.url
    """, record)

conn = init_db()
for record in records:
    upsert(conn, record)
conn.commit()

This is the exact pattern you'll carry to every scraping project:

CREATE TABLE IF NOT EXISTS makes initialization idempotent. Run it every time you start the scraper; if the table already exists, nothing happens.
ON CONFLICT(title) DO UPDATE is the upsert. title is the primary key, so the second time you scrape the same book, the existing row is updated in place instead of duplicated. Re-running the scraper produces exactly the same table, not an ever-growing pile of copies. This one clause is the difference between "scraper" and "duplicate generator."
conn.commit() — SQLite transactions are only durable after a commit. Forget it and your data evaporates when the process exits. Commit once per page, or once per batch; per-record commits are slow.
Parameterized queries (:title) instead of f-string SQL. This is not optional, even on your own data. F-string SQL is how you get a ValueError from a title containing an apostrophe — and how SQL injection gets in. Parameterized always.

For a beginner dataset this schema is right: a unique key (title), the data fields, and a url for provenance. Always store the URL you scraped from. When your data has a mystery value and you need to check the source, you'll thank the past you who kept it.

Reading the data back out

Storing is half of the database habit; the other half is remembering the data is now queryable. You don't need to load the whole table into Python to ask questions of it — SQL does the asking, and it runs in a few lines:

import sqlite3

conn = sqlite3.connect("books.db")
conn.row_factory = sqlite3.Row

# The five most expensive in-stock books
rows = conn.execute(
    "SELECT title, price FROM books WHERE price IS NOT NULL ORDER BY price DESC LIMIT 5"
).fetchall()
for row in rows:
    print(dict(row))

Two habits to copy here. First, conn.row_factory = sqlite3.Row makes rows behave like dicts (row["title"]) instead of anonymous tuples — tuple indexing is how you end up with row[7] in code and no memory of what column 7 was. Second, filter with WHERE in the query, not in a Python loop: WHERE price IS NOT NULL keeps out the rows your validation let through with a missing price, and the database does the filtering before you ever see the rows. When the dataset grows to tens of thousands of rows, WHERE in SQL is instant where a list comprehension over all of them is not. And when you want to hand a query's result to a human, it's the same csv writer as before — SELECT a table, DictWriter it to a file, done.

Step 6: Politeness — rate limit, retry, cache

This is the step that separates scripts from scrapers, and it's where the "scraping is easy" tutorials go silent. A polite scraper is one that (a) never asks the server for more than a slow human would, (b) handles the server saying "slow down" without dying, and (c) never asks for the same page twice.

Let's do the numbers first, because "be polite" without numbers is vibes. Here's what happens to your odds of a block as your request rate climbs on a typical mid-size site:

Block probability rises gently until roughly 5 requests per second and then bends sharply upward. Below 1 request per second, most sites will never notice you.

Those numbers are a typical shape, not a universal law — a fragile WordPress site with no protection will tolerate far more than a shop behind a WAF. But the shape is right, and it drives the rule: stay under about 1 request per second per domain, and you are inside the flat part of the curve. On books.toscrape.com, 1.5 to 2.5 seconds per request with jitter is comfortable.

Rate limit with jitter

A fixed 2-second sleep is obviously a bot — perfectly metronomic. A random interval between 1.5 and 2.5 seconds looks like a human and spreads your load. Python's random.uniform is the tool.

import time, random

for url in urls:
    resp = session.get(url, timeout=10)
    process(resp)
    time.sleep(random.uniform(1.5, 2.5))  # jittered politeness delay

If the target's robots.txt specifies a Crawl-delay, use it as the floor and add jitter on top. Crawl-delay: 2 means "at least 2 seconds" — a 2.0–3.0 second sleep honors it.

Retries with exponential backoff

Networks fail. Servers hiccup. A single timeout shouldn't kill a 200-page run. Retry, but retry well: only on transient failures (timeouts, 5xx, 429), never on a 404 (the page doesn't exist and retrying won't change that), and with backoff that grows each attempt.

import time, random

def fetch(session, url, attempts=4):
    for i in range(attempts):
        try:
            resp = session.get(url, timeout=10)
            if resp.status_code == 200:
                return resp.text
            if resp.status_code in (404, 410):
                return None
            if resp.status_code == 429:
                wait = int(resp.headers.get("Retry-After", str(2 ** i)))
                time.sleep(wait)
                continue
            if resp.status_code >= 500:
                time.sleep(2 ** i + random.random())
                continue
            return None
        except requests.RequestException:
            time.sleep(2 ** i + random.random())
    return None

The pieces that matter:

429 handling honors Retry-After. When a server says "slow down," the Retry-After header is its explicit instruction for how long to wait. Respecting it is how you get unblocked in minutes instead of hours.
Backoff is exponential with jitter. Attempt 2 waits ~2 seconds, attempt 3 ~4, attempt 4 ~8, plus a random fraction so simultaneous retries don't pile up. 2 ** i gives the exponential part; random.random() gives the jitter.
404/410 short-circuits. Retrying a 404 is pure load on the server for a guaranteed failure. Return None and let the caller skip it.
Non-200, non-4xx, non-5xx... — e.g. a 403 — also returns None rather than retrying forever. A 403 won't fix itself on the fifth try; fix the headers instead.

Cache to disk: never request the same page twice

This is the single highest-ROI habit in all of scraping, and it's free to implement. The idea: before you hit the network, check whether you've already saved this page. If you have, read it from disk and skip the request entirely. You'll thank it the first time your parser has a bug, you fix it, and you re-run without asking the site for a single page you already have.

During development you will re-run the parser over and over. Without a cache, every parser fix means re-requesting the whole site. With a cache, a parser fix costs zero requests and runs in milliseconds.

A disk cache is ~10 lines. Hash the URL into a filename so it's filesystem-safe, write the body, read it back:

import hashlib, pathlib, json

CACHE = pathlib.Path("cache")
CACHE.mkdir(exist_ok=True)

def cache_path(url):
    digest = hashlib.sha256(url.encode()).hexdigest()
    return CACHE / f"{digest}.json"

def cached_get(session, url):
    p = cache_path(url)
    if p.exists():
        return json.loads(p.read_text())["body"]
    resp = session.get(url, timeout=10)
    if resp.status_code != 200:
        return None
    p.write_text(json.dumps({"url": url, "body": resp.text}))
    return resp.text

Now every URL in the crawl is fetched from the network at most once, ever — across runs, across parser fixes, across a crash that killed the script at page 40. Rerunning a 50-page site takes a blink instead of two minutes of network calls. This is also the polite move: the target serves your data once, and you stop asking.

The SHA-256 hash of the URL as filename avoids the problem of URLs containing characters that aren't legal in filenames. If the site's data goes stale (you want the current price, not last week's), delete the cache folder or add a TTL — for a first version, cache-everything is the right default. The scaling guide covers cache invalidation, dedup, and queues for when "one folder" becomes "a million pages."

Log what you did, or you'll never know what broke

You will run this scraper, walk away, come back to find it returned 3,100 books when it should have returned 3,500 — and you will have no idea which 400 pages failed or why. That's a logging problem, and the fix costs three lines. The standard-library logging module is enough; no framework needed.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    filename="scrape.log",
)
logger = logging.getLogger("scraper")

# in your fetch function:
logger.info("fetched %s (%d bytes)", url, len(html))
logger.warning("got %s for %s, retrying", resp.status_code, url)
logger.error("skipping card, no link found in %s", url)

The rule that makes logs useful instead of noise: every decision point logs the URL. A successful fetch, a retry, a skipped card, a page with no "next" link — each one names its URL. When something's off, you grep the log for WARNING and you have a list of exact pages to investigate. That's a different debugging experience from staring at an empty output file and guessing. The timestamps in scrape.log are also your politeness audit trail: they show your real request rate, which is exactly the evidence you need if you ever suspect you're hammering a site harder than you meant to.

Step 7: When JavaScript breaks everything

You've done everything right, and the site returns a page that's just <div id="root"></div> with no data in it. That means the data is rendered by JavaScript after the page loads. requests doesn't run JavaScript. What do you do?

Most people's first instinct is a headless browser. That's the wrong first instinct, and it's worth understanding why. A headless browser is roughly 10x the CPU and memory of an HTTP client, and it's far more detectable — every fingerprint leak is a chance for a bot wall to refuse you. Before you pay that cost, check two things, in order:

The decision flow that saves you a headless browser 90% of the time: view source first, then look for the site's own JSON API. The browser is the last rung, not the first.

Step one: view the raw source. Open the page, right-click, View Source (not Inspect — Inspect shows the rendered DOM). Search for one of the fields you want. If it's there, your problem isn't JavaScript, it's your selector. If it's not there, the data really is injected.

Step two: check the Network tab for a JSON API. Open DevTools, reload, filter by XHR/Fetch. Most "JavaScript-rendered" sites aren't rendering anything — the browser fetched clean JSON from an API endpoint and the JavaScript just paints it onto the page. That JSON endpoint is a gift: call it directly with requests and you get structured data with zero parsing and zero browser overhead.

import requests

# What the site's own browser does when you open a product page:
# it calls https://api.example.com/products?page=2 and paints the JSON.
resp = requests.get("https://api.example.com/products", params={"page": 2}, timeout=10)
data = resp.json()
print(data["results"][0]["title"])

Look at how the Network tab shows the request, and replicate it in requests: the URL, the query params, and (critically) any headers the API needs — an Accept: application/json header, an API key, maybe a Referer that says it came from the page. The Playwright post on this blog covers the API-interception trick in detail, because it applies to real headless scraping too: even when you must drive a browser, intercepting the site's own API responses is more robust than scraping the DOM.

Only when there is no JSON API and the data genuinely requires executing JavaScript do you reach for a headless browser. That's Playwright's job, and it deserves its own guide — headless browser scraping with Playwright covers waiting strategies, stealth, and running the browser only when you must. For this tutorial, the rule to internalize is: the browser is the last resort, not the first tool.

Infinite scroll is just pagination in disguise

One more JavaScript pattern you'll meet constantly, because it's now the default for "load more" on the web: infinite scroll. You scroll, and more items appear. It looks like the worst-case scenario for a scraper — "I'd need to scroll forever!"

You almost never do. An infinite-scroll page still loads its data in chunks, and each chunk is a request you can see in the Network tab: ?page=1, ?page=2, or ?offset=0, ?offset=50, or a cursor parameter. Open the Network tab, scroll once, and look at what fired. That chunk URL is your pagination link, and it's the same urljoin loop from Step 3, just with params:

for page in range(1, 11):
    resp = requests.get(
        "https://example.com/api/items",
        params={"page": page, "limit": 50},
        timeout=10,
    )
    for item in resp.json()["results"]:
        process(item)

The page counter is your "next link," and the loop stops when the API returns fewer items than limit (the server's own way of saying "there is no page N+1") — the exact same termination logic as next disappearing on a classic site. The only real difference is that the chunked API gives you clean JSON instead of HTML you have to parse. When a site uses a cursor or a signed token in its API URLs, the pattern still holds: whatever the browser sends, you replicate. The moment you find yourself writing scroll logic, ask whether the browser is the one scrolling or just the one translating the same chunked requests you could send directly.

The complete scraper: all steps, one script

Here's everything above welded into one ~60-line scraper. It targets books.toscrape.com, follows pagination, extracts and cleans fields, caches every page to disk so re-runs are free, rate-limits politely, and upserts into SQLite so re-runs never duplicate. This is a real, runnable program — copy it, pip install requests beautifulsoup4 lxml, and it works.

import csv, hashlib, json, pathlib, random, re, sqlite3, time
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://books.toscrape.com/catalogue/page-1.html"
CACHE = pathlib.Path("cache"); CACHE.mkdir(exist_ok=True)
DB = pathlib.Path("books.db")
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

def cached_get(session, url):
    p = CACHE / (hashlib.sha256(url.encode()).hexdigest() + ".json")
    if p.exists():
        return json.loads(p.read_text())["body"]
    resp = session.get(url, timeout=10)
    if resp.status_code != 200:
        return None
    p.write_text(json.dumps({"url": url, "body": resp.text}))
    return resp.text

def clean_price(raw):
    return float(re.sub(r"[^\d.]", "", raw)) if raw else None

def clean_rating(classes):
    stars = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}
    return stars.get(next((c for c in classes if c != "star-rating"), ""))

def parse_card(card, page_url):
    link = card.select_one("h3 a")
    price = card.select_one(".price_color")
    rating = card.select_one(".star-rating")
    if not link or not price:
        return None
    return {
        "title": " ".join(link.get("title", link.get_text()).split()),
        "price": clean_price(price.get_text()),
        "rating": clean_rating(rating.get("class", [])),
        "url": urljoin(page_url, link["href"]),
    }

def run(max_pages=10):
    session = requests.Session()
    session.headers.update(HEADERS)
    conn = sqlite3.connect(DB)
    conn.execute("""CREATE TABLE IF NOT EXISTS books (
        title TEXT PRIMARY KEY, price REAL, rating INTEGER, url TEXT UNIQUE)""")

    url = BASE
    records = []
    for _ in range(max_pages):
        html = cached_get(session, url)
        if html is None:
            break
        soup = BeautifulSoup(html, "lxml")
        for card in soup.select("article.product_pod"):
            rec = parse_card(card, url)
            if rec:
                records.append(rec)
                conn.execute("""INSERT INTO books (title, price, rating, url)
                    VALUES (:title, :price, :rating, :url)
                    ON CONFLICT(title) DO UPDATE SET
                    price = excluded.price, rating = excluded.rating, url = excluded.url""", rec)
        nxt = soup.select_one("li.next a")
        if not nxt:
            break
        url = urljoin(url, nxt.get("href"))
        time.sleep(random.uniform(1.5, 2.5))
    conn.commit()
    conn.close()

    with open("books.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["title", "price", "rating", "url"])
        writer.writeheader()
        writer.writerows(records)
    print(f"done: {len(records)} books, {len(records)} rows in books.db, books.csv written")

if __name__ == "__main__":
    run()

Read it top to bottom and you'll recognize every decision from the earlier steps: the Session with real headers, the disk cache guarding every fetch, the null-checked parser, urljoin for pagination, the jittered delay, the SQLite upsert keyed on title, the CSV snapshot for humans. The politeness layer is two lines in the middle of the loop (cached_get + the sleep), and it's the reason this scraper can run ten times a day without anyone on the site noticing or caring.

Note the shape of parse_card: it takes one card element and returns one dict, or None. That's the extract step isolated so it's independently testable — paste an <article class="product_pod"> snippet into a test and you can verify parsing without touching the network. Isolating the pure function from the I/O is the single biggest factor in whether a scraper survives its first site redesign.

Run it once. Run it again. The second run prints in under a second and makes zero network requests. That's not magic; that's the cache doing exactly what it was built for.

Common beginner mistakes and fixes

Every mistake in this table is one I've made or watched someone make. They're the difference between "scraper doesn't work and I don't know why" and "scraper doesn't work and here's exactly where."

Error	Cause	Fix
`403 Forbidden` on every request	Server sees the default python-requests user agent, or bot detection on headers	Send a real User-Agent and Referer, reuse a Session; then try curl_cffi for TLS fingerprinting; then a browser
`429 Too Many Requests`	Requesting faster than the site allows	Honor Retry-After, exponential backoff, jittered delays, disk cache
Selectors return empty lists	Wrong selector, or data injected by JavaScript	Copy the selector from DevTools; View Source (not Inspect) to check the raw HTML; look for a JSON API
Mojibake: `Ã©` instead of `é`	Wrong charset assumption	Use `resp.text` with `resp.encoding`, or `resp.apparent_encoding`; always write with `encoding="utf-8"`
`ModuleNotFoundError: requests`	Package installed into a different Python or venv not activated	`source .venv/bin/activate`, then `pip list` to confirm, then `pip install requests`
`UNIQUE constraint failed`	Inserting duplicate primary keys	Use `ON CONFLICT(...) DO UPDATE` so re-runs upsert instead of failing
Works on page 1, breaks on page 2	Pagination link is relative and got concatenated wrong, or page 2 has different markup	Always `urljoin(base, href)`; log the URL before each fetch; cap pages during dev
Scraper runs for an hour and returns nothing	A silent error (timeout, parse miss) was swallowed	Log every non-200 and every skipped card with its URL; null-check selectors and record `None` fields

Two of these deserve an extra sentence because they eat the most beginner time.

The Inspect-vs-View-Source trap. You open DevTools, see your field right there in the Elements panel, and write a selector that finds nothing. That's because DevTools shows the rendered DOM, after JavaScript ran — but requests gets the source. If your selector works in DevTools and not in your script, view the source first. Nine times out of ten the difference is JavaScript, and the fix is the JSON API from Step 7, not a broken selector.

The silent-failure trap. A scraper that swallows errors returns an empty file and a clean exit code — the worst possible failure mode, because nothing tells you it failed. Every time you catch an exception, log the URL. Every time you skip a card, log the URL. The two-minute cost of a log line pays for itself the first time your output comes back empty.

Key takeaways

The whole job is a four-step loop — fetch, parse, extract, store — wrapped in a politeness policy. Build the loop right once and it scales.
Practice on sandbox sites (books.toscrape.com, quotes.toscrape.com) and read robots.txt before touching anything else.
Fetch with a requests.Session, real headers, and a timeout. Bare requests.get() with default headers is how beginners get 403s.
Parse with BeautifulSoup over lxml. Null-check every select_one, and use get_text(strip=True) for clean text.
Paginate with urljoin on the "next" link, capped during development.
Clean at the edge: strip currency, normalize whitespace and dates, before anything hits a file.
Save to CSV for humans, SQLite with ON CONFLICT DO UPDATE for re-runs. Upserts mean re-running never duplicates.
Stay under ~1 request per second with jitter, retry only transient errors with exponential backoff, and cache every page to disk so you never request anything twice.
JavaScript-rendered pages are a decision tree, not a panic: view source, look for a JSON API, and only then consider Playwright.
A scraper that fails silently is worse than one that crashes. Log URLs everywhere.

What to build next

You now have a working pipeline. The fastest way to make it stick is a second project that stretches exactly one muscle further. Pick one:

A price monitor. Scrape a category on books.toscrape.com into SQLite every day, add a scraped_at timestamp column, and write a query that shows which prices moved between runs. This exercises the upsert, the timestamps, and your first "data over time" query — and it's the skeleton of every real price-tracking project.
A multi-category crawl. Scrape all fifty categories on the site, not just the first page of one. You'll need to discover category links, deduplicate products that appear in several categories, and decide whether title is still a safe primary key. That's your first taste of crawl logic, and the scraping at scale guide becomes directly relevant.
A single-page app, done right. Take a site whose data is JS-rendered and run the Step 7 decision tree: view source, find the JSON API, call it directly. Practicing the API-interception path on a sandbox site means you'll recognize it instantly on a real one.
A report. Build a small script that reads your SQLite database and writes a weekly CSV summary — the scraper as a service to your own analysis, not a one-off.

Whichever you choose, keep the discipline from this guide: cache everything, log every decision, stay under a request per second, and re-run your scraper twice in a row — if the second run hits the network at all, your cache is broken. Fix that before you build anything on top.

Frequently Asked Questions

How long does it take to learn web scraping in Python?

You can have a working scraper in under an hour: requests to fetch, BeautifulSoup to parse, a loop to paginate, csv or sqlite3 to save. The next two to three weeks are where the real skill lives: handling bot detection, broken selectors, JavaScript-rendered pages, and rate limits. Most people who give up do so at exactly that stage.

What's the best Python library for web scraping?

Requests plus BeautifulSoup (with lxml as the parser) is the best default for the vast majority of projects. It's simple, fast, and hard to detect. Move to Scrapy when you have a real crawl with thousands of pages. Reach for Playwright only when the data is rendered by JavaScript and no JSON API exists.

Do I need a headless browser to scrape with Python?

No — and you should avoid it. A headless browser costs about 10x the CPU and memory of an HTTP client and is far easier to detect. First check whether the data is in the initial HTML. Then check the Network tab for a JSON API. Only if both fail do you need Playwright.

Can I scrape a website with just requests and BeautifulSoup?

Yes. If the data is in the HTML, plain requests plus BeautifulSoup handles it: fetch, parse, paginate, save. That covers the majority of real sites. You only need more tooling when JavaScript renders the content, when the site rate-limits you, or when you scale to thousands of pages.

How do I avoid getting blocked while learning?

Start on a sandbox site like books.toscrape.com. Practice on your own site or a local file when possible. Send a realistic User-Agent, reuse a Session, delay between requests with jitter, and cache responses to disk so you never re-request. Stay under a few requests per second and most sites will never notice you.

Is web scraping legal?

Scraping public data is generally legal, but it depends on what you scrape, where you live, and what you do with it. Check robots.txt and the Terms of Service. Never scrape behind a login or paywall, never grab personal data you don't need, and don't resell someone else's content wholesale. When in doubt, ask a lawyer who does internet law.

Why am I getting 403 Forbidden on every request?

The server thinks you're a bot. Nine times out of ten a real User-Agent and a Referer fixes it. If it still fails, the site is checking your TLS fingerprint — try curl_cffi, which impersonates a browser's TLS handshake, before reaching for a headless browser.

Should I save my scraped data to CSV or SQLite?

Both, for different jobs. CSV for data you'll open in a spreadsheet or hand to someone else. SQLite when you'll re-run the scraper, because INSERT OR REPLACE gives you idempotent upserts — re-running never creates duplicates. You can always export a table back to CSV later.

What do I do if my selectors find nothing but the page looks right?

Check the raw HTML, not the rendered page. Right-click, View Source, and search for your field. If it's not there, the content is injected by JavaScript — open DevTools, find the JSON API, and call it directly. If it is there, your selector is wrong: copy the selector from DevTools and test it in isolation.

Do I need proxies to scrape as a beginner?

No. Proxies solve IP reputation problems that you shouldn't have yet. On a sandbox site at one or two requests per second with a real User-Agent, you will not be blocked. Add proxies only when a legitimate target is rate-limiting you and politeness has failed.

Keep reading

Found this useful? Cite it as: webscraping.space. “Web Scraping with Python: The Complete 2026 Tutorial.” https://webscraping.space/blog/web-scraping-python-tutorial. Published 2026-08-08.