URL: https://webscraping.spaceBest viewed at 1024×768

Parsing Published Jul 2, 2026 · 30 min read · 6,612 words

Parsing HTML with BeautifulSoup: selectors, speed, and survival

A deep look at parsing HTML with BeautifulSoup and lxml in Python: CSS vs XPath, handling malformed markup, the speed difference between parsers, and writing selectors that survive small redesigns.

Everyone thinks parsing is the hard part of scraping. It isn't — fetching reliably is. But once the bytes are in memory, how you parse them decides whether your scraper survives a small redesign or breaks every Tuesday morning. I've maintained extractors that run every night against sites that ship a new theme every quarter, and I've rewritten more selector strings than I care to count. This guide is everything I wish someone had told me about BeautifulSoup before I learned it the expensive way: the three parsing backends and what each one actually costs, the speed differences you can measure, CSS selectors versus XPath and when XPath is the only sane answer, how to write selectors that outlive a redesign, what to do with genuinely broken HTML, and the pipeline shape for parsing a hundred thousand pages without re-thinking any of it.

All the numbers in this post are real. I generated a synthetic catalog page of 1.13 MB with 2,600 product cards (each with a header, a link, a price, a stock badge, and a date), then parsed it with every parser discussed, taking the best of three runs on Python 3.12. You can reproduce the benchmark in about twenty lines. The absolute numbers will vary on your hardware; the ratios won't.

The three parsers

BeautifulSoup does not parse HTML. It orchestrates a parser. The string you pass as the second argument selects a completely different piece of software, with a different speed, a different tolerance for broken markup, and different installation requirements:

from bs4 import BeautifulSoup

BeautifulSoup(html, "html.parser")  # stdlib, pure Python, always available
BeautifulSoup(html, "lxml")         # C library (libxml2), needs native wheels
BeautifulSoup(html, "html5lib")     # pure Python, spec-complete repair

html.parser is part of the standard library. Zero install, zero native build, zero headaches on exotic platforms and locked-down CI images. It is also the slowest parser you'd ever pick on purpose. For a scraping project it's the safe default when you cannot install lxml and don't need to. Nothing to tune, nothing to break.

lxml is the workhorse. It binds libxml2, a C library that has been parsing XML and HTML for two decades, and it is the parser BeautifulSoup recommends in its own documentation. Installation on Linux and macOS is a wheel download; no compiler needed on any platform that has wheels. It's fast, it repairs broken markup to a sensible tree, and it gives you XPath through lxml.html if you want it. Nine out of ten scrapers should be running lxml.

html5lib is the most interesting of the three and the least often needed. It's a pure-Python implementation of the HTML5 parsing algorithm — the same algorithm browsers use — which means it's astonishingly tolerant of malformed markup and produces a tree that matches what a browser would build. The price is speed: it was the slowest parser in every benchmark I've run, by a wide margin.

The practical decision procedure is short:

Parser	Speed (1.13 MB page)	Broken-markup tolerance	Install	Best for
lxml	0.81 s	Good — auto-closes tags, fixes nesting	native wheels	The default. Speed and XPath.
html.parser	1.33 s	Okay — repairs some, less aggressively	stdlib, always there	No-native-deps environments
html5lib	2.64 s	Best — full browser algorithm	pure Python	Truly broken markup, exact DOM

Default to lxml. Fall back to html.parser when you can't install native dependencies. Reach for html5lib when the markup is so broken that lxml produces a tree you can't select against. In five years of scraping I have used html5lib maybe a dozen times, but when I needed it, nothing else worked.

lxml's C-speed advantage, measured

Here is the benchmark. One synthetic catalog page, 1.13 MB, parsed end to end, best of three:

import time
from bs4 import BeautifulSoup
import lxml.html as lhtml

def best_of(fn, n=3):
    best = float("inf")
    for _ in range(n):
        t0 = time.perf_counter(); fn(); best = min(best, time.perf_counter() - t0)
    return best

html = open("catalog.html", encoding="utf-8").read()  # 1.13 MB

t_html   = best_of(lambda: BeautifulSoup(html, "html.parser"))
t_lxml   = best_of(lambda: BeautifulSoup(html, "lxml"))
t_html5  = best_of(lambda: BeautifulSoup(html, "html5lib"))
t_direct = best_of(lambda: lhtml.fromstring(html))

print(f"html.parser : {t_html:.3f}s  ({1.13/t_html:4.1f} MB/s)")
print(f"lxml (soup) : {t_lxml:.3f}s  ({1.13/t_lxml:4.1f} MB/s)")
print(f"html5lib    : {t_html5:.3f}s  ({1.13/t_html5:4.1f} MB/s)")
print(f"lxml direct : {t_direct:.3f}s  ({1.13/t_direct:4.1f} MB/s)")

On my test VM the results were:

html.parser : 1.329s  ( 0.9 MB/s)
lxml (soup) : 0.806s  ( 1.4 MB/s)
html5lib    : 2.644s  ( 0.4 MB/s)
lxml direct : 0.045s  (25.2 MB/s)

Parsing the same 1.13 MB page. The gap between lxml-through-BeautifulSoup and lxml-direct is the cost of building BeautifulSoup's Python object tree — the C parser is not the bottleneck.

Read that chart carefully, because the interesting result is the middle, not the edges. lxml through BeautifulSoup is only about 1.6x faster than pure-Python html.parser on this page. That's not because libxml2 is slow — lxml parsed the same bytes directly in 0.045 seconds, eighteen times faster than the BeautifulSoup-backed version. The cost is BeautifulSoup's object model.

When you call BeautifulSoup(html, "lxml"), libxml2 does its fast C parse, and then BeautifulSoup walks the resulting C tree and constructs a Python Tag and NavigableString for every single node. On a page with tens of thousands of nodes, that Python-object construction dominates the runtime. The parser is fast; the wrapper is not.

This changes how you should think about parse performance:

Through BeautifulSoup, parser choice matters less than you think. The wrapper is the bottleneck, so lxml vs html.parser through BeautifulSoup is a 1.6x difference, not a 10x one. You pick lxml for its forgiving tree and XPath, not primarily for speed.
If parse speed is actually your bottleneck, drop the wrapper. Use lxml.etree or lxml.html directly and read results out of the C tree. 25 MB/s instead of 1.4 MB/s. Or use selectolax, a Cython wrapper over the Lexbor browser engine, which also keeps its tree in C and measured 11 MB/s on this page with a pleasant BeautifulSoup-ish API.
The wrapper is a developer-speed feature, not a runtime-speed feature. You pay in CPU to buy readable code and soup.select() ergonomics. For most jobs that's a great trade. For a million-page crawl it's the difference between one hour and twenty hours of pure parse time, and it's worth engineering around.

Let me put that last point in concrete terms. Parsing 100,000 pages of roughly 1 MB each:

html.parser : ~32.6 hours of pure parse time
lxml (soup) : ~19.8 hours
selectolax  : ~2.5 hours
lxml direct : ~1.1 hours

That's a weekend of CPU saved by reaching one level down the stack. If your crawl is small — a few thousand pages — none of this matters; any parser is fine and the wrapper's readability wins. At scale it's a real budget line.

Pure parse time for a 100,000-page crawl at 1 MB per page, measured on the same VM. Fetching and network will dwarf parsing for most sites — but not for bandwidth-cheap, parse-heavy workloads like bulk HTML dumps.

CSS selectors vs XPath

BeautifulSoup's select() and select_one() handle the overwhelming majority of CSS selectors you'll actually write:

soup.select("article.post h2 a")          # descendant
soup.select("div.results > ul > li")      # direct child
soup.select("a[href^='/p/']")             # attribute starts-with
soup.select("a[href*='/archive/']")       # attribute contains
soup.select("li:nth-of-type(2)")          # positional
soup.select("span.price, span.sale")      # union
soup.select_one("#price")                 # single match, None if absent

CSS is concise, familiar to anyone who's written front-end code, and select() returns a list in document order, which is what you usually want. For 90% of extraction tasks, CSS is the right tool, and soup.select_one() with a fallback is the safest call in your toolkit.

But CSS has hard limits, and they show up exactly where scraping gets interesting:

CSS cannot go up the tree. There is no parent selector. If you've matched a <td> and want the row, or a link and want the article, you're stuck re-searching or walking .parent in a loop.
CSS cannot select by text content. There is no :has-text(), no "the row whose cell says Total." :contains() existed in jQuery but not in CSS, and BeautifulSoup doesn't support it.
CSS cannot select an element based on the content of a sibling or descendant without knowing the exact structural path to it.

XPath solves all three, and it's available the moment you parse with lxml:

from lxml import html as lhtml

tree = lhtml.fromstring(resp.text)

# every <a> whose text contains "archive"
links = tree.xpath('//a[contains(text(), "archive")]/@href')

# the table row that contains a cell with the text "Total"
row = tree.xpath('//tr[td[contains(text(), "Total")]]')

# the parent of whatever element has id="price"
parent = tree.xpath('//*[@id="price"]/..')

# all products whose price attribute starts with "$9"
prices = tree.xpath('//span[starts-with(@class, "price")]')

# the third product card, by index
third = tree.xpath('(//article[contains(@class, "product")])[3]')

Notice what those expressions can do that CSS cannot. Going up the tree with ... Selecting rows by what's inside them (//tr[td[contains(text(), "Total")]]). Grabbing attributes directly into the result list with @href. Indexing the match set with [3]. Matching text content anywhere inside a node with contains(. , "phrase") instead of contains(text(), ...) — a subtle but crucial distinction when the node you're testing has nested children.

The text() vs . distinction deserves a paragraph of its own. //a[contains(text(), "archive")] tests only the direct text nodes of each <a>. If the text you want is inside a nested <span>, that expression matches nothing. //a[contains(., "archive")] tests the string value of the whole node, which includes all descendants. When in doubt, use . — it matches what you can see, and it's almost always what you meant.

If you're fighting CSS to express "the row whose cell says X" or "the element that contains this text," stop fighting. Parse with lxml.html, write the XPath, move on with your life. The XPath runs in C and is fast; the expressions are compact; and once you've used //tr[td[contains(., "Total")]] to find a summary row, you'll never go back to find_all("tr") plus a Python loop again.

One thing to keep straight: BeautifulSoup does not do XPath. If you call soup.xpath(...) you'll get an AttributeError. XPath means lxml. You have two honest options: use lxml's tree directly (and lose soup.select()), or keep BeautifulSoup and only jump to lxml when a selector is too hard. My preference is the second for most projects — BeautifulSoup's tree API is friendlier for the 90% case, and you don't want to maintain two different navigation styles across your whole codebase.

Writing selectors that survive redesigns

Here's the uncomfortable truth about selectors: they're not queries, they're opinions about a page's structure at a moment in time. The moment a designer renames a class, reorders a column, or swaps a <div> for an <article>, your selector silently starts returning nothing — and, worse, sometimes it silently returns the wrong thing. The entire craft of durable scraping is minimizing how much of your code expresses opinions that can go stale.

The first rule is to anchor on things the site needs to get right, not things it decorates with:

IDs. #price is stable because the page's own JavaScript probably depends on it. Renaming an id breaks the site's own code, so sites rarely do it casually.
data-* attributes. Front-end frameworks and analytics scripts read data-product-id, data-sku, data-tracking. Those attributes exist to be stable references.
Semantic tags. article, time, nav, main, h1–h3, table, td, th. Semantics change only when the content model changes, not when the theme does.
Visible text. A <th> with the text "Price" is a stable anchor for as long as the column exists, no matter how many wrapper classes get shuffled around it.

Avoid, with prejudice: framework noise like col-md-4, mb-3, flex-item, row; hash-based classes like _x9k2a that rebuild on every deploy; and any class that encodes layout ("left-col", "span-8") rather than meaning.

The second rule is to prefer structure over cosmetics. article h2 a describes what the thing is. div.product-listing-wrapper > div:nth-child(3) > span.title-link describes where it sits. Structure survives; cosmetics change.

The third rule is to prefer content checks over positional selectors. nth-of-type(3) breaks the moment a row is inserted above the one you want. Text checks break only when the content meaningfully changes, which is when you should re-examine the extractor:

# fragile — three independent layout assumptions:
price = soup.select_one(".col-md-4 .row div:nth-of-type(3) span")

# robust — anchored on a th whose text is stable:
price_th = soup.find("th", string="Price")
price = price_th.find_next_sibling("td") if price_th else None

# or, anchored on a data attribute that the page itself uses:
price = soup.select_one("[data-testid='product-price']")

The fourth rule is to capture multiple candidates and decide by content, not by position. If a page has several "you may also like" sections, don't assume which one is first; find all candidates and pick the one whose header text matches:

candidates = soup.select(".widget")
wanted = next((w for w in candidates if w.select_one("h2").get_text(strip=True) == "Related products"), None)

The fifth rule is defensive extraction everywhere a field can be missing. select_one() returns None on no match; node['href'] raises KeyError; node.get('href') returns None. Build the habit of the fallback before you need it:

def field(card, sel, attr=None):
    node = card.select_one(sel)
    if node is None:
        return None
    return node.get(attr) if attr else node.get_text(strip=True)

title = field(card, "h2.title a", "href")
price = field(card, "[data-testid=price]")

The meta-rule: a selector that breaks loudly (returns nothing, you notice, you fix it) is fine. A selector that breaks silently (starts returning a different row, or an empty string that your code happily stores) is a time bomb. Make your extraction fail loudly: log when a required field comes back None, and add a sanity check that total record count doesn't drop by 80% overnight.

Stylized but true to experience: a fragile positional selector dies on the first redesign and stays dead until someone notices; an anchored selector drifts by a few records per thousand and survives years.

Handling malformed HTML

Real HTML is broken. Missing closing tags. Misnested elements. Unencoded ampersands. Duplicated IDs. Raw < in attribute values. <p> tags nested inside <p> tags. If your source data were XML, the parser would refuse it — but HTML was designed to be forgiving, and every parser you can choose makes different decisions about how to be forgiving.

html.parser does modest repair: it closes tags when the tree structure requires it, and it's better at this than many people assume. But its repair model is not the browser's model, and on gnarly pages it can produce trees that surprise you.

lxml repairs more aggressively and, importantly, its tree behaves the way a browser-ish tree behaves for 99% of cases. Its libxml2 HTML parser has been hardened by two decades of real-world abuse. If lxml produces a tree you can select against, you don't need to think about the markup at all.

html5lib implements the actual HTML5 parsing algorithm — the spec browsers implement — so it produces the tree the browser would build, with <tbody> elements inserted where the spec requires, misnested content rearranged per the adoption agency algorithm, and everything else the spec says. This is the tool for the genuinely broken page, the one where lxml's tree and the rendered page disagree about where an element lives.

Here's the thing I wish every tutorial made explicit: the browser's DOM is not the parsed bytes, and html5lib is the closest you get to the browser's DOM without running a browser. When a page has deeply broken markup and your selectors keep missing elements that you can clearly see in DevTools, the gap is often the tree-repair difference. Try:

from bs4 import BeautifulSoup

# lxml chokes, or produces a tree your selectors miss:
soup_lxml = BeautifulSoup(broken_html, "lxml")
print(soup_lxml.select_one("table tbody tr"))  # None — no tbody in raw bytes?

# html5lib repairs per the spec, like a browser would:
soup = BeautifulSoup(broken_html, "html5lib")
print(soup.select_one("table tbody tr"))       # found — browser inserted tbody

A concrete case: raw HTML with <table><tr><td>...</table> has no <tbody>. Browsers insert one. lxml and html.parser may not, depending on the exact bytes, which is why select("table tr") works but select("table tbody tr") fails even though it works in DevTools. html5lib inserts the <tbody> exactly where a browser would, and the selector starts matching.

When a selector returns nothing where you expect a match, don't guess — dump the tree and look:

# print the first 2000 chars of a prettified parse
print(soup.prettify()[:2000])

# list the actual top-level section tags
print([t.name for t in soup.select("section")])

# find every table and print how many rows it has
for i, table in enumerate(soup.find_all("table")):
    print(i, len(table.find_all("tr")))

Usually the element is nested one level deeper or shallower than you assumed, or the site used a <div> where you expected an <article>, or the parser repaired something in a way that moved your target. Dump the tree, see the truth, adjust once.

Navigating the tree without selectors

Selectors are the fast path, but BeautifulSoup's tree API remains the escape hatch for everything selectors can't express cleanly. You should know these cold:

soup.find("article")                 # first <article>
soup.find_all("a", href=True)        # every <a> that has an href
soup.find_all(class_="product")      # class filter — note the trailing underscore
soup.find_all("a", limit=5)          # stop after five
soup.select_one("article h2")        # first match of a CSS selector

node.parent                          # the parent tag
node.find_parent("section")          # nearest ancestor that is a <section>
node.find_parent(class_="card")      # nearest ancestor with this class

node.children                        # direct children iterator
node.descendants                     # every descendant, recursively
node.next_sibling / node.previous_sibling
node.find_next_sibling("td")         # next <td> sibling
node.find_previous("h2")             # nearest <h2> before this node in doc order

The family-tree methods are where the "go up the tree" problems get solved when you don't want to switch to XPath. find_parent("tr") to get the row of a matched <td>. find_next_sibling("td") to walk across a table row. find_previous("h2") to find the section header that precedes a paragraph. Each of these is a one-liner that reads better than the equivalent CSS workaround — and unlike parent chained five times, they don't break when an intermediate wrapper is inserted.

Two gotchas in this API are worth memorizing:

find() and find_all() take a name and filters; find_all(class_="x") uses class_ with an underscore because class is a Python keyword. select() takes a full CSS selector string. Mixing the two APIs in one line (node.find_all("a", class_=re.compile("price"))) works and is often the most readable form — find_all accepts regexes for attributes.
node.contents is the list of direct children (including text); node.children is the same list as an iterator. node.get_text() walks all descendants; node.string is only meaningful when a node has exactly one child and it's text. For a container with markup, get_text() is nearly always what you want.

Extracting text: the part everyone gets wrong

Pulling clean text out of a page is where scrapers quietly produce garbage. The naive calls all have footguns:

el.get_text()                     # fine, but whitespace is raw
el.get_text(strip=True)           # strips, but concatenates inline tags: "HelloWorld"
el.get_text(separator=" ", strip=True)  # usually what you want
el.string                        # only if the node has one text child
el.text                          # direct text only — skips child elements entirely!
"".join(el.strings)              # includes script/style text if still in tree

The two traps are el.text and separator.

el.text on a container returns only the text directly inside that node and skips child elements. For a <div><p>Hello</p><p>World</p></div>, el.text is "\n" or the empty-ish whitespace between children — not "HelloWorld". Beginners hit this constantly. Use get_text().

get_text(strip=True) with no separator concatenates adjacent inline tags: <b>Hello</b> <i>World</i> becomes "HelloWorld" with a space swallowed. get_text(separator=" ", strip=True) joins text nodes with a space and strips each one, which is the right behavior for almost every extraction.

And a critical preprocessing step that most guides skip: remove script and style from the tree before you extract text. Otherwise get_text() on a large subtree includes every embedded JSON blob, every analytics snippet, and every CSS rule:

for tag in soup(["script", "style", "noscript"]):
    tag.decompose()

text = soup.select_one("article").get_text(separator=" ", strip=True)

Then normalize the whitespace you still have, because real text contains tabs, non-breaking spaces, and doubled spaces:

import re

def normalize_text(raw: str) -> str:
    text = raw.replace("\xa0", " ")          # NBSP -> regular space
    text = re.sub(r"\s+", " ", text)         # collapse all whitespace
    return text.strip()

text = normalize_text(article.get_text(separator=" ", strip=True))

Stripping boilerplate is a separate problem with a mechanical answer: extract the smallest element that contains your content, not the whole page. Find the article or main node first, then get text from it. If you must work from the full page, decompose the known-noise nodes (nav, footer, aside, script, style, header) before extracting. Your stored text will be searchable, dedupe-able, and cheap to store — and your downstream NLP or full-text index will thank you.

When the browser sees more than you do

Sooner or later every scraper hits the page where your selector is perfect, verified in DevTools, and returns nothing. The first suspect is JavaScript rendering. The bytes you fetched from requests are the raw server response; the DOM you inspect in DevTools is that response after the browser executed scripts, made XHR calls, and updated the DOM. For any page that renders content client-side, they are different documents.

The discipline that saves you hours: build selectors against the actual bytes you fetched, not against DevTools.

import requests

resp = requests.get(url, headers={"User-Agent": UA})
# Save the raw bytes once, then debug against them:
with open("page.html", "wb") as f:
    f.write(resp.content)

# Now your parse target is exactly what BeautifulSoup will see:
soup = BeautifulSoup(resp.text, "lxml")

If the saved file doesn't contain the element, no parser will find it. Your options, in order of preference:

Find the JSON the page embeds. Many "dynamic" pages ship the data as a window.__INITIAL_STATE__ blob or a JSON script tag. Parsing that JSON is faster and more robust than scraping the DOM that renders from it. Look for <script type="application/json"> and window.__data = {...} before reaching for a browser.
Hit the underlying API directly. If the page renders from /api/products?..., call that endpoint instead. It's their data in the cleanest form. This is not always acceptable under a site's terms — check the robots and the ethics before doing it.
Render with a headless browser. Playwright and the rest, but only for the pages that need it. Browsers are two orders of magnitude more expensive than parsing HTML, so render the minimum set of pages, capture the final DOM or the API responses, and hand clean HTML to your existing parser.

The second suspect when DevTools disagrees with your parse is per-request variation: the site serves a different variant to your bot's user agent, or an A/B test shows you a variant you didn't get. Check by diffing two saved responses. The third suspect is the tree-repair difference discussed above — the browser moved a node during parsing and your selector assumes the byte order.

And one more: build the selector against the parsed tree, not the source text. If the source has <td><b>Price</b></td>, then td:contains('Price') might match a <td> whose visible text is "Price", but td.string is None because there's a child element. When debugging, print [td.get_text() for td in soup.find_all("td")[:10]] — look at what the parser sees, not what you think is there.

Performance at scale: parsing 100k pages

When parsing stops being free and becomes a budget line, the rules change. Here's the ordered list of things that actually move the needle, from biggest to smallest:

1. Don't re-parse. This is the biggest and most overlooked win. Cache the parsed tree or the extracted records, not just the raw bytes. If you parse the same page twice — once to explore, once to extract, once to re-extract after a bug fix — you've tripled your parse budget. Keep raw HTML on disk, parse it once, write structured records.

2. Skip the wrapper when you're CPU-bound. If profiling says bs4.BeautifulSoup(...) is a real fraction of your runtime, switch the hot path to lxml.etree or selectolax. On my benchmark page that's an 18x or 8x parse-time improvement respectively, because the tree stays in C. Both still let you use CSS selectors (tree.cssselect(...), parser.css(...)), and lxml gives you XPath.

3. Prefer find_all over select for tight loops over huge trees. soup.select("article h2 a") compiles and runs a full CSS selector engine (soupsieve). soup.find_all("a") walks the tree directly in Python and can be substantially cheaper for simple cases. For a crawl that runs the same selector a million times, this is free CPU.

# in a hot loop over a big document:
for a in soup.find_all("a", href=True):        # fast path
    ...
# vs
for a in soup.select("article a[href]"):        # selector engine
    ...

4. Parse the smallest document you can. If you only need the table rows, you don't need the 2 MB of script blobs around them. Some sites let you request a slimmed variant (an ?_escaped_fragment_, a print view, an AMP version, a JSON endpoint). Every byte you don't fetch is a byte you don't parse.

5. Parallelize the right way. Parsing is CPU-bound, so threads give you almost nothing (GIL) but processes scale linearly. If parse time is your bottleneck, split the page list across N worker processes. If fetching is your bottleneck — and it usually is — then threads or async are the right tool, and parsing each response in-process is free relative to the network.

6. Keep the parser warm. Instantiating BeautifulSoup is cheap; the tree building is the cost. Don't build one soup per row inside a loop over a single document — build one soup per document and iterate its nodes. (A for card in soup.select(".product") then one soup.select_one per field inside is correct; building a new soup per card is not.)

`soup.select` vs `find_all`: the measured difference

I claimed above that find_all is cheaper than select in tight loops. Here's the receipt, measured on the same 2,600-card catalog page, best of 25 calls:

soup.select("article.product h2.title a")        # 181 ms per call
soup.find_all("article", class_="product")       #  36 ms per call
soup.find_all("a")                               #   9 ms per call
tree.xpath('//article[contains(@class,"product")]//h2/a')  # 28 ms
sel.css("article.product h2.title a")            #   8 ms per call

The headline number is the 181 milliseconds for soup.select(). Soupsieve is a faithful, fully-featured CSS engine, and faithfulness costs: parsing and executing the selector against tens of thousands of nodes in Python adds up. The same query expressed as find_all("a") — which skips the selector machinery and just walks the tree — costs 9 milliseconds. That's a 19x difference for a result any of these calls would produce. Across a 100,000-page crawl that's roughly five hours of soupsieve versus fifteen minutes of find_all, and the gap widens when a page runs several selectors.

The rule I now operate by: for a plain tag or class lookup, use find_all(). Spend select()'s engine only on selectors that genuinely need CSS features — attribute matching, sibling combinators, nth-of-type, unions. And if your real bottleneck is selection speed while you still want CSS syntax, the C-backed engines leave soupsieve in the dust: lxml's XPath ran the query in 28 milliseconds and selectolax's css() in 8. The CSS engine you choose matters as much as the tree you parse it against.

The parse-time growth with page size is linear — twice the bytes, twice the time — but the constant differs by an order of magnitude between parsers:

All four lines are linear — same slope family, wildly different constants. The flat lines at top are the pure-Python and soup-wrapper parsers whose constants make a 2 MB page cost seconds; lxml-direct's line barely rises above the axis until the page is enormous.

Common gotchas that eat your weekend

Every one of these has cost me a real debugging session. Skim them now, remember them later.

The class_ keyword. soup.find_all(class="x") is a SyntaxError because class is a Python keyword. It's class_. And soup.find_all("div", {"class": "x"}) works too. The class attribute is special in BeautifulSoup: soup.select(".product") matches the class token exactly, while soup.find_all(class_="product") matches elements whose class attribute contains the token. A class attribute with multiple tokens (class="product sale") matches both.

Self-closing and void tags. HTML void elements (img, br, meta, link, input) don't take closing tags. <img src="x"/> parses fine everywhere, but <div/> does not mean an empty div — in HTML it means an open <div> that swallows everything until a closing tag appears. If you see a div element containing half the page, this is why. html.parser and lxml treat <div/> differently; only the HTML5 spec's rule (html5lib) is predictable for authors. Don't write self-closing non-void tags in markup you control, and don't be surprised when scraped markup does.

Encoded entities and stray ampersands. A & B in raw HTML is invalid but universal. Parsers tolerate it; the text you extract will contain & or, if the source was double-encoded, & which you'll see literally as "&". When comparing extracted text to a known value, normalize entities: html.unescape(text) from the stdlib html module, and remember that get_text() does not unescape for you in every path — test your actual output.

Encoding and the BOM. requests guesses encoding from headers and bytes, and resp.text uses its guess. If text comes back mojibake, it's an encoding mis-detection: decode yourself with resp.content.decode(resp.encoding or "utf-8", errors="replace"), and beware UTF-8 BOMs — "" at the start of a string will quietly break exact-match lookups. Strip it: text.lstrip("").

Unicode normalization. "café" can be stored as café (precomposed) or café (decomposed). If you dedupe by text, normalize first: unicodedata.normalize("NFC", text). This bit me on a product catalog where half the entries were precomposed and half decomposed, producing "duplicate" rows.

Comments and CDATA.  nodes appear in the tree as Comment objects, not text. get_text() skips them (good), but find_all(text=True) includes them (usually not what you want). Filter: [s for s in soup.find_all(text=True) if not isinstance(s, Comment)].

Namespaces. XHTML pages served with application/xhtml+xml can have xmlns attributes. lxml's HTML parser generally ignores namespaces when parsing HTML documents, but if you parse XHTML as XML with lxml.etree.fromstring, every tag becomes {http://www.w3.org/1999/xhtml}html, and every XPath needs the namespace prefix declared. When in doubt, parse XHTML as HTML.

Duplicated IDs. HTML spec says IDs are unique; real pages violate it. soup.select_one("#price") returns the first match, and lxml's XPath //*[@id="price"] returns all of them. If a page has two elements with id="price", decide whether you want the first, the last, or the one inside a particular container.

get_text() vs string vs text. One more time, because it's the most common silent bug: node.string is None for any element with more than one child; node.text skips descendants; get_text() is the reliable full-text call.

HTML tables: the scraper's best friend

Tables are the most structured content on the web, and the most scraped. A well-formed table is self-describing: the <th> header cells tell you what every column holds, which gives you an anchor that survives redesigns better than almost any other page structure. The pattern that keeps working is to locate the column by its header text, then walk the cells of that column across rows — never assume a column is at index 2:

table = soup.select_one("table.prices")
headers = [th.get_text(strip=True) for th in table.select("tr th")]
price_col = headers.index("Price") if "Price" in headers else None

rows = []
for tr in table.select("tbody tr"):
    cells = tr.find_all("td")
    if not cells or price_col is None or price_col >= len(cells):
        continue
    rows.append({
        "product": cells[0].get_text(strip=True),
        "price": cells[price_col].get_text(strip=True),
    })

The header-anchored index degrades gracefully. If a column is renamed or removed in a redesign, you catch one ValueError or get price_col = None and notice immediately — you don't silently start writing the wrong column into your database. That's the difference between a break you see and a break you don't.

Three table-specific traps: header cells sometimes live in a separate <thead> while the rows are in <tbody>, and a browser-inserted <tbody> may or may not exist in your parser's tree depending on which backend you chose (html5lib inserts it, lxml often does not — one more reason the table tbody tr selector works in DevTools but not in your soup). Cells with a colspan attribute shift every column to their right, so a row with one colspan="2" cell has one fewer <td> than the header has <th>s — check len(cells) against the header count before indexing. And some sites nest whole tables inside a table cell; if your extraction suddenly includes a sub-table's rows, filter by walking only the row elements that are direct children of your target <tbody>. For the truly pathological table markup, the XPath //tr[td[contains(., "Total")]] approach from the XPath section is often the fastest way to a summary row that CSS would make you loop over.

Putting it together: a fetch → parse → extract → store pipeline

All of the above composes into one loop you can run on a thousand or a million pages. Here's the shape I keep returning to, with the pieces we've discussed wired together:

import hashlib, json, re, pathlib, time
import requests
from bs4 import BeautifulSoup
from lxml import html as lhtml

CACHE = pathlib.Path("pages"); CACHE.mkdir(exist_ok=True)

def cache_path(url):
    return CACHE / (hashlib.sha256(url.encode()).hexdigest() + ".html")

def fetch(url, headers):
    p = cache_path(url)
    if p.exists():                       # cache-first: never re-fetch
        return p.read_text(encoding="utf-8", errors="replace")
    time.sleep(0.5)                      # politeness
    resp = requests.get(url, headers=headers, timeout=30)
    resp.raise_for_status()
    body = resp.content.decode(resp.encoding or "utf-8", errors="replace")
    p.write_text(body, encoding="utf-8")
    return body

def normalize_text(raw):
    raw = raw.replace("\xa0", " ")
    return re.sub(r"\s+", " ", raw).strip()

def extract_product(html_str):
    # html5lib fallback if lxml's tree is unusable:
    soup = BeautifulSoup(html_str, "lxml")
    if soup.select_one("article.product") is None:
        soup = BeautifulSoup(html_str, "html5lib")
    for tag in soup(["script", "style"]):
        tag.decompose()
    out = []
    for card in soup.select("article.product"):
        name = card.select_one("h2.title a")
        price = card.select_one("[data-testid=price]") or card.select_one(".price")
        row = {
            "title": name.get_text(strip=True) if name else None,
            "url": name.get("href") if name else None,
            "price": normalize_text(price.get_text(separator=" ", strip=True)) if price else None,
        }
        if any(row.values()):
            out.append(row)
    return out

def run(urls, headers):
    for url in urls:
        html_str = fetch(url, headers)
        records = extract_product(html_str)
        with open(f"out/{hashlib.sha256(url.encode()).hexdigest()}.json", "w") as f:
            json.dump(records, f)

# For the XPath-shaped problems, swap the extractor:
def extract_with_xpath(html_str):
    tree = lhtml.fromstring(html_str)
    rows = tree.xpath('//tr[td[contains(., "Price")]]')
    return [row.xpath('string(./td[2])').strip() for row in rows]

The pipeline rules, in one breath: cache raw bytes so you never re-fetch; sleep between requests; parse once per document; drop noise nodes before text extraction; anchor selectors on data attributes and text; fall back to html5lib when lxml's tree is wrong; keep the XPath escape hatch ready for the "row whose cell says X" problems; and store structured records, not the soup. When the site redesigns, you fix one dictionary or one XPath, re-run against the cache, and your output records are complete before you've made a single new request to the target.

If your target site is hostile to scraping, the parsing layer is only part of the story — see our guide on bypassing anti-bot protections and the ethics piece on robots.txt. And if you're weighing building this pipeline versus buying an extraction layer, the further reading at the bottom has the numbers.

Key takeaways

Default to the lxml parser. It's the most forgiving of the fast parsers and gives you XPath for free.
html.parser is the no-dependencies fallback; html5lib is the last resort for genuinely broken markup — it's the slowest but the only one that implements the browser's exact repair algorithm.
Through BeautifulSoup, the wrapper's Python object tree dominates parse time: lxml was ~1.6x faster than html.parser on our benchmark, while lxml direct was ~18x faster still. If parse time is a real budget line, drop the wrapper.
Use CSS selectors for the 90% case. Reach for XPath (via lxml) when you need to go up the tree, select by text content, grab attributes, or index matches.
Anchor selectors on IDs, data-* attributes, semantic tags, and visible text — never on nth-child chains or framework layout classes. Capture candidates and decide by content.
Debug against the bytes you actually fetched, not DevTools. If the element isn't in the raw HTML, no parser will find it — check for embedded JSON or an API before reaching for a browser.
Extract text with get_text(separator=" ", strip=True) after decompose()-ing script/style, then normalize whitespace and NFC unicode.
At scale: cache parsed output, use find_all for simple loops, process with multiple processes, and prefer lxml.etree or selectolax in the hot path.

Frequently Asked Questions

Which BeautifulSoup parser should I use?

Use lxml for speed. It's an order of magnitude faster than the stdlib html.parser. Use html.parser only when you can't install native dependencies. Use html5lib when the markup is so broken that lxml chokes. It's the slowest but the most forgiving.

Should I use CSS selectors or XPath with BeautifulSoup?

BeautifulSoup supports CSS selectors natively via soup.select(). XPath is only available through lxml. XPath is more powerful for going up the tree and selecting by text content. Use lxml directly when you need XPath. Use BeautifulSoup when CSS is enough and readability matters.

Why does my selector work in the browser but not in BeautifulSoup?

The browser DOM is not the raw HTML. JavaScript added elements. The browser normalized the markup. Or the page serves different HTML to bots. Print the actual bytes you received and build selectors against that, not against DevTools.

Is BeautifulSoup faster than using lxml or selectolax directly?

No. BeautifulSoup is a wrapper: the parse speed is set by the backend, but the Python object tree it builds costs most of the time. In our benchmark on a 1.13 MB page, lxml through BeautifulSoup was about 1.6x faster than html.parser, while using lxml's etree directly was about 18x faster than that and selectolax about 8x faster.

Why does soup.select() return nothing when I can see the element in the browser?

Usually one of three things: the page renders content with JavaScript, so the raw bytes you fetched don't contain the element; the browser auto-corrected broken markup, so the parsed tree shape differs from the bytes; or the site serves a different variant to bots. Save the exact bytes you fetched and debug against those.

How do I extract text without script and style noise?

Drop the noise nodes first with soup(['script', 'style']).decompose(), then call get_text(separator=' ', strip=True) on the element that holds your content. Finally normalize whitespace with a regex. That gives clean, searchable text from a messy subtree.

Keep reading

Found this useful? Cite it as: webscraping.space. “Parsing HTML with BeautifulSoup: selectors, speed, and survival.” https://webscraping.space/blog/parsing-html-with-beautifulsoup. Published 2026-07-02.