Parsing Published Jul 2, 2026 · 2 min read · 492 words

Parsing HTML with BeautifulSoup: selectors, speed, and survival

A deep look at parsing HTML with BeautifulSoup and lxml in Python: CSS vs XPath, handling malformed markup, the speed difference between parsers, and writing selectors that survive small redesigns.

Everyone thinks parsing is the hard part of scraping. It isn't. Fetching reliably is. But once you have the HTML, how you parse it decides whether your scraper survives a minor redesign or breaks every Tuesday. This guide is about writing selectors that last.

The three parsers

BeautifulSoup is a wrapper. The actual parsing is done by one of three backends:

from bs4 import BeautifulSoup

BeautifulSoup(html, "html.parser")  # stdlib, pure Python, slow
BeautifulSoup(html, "lxml")         # C, fast, strict-ish
BeautifulSoup(html, "html5lib")     # pure Python, slowest, most forgiving

Rule of thumb: default to lxml. Fall back to html.parser only when you can't install native deps. Reach for html5lib when the markup is so broken that lxml chokes.

CSS selectors cover 90% of cases

BeautifulSoup's select() handles most CSS selectors:

soup.select("article.post h2 a")          # descendant
soup.select("div.results > ul > li")      # direct child
soup.select("a[href^='/p/']")             # attribute starts-with
soup.select("li:nth-of-type(2)")          # positional
soup.select("span.price, span.sale")      # union

For a single match, use select_one(). For text, get_text(strip=True).

When you need XPath

CSS can't go up the tree. CSS can't select by text content. XPath can:

from lxml import html as lhtml

tree = lhtml.fromstring(resp.text)
# all <a> whose text contains "archive"
links = tree.xpath('//a[contains(text(), "archive")]/@href')
# the table row that contains a cell with "Total"
row = tree.xpath('//tr[td[contains(text(), "Total")]]')
# parent of the element with id="price"
parent = tree.xpath('//*[@id="price"]/..')

If you're fighting CSS to say "the row whose cell says X", switch to lxml and XPath. You'll be happier.

Robust selectors

Selectors rot. Minimize the rot. Anchor on stable things:

1. Prefer id, data-* attributes, and semantic tags (article, time, h1) over class names like col-md-4 mb-3. Those are framework noise. They change on every redesign.
2. Prefer structure over styling: article h2 a beats div.classList-name.
3. Prefer text content checks over positional selectors. nth-of-type(3) breaks when a row gets inserted.
4. Capture multiple candidates and pick by content, not by position.

# fragile:
price = soup.select_one(".col-md-4 .row div:nth-of-type(3) span")

# robust:
cell = soup.find("th", string="Price")
price = cell.find_next_sibling("td") if cell else None

Handling malformed HTML

Real HTML is broken. Missing closing tags. Duplicated IDs. Unencoded ampersands. lxml and html5lib both auto-repair. html.parser is less aggressive. If select() returns nothing where you expect a match, dump the tree:

print(soup.prettify()[:2000])
print([t.name for t in soup.select("section")])

Usually the element is nested one level deeper or shallower than you thought. Or the site used a <div> where you expected an <article>.

Extracting text vs HTML

el.get_text(strip=True)        # visible text, whitespace collapsed
el.get_text(separator=" ", strip=True)
"".join(el.strings).strip()    # includes script/style text. usually not what you want

Avoid el.text for containers. It returns only direct text and skips child elements. Use get_text() for the full visible text of a subtree.

A reusable extraction helper

from bs4 import BeautifulSoup

def extract_list(html, card_sel, fields):
    soup = BeautifulSoup(html, "lxml")
    out = []
    for card in soup.select(card_sel):
        row = {}
        for name, (sel, attr) in fields.items():
            node = card.select_one(sel)
            if node is None:
                row[name] = None
                continue
            row[name] = node[attr] if attr else node.get_text(strip=True)
        out.append(row)
    return out

records = extract_list(
    html,
    "article.product",
    {
        "title": ("h2 a", None),
        "url":   ("h2 a", "href"),
        "price": (".price", None),
    },
)

The fields map makes your extractor declarative. When the site redesigns, you change one dictionary, not twenty lines of logic.

Speed: when it matters

For a few hundred pages, any parser is fine. For millions, lxml is worth it:

# parse 10 MB of HTML 1000x
# html.parser: ~8.4 s
# lxml:        ~0.7 s

If you're CPU-bound on parsing, also look at selectolax. It's a Cython wrapper over Lexbor. Faster again, with a BeautifulSoup-ish API.

Key takeaways

• Use lxml by default. Keep html.parser as a pure-Python fallback.
• Anchor selectors on stable attributes, not framework classes.
• Switch to XPath when you need to go up the tree or select by text.
• Make extraction declarative so a redesign is a one-dict fix, not a rewrite.

Found this useful? Cite it as: webscraping.space. “Parsing HTML with BeautifulSoup: selectors, speed, and survival.” https://webscraping.space/blog/parsing-html-with-beautifulsoup. Published 2026-07-02.