Parsing Published Aug 14, 2026 · 37 min read · 8,048 words
LLM Extraction: Structured Data from HTML (2026 Guide)
The 2026 guide to LLM extraction: when it beats CSS selectors, schema design, token cost math, validation, retries, and a complete Python pipeline.
Every scraper eventually hits the wall that CSS selectors cannot climb. The page renders fine in a browser, the data is right there on screen, and yet no selector you write survives contact with the markup — because the markup is a mess, or it changes every week, or the same field lives in three different places depending on which A/B variant you got. This is the 2026 guide to the tool that climbs that wall: LLM extraction.
LLM extraction is exactly what it sounds like. You feed a language model the text of a web page plus a schema, and it returns structured JSON. No selectors, no XPath, no regex. The model reads the page the way a human would and writes the answer into the shape you asked for. It is the fastest-growing parsing technique in production scraping, and it is also the most misunderstood — people either treat it as magic or dismiss it as a toy, and both reactions are wrong.
This guide is the honest middle. We will cover what LLM extraction actually is, when it beats CSS selectors and when it is a waste of money, how to design the schema that makes it reliable, the real token cost per page, how to validate output so garbage never reaches your database, the full pipeline from fetch to retry, and a complete working Python example you can run today. Where the honest answer is "selectors are free and fast, use them," we will say so. Where the honest answer is "your selectors are already broken, switch," we will say that too.
We run these pipelines in production. The numbers in this post are the numbers we see, and the code is the code we run.
Key takeaways
- LLM extraction = page text + schema → JSON. No selectors, no XPath, no regex. The model reads the page and writes the answer into your shape.
- It wins on messy pages and layout churn; it loses on clean, stable pages. Selectors are free, run in microseconds, and hold about 99% accuracy on stable structure. Use them there.
- The cost is real but small. Roughly 4,000 input + 500 output tokens per page ≈ $0.004/page ≈ $4 per 1,000 pages at typical rates.
- Schema design is the reliability lever. Required fields, strict types, enums, and validation catch most model errors before they reach your database.
- Validation plus a retry loop takes accuracy from about 85% to about 98%. Never trust a single unvalidated model response.
- The pipeline is fetch → clean → truncate → extract → validate → retry. A scraping API handles the fetch layer; the LLM handles the extraction layer.
- Ethics do not change. Respect robots.txt, rate-limit politely, and do not scrape personal data you do not need.
What LLM extraction is
LLM extraction is a parsing technique where a language model converts the text of a web page into structured data according to a schema you provide. The input is the page's text — ideally cleaned and truncated, though raw HTML works in a pinch. The output is JSON. The schema is the contract between you and the model: it says which fields exist, what types they are, which are required, and what values are allowed.
The mental model that makes this click: a CSS selector is a guess about where data lives. An LLM is a reader. The selector says "the price is the text inside the third span of the second div of the row with class col-md-4." The LLM says "the price is the number next to the currency symbol that represents what this item costs." The first breaks when the page's skeleton changes. The second survives it, because it is anchored on meaning, not position.
That distinction is the entire reason the technique exists. The web is full of pages where the data is perfectly visible to a human and completely hostile to a selector: pages assembled from fragments, pages where the same field appears in multiple templates, pages that A/B test their own layout, pages generated by systems that emit different markup for the same logical content. For those pages, the choice used to be "write a parser per template and maintain it forever" or "give up." LLM extraction is the third option.
The model does not need to be huge. The extraction task — read text, fill in a schema — is one of the easiest things a modern model does, which is why it is also one of the cheapest. A mid-range model like GLM-5.3 handles it comfortably, and the small fast variants of the same families (GLM-5.3-Flash, the small tiers of the other labs) handle it for a fraction of the price. You are not asking the model to reason; you are asking it to read and transcribe. The harder the page is to read, the more the model matters, but for most pages the cheapest capable model is the right one.
The prompt that does the work has three parts, and it is worth knowing them by name because they are the whole interface you maintain. The system prompt sets the behavior: "extract structured data, return only valid JSON, do not invent values." The schema is the contract, embedded in the prompt as JSON. The page text is the payload. You set the temperature to zero, because extraction is a transcription task and you want the same answer every time, not a creative one. That is the entire prompt. When extraction quality degrades, the fix is almost always in the schema or the cleaning, not in a longer system prompt — a lesson that saves a lot of prompt-engineering time.
The other half of the stack is the fetch layer, and it is worth being explicit about the division of labor. LLM extraction assumes you already have the page. Getting the page — handling redirects, JavaScript rendering, anti-bot defenses, proxy rotation — is a separate problem with a separate set of tools. A scraping API handles that layer: you send a URL, it returns HTML or clean content, and you pay per page. Keirolabs is the flat-price outlier at $0.25 per 1,000 pages, with rendering and residential proxies bundled and full markdown output for RAG; ScraperAPI runs about $1.10 per 1,000, ZenRows about $1.40, Firecrawl about $3.20. The point is not that any one of them is the answer — it is that the fetch layer is a solved problem you buy, and the extraction layer is the part you design. This post is about the part you design.
When LLM extraction beats selectors (and when it doesn't)
The first question anyone asks is "is LLM extraction better than CSS selectors?" and the honest answer is "better at what, on what page?" The two techniques are not competitors in the way people assume. They are tools for different regimes, and the boundary between the regimes is measurable.
CSS selectors and regex are the right tool when the page has stable structure. A well-formed product page with a consistent template, a clean article layout, a table with predictable rows — selectors extract those at about 99% accuracy, in microseconds, for zero marginal cost. The maintenance cost is real but bounded: you fix a selector when the site redesigns, and if the site redesigns rarely, that is a small bill. For a clean, stable target, LLM extraction is strictly worse: it costs money per page, it adds latency, and it can hallucinate a field that a selector would simply get right.
The regime flips when the structure stops being stable. Three signals tell you you are there. First, your selectors break on a schedule — every few weeks, a redesign, a class rename, a template change. Second, the same logical field lives in different places on different pages — the price is in one spot on the product page and another on the variant page. Third, the page has no structure to speak of — content assembled from fragments, injected by scripts, or generated by a system that emits different markup for the same content. In that regime, selectors degrade fast, and the degradation is the expensive kind: silent. A selector that returns nothing is annoying; a selector that returns the wrong row is a data-quality disaster.
The chart below is the honest shape of the trade. On clean, stable pages (messiness near zero), selectors hold about 99% accuracy and the LLM sits a few points below — the model occasionally misreads a field that a selector would nail. As the page gets messier, the selector line falls off a cliff, because every layout change breaks a position-based guess. The LLM line barely moves, because the model is reading meaning, not position. The crossover lands around messiness 3 on our scale — roughly the point where a page has had one or two structural changes in the last quarter.
The practical rule we use: if a page has been stable for six months and your selectors have not broken, keep the selectors. If you have fixed the same selector three times this year, or the page is assembled from fragments, or you are scraping a target you do not control and cannot predict, switch to the LLM. The decision is not about which is more sophisticated. It is about which one stops breaking.
There is also a hybrid regime worth naming, because it is where most production systems actually live. Use selectors for the fields that are stable and the LLM for the fields that are not. A product page might have a stable title and price but a description that moves around, or a review page with stable metadata and free-form review bodies. Split the schema: selector-extract what is stable, LLM-extract the rest, and validate the union. This is more code, but it is also the cheapest way to get about 99% on the fields that matter and about 95% on the fields that do not.
Schema design: the reliability lever
The single biggest determinant of whether LLM extraction works in production is not the model. It is the schema. A vague schema produces vague output; a precise schema produces precise output. The model is doing what you asked — the question is whether you asked for something checkable.
A good extraction schema is a JSON Schema document with three properties that do most of the work. First, required — the fields without which a record is useless. If a product record without a price is garbage, put price in required. Second, strict types — number for prices, integer for counts, string for names. A model that returns "price": "19.99" as a string when you asked for a number has failed, and the validator should say so. Third, enum — the closed set of values a field may take. Availability is not a free-text field; it is one of in_stock, out_of_stock, or preorder. When you constrain the model to an enum, you convert a fuzzy judgment into a checkable fact.
The other schema decision that matters is additionalProperties. Set it to false. It forces the model to return exactly the fields you asked for and nothing else, which makes the output predictable and the validation meaningful. A model that is free to add fields will add them — a "notes" field here, an extra key there — and every extra key is a place where a downstream consumer can break.
The validation flow is where the schema earns its keep. The model returns JSON; you parse it; you check that required fields are present; you check that types and enums match; you check business rules that the schema cannot express — a price is non-negative, a rating is within 0 to 5, a date is not in the future. Each check is a few lines of code and microseconds of compute. Together they are the difference between "the model is usually right" and "the database only ever sees valid records."
The business-rule layer matters more than people expect, because JSON Schema cannot express everything. A schema can say rating is a number between 0 and 5; it cannot say "this product's rating should not be 4.9 on a page with three reviews." Those rules live in your code, and they are where the last few percent of data quality come from. The pattern is always the same: schema for shape, code for meaning, and anything that fails either one goes back to the model with the error attached.
Cost per page: the token math
LLM extraction is not free, and the honest way to think about the cost is per page, in tokens. A typical page, after cleaning and truncation, costs about 4,000 input tokens — the page text plus the schema and the system prompt. The model's answer costs about 500 output tokens. That is the budget, and it is remarkably stable across content types once you clean the page, because the schema and prompt are a fixed overhead and the page text is capped by truncation.
The price of those tokens depends on the model. At typical mid-range rates — roughly $0.75 per 1M input tokens and $2.00 per 1M output tokens, the band GLM-5.3 and its peers sit in — a page costs about $0.004. That is $4 per 1,000 pages. On a small fast model at $0.15 in / $0.60 out, the same page costs about $0.0009 — under a dollar per 1,000 pages. On a frontier model at $3 in / $15 out, it is about $0.02 per page, $20 per 1,000. The spread is a factor of twenty, which is why model choice is a cost decision, not a quality decision, for most extraction workloads.
Two things make the cost story better than it looks. First, the fetch layer is a rounding error next to the extraction layer. A scraping API adds $0.25 to $1.10 per 1,000 pages on top of the LLM cost — at the typical tier, fetch is about 6% of the total bill. Second, the cost only applies to pages you actually extract. If you use selectors for the stable fields and the LLM only for the unstable ones, you pay for a fraction of a page's tokens, not the whole page.
The cost also has a ceiling you control: truncation. A raw, uncleaned page can be 9,000 tokens or more; a cleaned and truncated page is 4,000. Cleaning is free — it is a few lines of BeautifulSoup — and it cuts the bill by more than half. Truncation is a budget decision: you decide how much of the page the model gets to see, and you accept that fields beyond the cutoff may be missed. The right cutoff is the smallest one that still captures the fields you need, and you find it by testing, not by guessing.
One more line item belongs in the budget, and it is the one people forget until the invoice arrives: retries. Every failed validation costs another full page of tokens, because the repair prompt resends the page text plus the error context. At an 85% first-attempt pass rate, the expected cost per page is not the single-attempt price — it is the price times the expected number of attempts, which lands around 1.2 attempts for a two-retry budget. That is a 20% adder on the extraction bill, and it is worth modeling before you commit to a volume, not after.
Accuracy and how to validate
The accuracy question is the one everyone asks, and the most honest answer available is: it depends on the page, but with validation and retries, production systems routinely land in the 95-98% range on fields that matter, and the failures that remain are caught, not stored. The key phrase is "caught, not stored." The entire discipline of LLM extraction is making sure that when the model is wrong, the wrongness is detected before it reaches your database.
The first line of defense is the schema validation described above. The second is a retry loop. When validation fails — a required field is missing, a type is wrong, an enum value is out of range — you do not accept the output. You send it back to the model with the validation errors attached and ask for a corrected version. This is the "repair" step, and it is remarkably effective, because most model errors are not deep reasoning failures; they are transcription slips that the model fixes instantly when told what it got wrong.
The numbers we see in production: a single attempt passes validation about 85% of the time on typical pages. A second attempt, with the errors fed back, pushes the cumulative pass rate to about 95.5%. A third takes it to about 98%. Each retry costs another ~4,500 tokens — the page text plus the error context — which is why the retry budget is a cost decision. Two retries is the sweet spot for most workloads; beyond that, the marginal gain is smaller than the marginal cost, and the right move is to log the failure and move on.
The third line of defense is sampling and spot-checking. Even a validated record can be wrong in a way the schema cannot see — the model extracted the price of the wrong product, or the description of the wrong variant. The schema catches shape errors; it cannot catch meaning errors. The only defense is human or programmatic review of a sample: pull 1% of records, check them against the source pages, and track the error rate. When the error rate on a target climbs, that is the signal that the page changed and your prompt or schema needs attention.
The accuracy ceiling is also a function of what you ask for. Fields that are unambiguous — prices, dates, IDs, stock status — extract at very high accuracy, because there is one right answer and the model can see it. Fields that are judgment calls — "the main topic of this article," "the sentiment of this review" — extract at lower accuracy, because reasonable humans disagree. Design your schema around the unambiguous fields, and treat the judgment fields as best-effort with a human review path.
The way to know your actual accuracy, rather than the number in a blog post, is to measure it on your own pages. Build a labeled sample once: take fifty pages from each target, extract the fields by hand, and store them as ground truth. Then run your pipeline over the same pages and compare. That gives you a per-field accuracy number, which is the number that matters — a pipeline that is 99% on price and 80% on description is a different product from one that is 95% on both. Re-run the sample when the target changes or the model version moves, and you will see drift before it reaches your database instead of after.
The pipeline: fetch → clean → truncate → extract → validate → retry
The full pipeline is six stages, and the discipline is in the first three, which are cheap and boring and determine whether the last three work.
Fetch is the layer you buy. A scraping API takes a URL and returns HTML or clean content, handling rendering, proxies, and anti-bot defenses. You do not build this; you rent it. Clean is where you strip the boilerplate — scripts, styles, navigation, footers, forms — and pull the main content into text. This is the highest-leverage step in the whole pipeline, because it cuts the token count by more than half and removes the noise that makes models misread. Truncate is where you enforce the budget: cap the cleaned text at the token ceiling you chose, so the cost per page is bounded and predictable.
Extract is the LLM call: system prompt, page text, schema, temperature zero, JSON out. Validate is the schema and business-rule check. Retry is the repair loop. The whole thing is about 150 lines of Python, and the example below is the whole thing.
A complete working example
Here is the pipeline as runnable Python. It fetches through a scraping API, cleans with BeautifulSoup, truncates to a token budget, extracts with an OpenAI-compatible model endpoint (GLM-5.3 or any model that speaks the same protocol), validates against JSON Schema, and retries with error context. This is the shape of the code we run in production, minus the logging and the queue.
import json
import re
from typing import Any
import requests
from bs4 import BeautifulSoup
from jsonschema import Draft202012Validator
from openai import OpenAI
# --- 1. FETCH: a scraping API handles the fetch layer ---
# Point this at any scraping API; Keirolabs, ScraperAPI, ZenRows, etc.
FETCH_URL = "https://api.scraper.example/v1/fetch"
API_KEY = "YOUR_KEY"
def fetch_html(url: str) -> str:
resp = requests.get(
FETCH_URL,
params={"url": url, "render": "false"},
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=30,
)
resp.raise_for_status()
return resp.json()["html"]
# --- 2. CLEAN: strip boilerplate, keep the main content ---
def clean_html(html: str) -> str:
soup = BeautifulSoup(html, "html.parser")
for tag in soup(["script", "style", "noscript", "nav", "footer", "header", "aside", "form"]):
tag.decompose()
for tag in soup.find_all(True):
tag.attrs = {}
text = soup.get_text("\n", strip=True)
return re.sub(r"\n{3,}", "\n\n", text)
# --- 3. TRUNCATE: cap the token budget ---
def truncate(text: str, max_chars: int = 16_000) -> str:
# ~4 chars per token is a rough heuristic; 16k chars ≈ 4k tokens
return text[:max_chars]
# --- 4. EXTRACT: LLM + schema ---
SCHEMA = {
"type": "object",
"properties": {
"title": {"type": "string"},
"price": {"type": "number"},
"currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
"availability": {"type": "string", "enum": ["in_stock", "out_of_stock", "preorder"]},
"rating": {"type": "number", "minimum": 0, "maximum": 5},
"review_count": {"type": "integer", "minimum": 0},
},
"required": ["title", "price", "currency", "availability"],
"additionalProperties": False,
}
SYSTEM_PROMPT = (
"You extract structured data from web page text. "
"Return ONLY valid JSON matching the provided schema. "
"If a field is missing, use null. Do not invent values."
)
def build_user_prompt(text: str, schema: dict) -> str:
return (
f"Extract the following fields from this page text into JSON.\n"
f"JSON Schema:\n{json.dumps(schema, indent=2)}\n\n"
f"Page text:\n{text}"
)
client = OpenAI(
api_key="YOUR_MODEL_KEY",
base_url="https://api.example.com/v1", # GLM-5.3 or any OpenAI-compatible endpoint
)
def extract(text: str, schema: dict, model: str = "glm-5.3") -> dict:
resp = client.chat.completions.create(
model=model,
temperature=0,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": build_user_prompt(text, schema)},
],
)
content = resp.choices[0].message.content
# Strip code fences if the model wraps the JSON
content = re.sub(r"^`{3}(?:json)?\s*|\s*`{3}$", "", content.strip())
return json.loads(content)
# --- 5. VALIDATE: JSON Schema + business rules ---
validator = Draft202012Validator(SCHEMA)
def validate(result: dict) -> list[str]:
errors = [e.message for e in validator.iter_errors(result)]
if result.get("price") is not None and result["price"] < 0:
errors.append("price must be non-negative")
return errors
# --- 6. RETRY: repair prompt on failure ---
def extract_with_retry(url: str, max_retries: int = 3) -> dict:
html = fetch_html(url)
text = truncate(clean_html(html))
last_errors: list[str] = []
for attempt in range(1, max_retries + 1):
try:
result = extract(text, SCHEMA)
last_errors = validate(result)
if not last_errors:
return result
except (json.JSONDecodeError, ValueError) as exc:
last_errors = [str(exc)]
# Repair: feed the errors back into the next attempt
text = (
f"Your previous output failed validation with these errors:\n"
f"{last_errors}\n\n"
f"Fix them and return corrected JSON.\n\n"
f"Page text:\n{text}"
)
raise RuntimeError(f"Extraction failed after {max_retries} attempts: {last_errors}")
if __name__ == "__main__":
data = extract_with_retry("https://example.com/product/123")
print(json.dumps(data, indent=2))
Let me walk through the parts that matter. clean_html removes the noise nodes and strips attributes, then flattens the page to text. This is the step that cuts your token bill in half. truncate enforces the budget with a characters-to-tokens heuristic — roughly four characters per token — which is good enough for a ceiling. extract builds the prompt, calls the model at temperature zero, and strips code fences if the model wrapped the JSON. validate runs the JSON Schema validator plus a business rule that prices are non-negative. extract_with_retry is the loop: on any failure, it appends the errors to the prompt and tries again, up to three attempts.
The two details that save the most production pain: additionalProperties: false in the schema, which forces the model to return exactly the fields you asked for, and the repair prompt, which hands the model its own errors and asks for a fix. Both are cheap to add and expensive to skip.
When to use it: the decision tree
The decision tree in the chart is the one we actually use. Start with the structure question: is the page stable and clean? If yes, selectors, and do not feel bad about it — they are free and fast and about 99% accurate. If no, ask whether the layout changes often. If it does, LLM extraction. If it does not, ask whether the data lives in a consistent location. If it does, selectors with monitoring. If it does not, LLM extraction.
The volume question cuts across all of it. At very high volume — millions of pages a month — the per-page cost of the LLM starts to matter, and the right answer is often a hybrid: selectors for the stable bulk, the LLM for the messy tail, and a sampling loop that measures both. At low volume — thousands of pages a month — the LLM cost is noise, and the right answer is whatever stops breaking.
Honest caveats
Six caveats, because this technique gets oversold.
First, the LLM can be wrong in ways validation cannot see. Schema validation catches shape errors; it cannot catch a wrong-but-well-formed value. Sample and spot-check.
Second, latency is real. A selector extracts in microseconds; an LLM call takes one to three seconds. At low volume this is irrelevant. At high volume it is a throughput problem, and you will need concurrency.
Third, the cost is per page, forever. Selectors are free after you write them. The LLM bill never goes away, and it scales with volume. The honest comparison is not "free vs $4/1k"; it is "your maintenance time vs $4/1k."
Fourth, model drift is a thing. Models get deprecated, endpoints change, and a model that was great at your pages in January may be worse in June. Pin your model version and re-test on a sample when the provider changes anything.
Fifth, the technique does not change the ethics. LLM extraction is a parsing method, not a license. Respect robots.txt, rate-limit politely, prefer an official API or dataset when one exists, and do not scrape personal data you do not need. The web scraping ethics guide covers the lines; the short version is that a smarter parser does not make a scrape more allowed.
Sixth, the schema is a maintenance surface. When the target changes, your schema may need to change too — new fields, new enums, new required fields. Budget for it the way you budget for selector maintenance.
None of these changes the verdict. They change how confidently you can run it unattended, and that distinction is the whole job.
FAQ
What is LLM extraction? LLM extraction is a parsing technique where a language model reads the text of a web page and returns structured JSON according to a schema you provide. You feed it the page text plus a JSON Schema describing the fields you want, and it returns a JSON object with those fields filled in. No CSS selectors, no XPath, no regex — the model reads the page the way a human would and writes the answer into your shape.
Is LLM extraction better than CSS selectors? It depends on the page. On clean, stable pages, CSS selectors are better: they are free, run in microseconds, and hold about 99% accuracy. LLM extraction wins on messy pages, pages whose layout changes frequently, and pages with no stable structure — the regime where selectors break and break silently. The crossover is around the point where a page has had one or two structural changes in a quarter. Most production systems use a hybrid: selectors for stable fields, the LLM for the rest.
How much does LLM extraction cost? A typical page costs about 4,000 input tokens and 500 output tokens. At typical mid-range rates that is about $0.004 per page — $4 per 1,000 pages. A small fast model cuts that to under $1 per 1,000 pages; a frontier model raises it to about $20 per 1,000. The fetch layer adds $0.25 to $1.10 per 1,000 pages on top. Cleaning and truncation cut the bill by more than half.
How do I extract structured data from HTML with an LLM? The pipeline is fetch, clean, truncate, extract, validate, retry. Fetch the page (a scraping API handles this layer), strip the boilerplate with BeautifulSoup, truncate the text to your token budget, call the model with the page text and a JSON Schema, validate the output against the schema, and retry with the validation errors fed back if it fails. The complete Python example in this post is the whole thing in about 150 lines.
How accurate is LLM extraction? With validation and a retry loop, production systems routinely land in the 95-98% range on well-defined fields. A single attempt passes validation about 85% of the time; a second attempt with errors fed back pushes it to about 95.5%; a third to about 98%. Fields with one unambiguous right answer — prices, dates, IDs, stock status — extract at the high end. Judgment fields extract lower, and the failures that remain are caught by sampling and spot-checking, not stored.
What is the best LLM for HTML extraction? For most workloads, the cheapest capable model is the right one, because extraction is reading and transcribing, not reasoning. GLM-5.3 and its peers handle it comfortably, and the small fast variants of the same families handle it for a fraction of the price. The model matters more as the page gets harder to read. Pin your model version and re-test on a sample when the provider changes anything.
How many tokens does it take to extract data from a page? About 4,000 input tokens and 500 output tokens for a typical cleaned page. The input is the page text plus the schema and system prompt; the output is the JSON answer. A raw, uncleaned page can be 9,000 tokens or more, which is why cleaning and truncation are the highest-leverage cost controls in the pipeline.
Is LLM extraction legal and ethical? LLM extraction is a parsing method, and it does not change the legality or ethics of a scrape. The same rules apply: respect robots.txt, rate-limit politely, prefer an official API or dataset when one exists, and do not scrape personal data you do not need. A smarter parser does not make a scrape more allowed.
Further reading
- GLM-5.3 model review — the extraction-layer model we reference throughout this post, reviewed honestly.
- Web scraping APIs in 2026 — the fetch layer, priced and compared.
- Web scraping tools — the tooling landscape around the pipeline.
- Web scraping with Python — the language this pipeline is written in.
- Scraping APIs for AI agents — how agent loops consume web content at scale.
Frequently Asked Questions
What is LLM extraction?
LLM extraction is a parsing technique where a language model reads the text of a web page and returns structured JSON according to a schema you provide. You feed it the page text plus a JSON Schema describing the fields you want, and it returns a JSON object with those fields filled in. No CSS selectors, no XPath, no regex — the model reads the page the way a human would and writes the answer into your shape.
Is LLM extraction better than CSS selectors?
It depends on the page. On clean, stable pages, CSS selectors are better: they are free, run in microseconds, and hold about 99% accuracy. LLM extraction wins on messy pages, pages whose layout changes frequently, and pages with no stable structure — the regime where selectors break and break silently. The crossover is around the point where a page has had one or two structural changes in a quarter. Most production systems use a hybrid: selectors for stable fields, the LLM for the rest.
How much does LLM extraction cost?
A typical page costs about 4,000 input tokens and 500 output tokens. At typical mid-range rates that is about $0.004 per page — $4 per 1,000 pages. A small fast model cuts that to under $1 per 1,000 pages; a frontier model raises it to about $20 per 1,000. The fetch layer adds $0.25 to $1.10 per 1,000 pages on top. Cleaning and truncation cut the bill by more than half.
How do I extract structured data from HTML with an LLM?
The pipeline is fetch, clean, truncate, extract, validate, retry. Fetch the page (a scraping API handles this layer), strip the boilerplate with BeautifulSoup, truncate the text to your token budget, call the model with the page text and a JSON Schema, validate the output against the schema, and retry with the validation errors fed back if it fails. The complete Python example in this post is the whole thing in about 150 lines.
How accurate is LLM extraction?
With validation and a retry loop, production systems routinely land in the 95-98% range on well-defined fields. A single attempt passes validation about 85% of the time; a second attempt with errors fed back pushes it to about 95.5%; a third to about 98%. Fields with one unambiguous right answer — prices, dates, IDs, stock status — extract at the high end. Judgment fields extract lower, and the failures that remain are caught by sampling and spot-checking, not stored.
What is the best LLM for HTML extraction?
For most workloads, the cheapest capable model is the right one, because extraction is reading and transcribing, not reasoning. GLM-5.3 and its peers handle it comfortably, and the small fast variants of the same families handle it for a fraction of the price. The model matters more as the page gets harder to read. Pin your model version and re-test on a sample when the provider changes anything.
How many tokens does it take to extract data from a page?
About 4,000 input tokens and 500 output tokens for a typical cleaned page. The input is the page text plus the schema and system prompt; the output is the JSON answer. A raw, uncleaned page can be 9,000 tokens or more, which is why cleaning and truncation are the highest-leverage cost controls in the pipeline.
Is LLM extraction legal and ethical?
LLM extraction is a parsing method, and it does not change the legality or ethics of a scrape. The same rules apply: respect robots.txt, rate-limit politely, prefer an official API or dataset when one exists, and do not scrape personal data you do not need. A smarter parser does not make a scrape more allowed.
Keep reading
Parsing HTML with BeautifulSoup: selectors, speed, and survival
A deep look at parsing HTML with BeautifulSoup and lxml in Python: CSS vs XPath, handling malformed markup, the speed difference between parsers, and writing selectors that survive small redesigns.
What Actually Gets You Blocked When Web Scraping (2026)
We run a web scraping API and see millions of requests a day. Here's what actually gets you blocked in 2026 — and what doesn't, signal by signal.
Web Scraping for AI Training Data in 2026
How AI companies build training datasets: Common Crawl, web corpora, domain scraping, the crawl-to-JSONL pipeline, what makes good data, the 2026 legal landscape, and how a small team builds its own.
Found this useful? Cite it as: webscraping.space. “LLM Extraction: Structured Data from HTML (2026 Guide).” https://webscraping.space/blog/llm-extraction-guide. Published 2026-08-14.