webscraping.space
A field guide to scraping the modern web.
A no-nonsense, hands-on blog about extracting data from websites. From your first requests.get() to scrapers that survive at scale. We cover Python and Node.js, HTML parsing, headless browsers, anti-bot defenses, and how to stay on the right side of robots.txt. New posts whenever the spider finishes crawling.
What is web scraping?
Web scraping is the automated extraction of data from web pages. A scraper is a program that downloads HTML (or intercepts a site API responses), parses the structure, and pulls out the fields you care about. Prices, articles, contact info, product catalogs, research datasets.
In practice a modern scraper combines three pieces: a fetcher (an HTTP client or a headless browser), a parser (CSS or XPath selectors over the DOM), and a pipeline (dedup, storage, scheduling). The hard part is rarely the parsing. It is fetching reliably when the site rate-limits you, fingerprints your TLS handshake, or hides content behind JavaScript and bot detection.
If a human can see it in a browser, a scraper can capture it. The only real questions are how much load you place on the target and whether the site terms let you.
Latest from the log
FEATURED What Actually Gets You Blocked When Web Scraping (2026)
What Actually Gets You Blocked When Web Scraping (2026)
We run a web scraping API and see millions of requests a day. Here's what actually gets you blocked in 2026 — and what doesn't, signal by signal.
Web Scraping for AI Training Data in 2026
How AI companies build training datasets: Common Crawl, web corpora, domain scraping, the crawl-to-JSONL pipeline, what makes good data, the 2026 legal landscape, and how a small team builds its own.
E-commerce Scraping: The Complete 2026 Guide
The complete 2026 guide to e-commerce scraping: what data people scrape, platform anti-bot profiles, price-monitoring pipelines, cost math, working code, and the legal reality.
LLM Extraction: Structured Data from HTML (2026 Guide)
The 2026 guide to LLM extraction: when it beats CSS selectors, schema design, token cost math, validation, retries, and a complete Python pipeline.
GLM-5.3 Review: Coding, Security, Benchmarks (2026)
An engineer's review of GLM-5.3 (Zhipu AI, 2026): post-training wins, Terminal-Bench 28.3, ExploitBench 54.4, the open-weights safety debate, cost, self-hosting.
Where to start
- New to scraping? Start with Web scraping with Python requests.
- Hitting a Cloudflare wall? Read Bypassing anti-bot protections.
- Need to render JS? See Headless browser scraping with Playwright.
- Going to production? Scraping at scale without getting banned.