The Scraping Log
7 posts. Newest first.
AllAnti-BotEthics & LawNode.jsParsingPythonScaling
Web scraping ethics and robots.txt: the lines you don't cross
A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.
ethicsrobots-txtlegaltosBuilding search infrastructure for 1M+ queries a month: the stuff nobody tells you
A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.
searchinfrastructureinverted-indexgoScraping at scale: queues, caching, and not getting banned
How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.
scalingarchitecturecachingconcurrencyBypassing anti-bot protections: TLS, fingerprints, and Cloudflare
A frank guide to anti-bot defenses and how scrapers get past them: TLS/JA3 fingerprinting, Cloudflare's challenge, PerimeterX, CAPTCHAs, residential proxies, and the libraries (curl_cffi, Camoufox) that actually work in 2026.
anti-botcloudflaretls-fingerprintproxiesHeadless browser scraping with Playwright (Python & Node.js)
When the data lives behind JavaScript, requests is not enough. This guide covers headless browser scraping with Playwright in Python and Node.js: stealth, waiting strategies, intercepting API calls, and not running the browser when you don't have to.
playwrightheadless-browserjavascriptnodejsParsing HTML with BeautifulSoup: selectors, speed, and survival
A deep look at parsing HTML with BeautifulSoup and lxml in Python: CSS vs XPath, handling malformed markup, the speed difference between parsers, and writing selectors that survive small redesigns.
beautifulsouplxmlparsingcss-selectorsWeb scraping with Python requests: a practical starter
A hands-on guide to web scraping with Python's requests library: downloading pages, setting headers, handling redirects, parsing with BeautifulSoup, and avoiding the most common beginner mistakes.
pythonrequestsbeautifulsouphttp