The Scraping Log
5 posts in Scaling. Newest first.
AllAI AgentsAnti-BotEthics & LawFundamentalsJavaScriptNode.jsParsingProxiesPythonScaling
E-commerce Scraping: The Complete 2026 Guide
The complete 2026 guide to e-commerce scraping: what data people scrape, platform anti-bot profiles, price-monitoring pipelines, cost math, working code, and the legal reality.
The Complete Proxy Guide for Web Scraping (2026)
The honest 2026 proxy guide: what a proxy does, the four types compared, rotation, detection, real pricing, cost math, when you need one, and working Python code.
Web Scraping APIs in 2026: The Complete Guide
The honest 2026 guide to web scraping APIs: when to buy vs build, the five stages inside a call, real pricing for every major provider, cost math, code, and verdicts.
Building search infrastructure for 1M+ queries a month: the stuff nobody tells you
A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.
Scraping at scale: queues, caching, and not getting banned
How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.