The Scraping Log

2 posts in Scaling. Newest first.

AllAnti-BotEthics & LawNode.jsParsingPythonScaling

• Jul 13, 2026 · Scaling · 23 min read · HOT

  Building search infrastructure for 1M+ queries a month: the stuff nobody tells you

  A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.

  searchinfrastructureinverted-indexgo
• Jul 12, 2026 · Scaling · 2 min read

  Scraping at scale: queues, caching, and not getting banned

  How to take a working scraper to millions of pages without melting the target or getting banned: work queues, bounded concurrency, on-disk caching, dedup, retries with backoff, and polite scheduling.

  scalingarchitecturecachingconcurrency