The Scraping Log

3 posts in Ethics & Law. Newest first.

AllAI AgentsAnti-BotEthics & LawFundamentalsJavaScriptNode.jsParsingProxiesPythonScaling

• Aug 14, 2026 · Ethics & Law · 37 min read · HOT

  Web Scraping for AI Training Data in 2026

  How AI companies build training datasets: Common Crawl, web corpora, domain scraping, the crawl-to-JSONL pipeline, what makes good data, the 2026 legal landscape, and how a small team builds its own.

  ai-trainingtraining-datacommon-crawlweb-corpora
• Aug 8, 2026 · Ethics & Law · 46 min read

  Is Web Scraping Legal? The 2026 Legal Landscape, Explained

  A practical, verdict-first explainer of web scraping law in 2026: the CFAA, hiQ v. LinkedIn, Terms of Service, GDPR and personal data, robots.txt, login walls, jurisdiction by jurisdiction, and the difference between public and allowed. Education, not legal advice.

  legalcfaaterms-of-servicegdpr
• Jul 13, 2026 · Ethics & Law · 36 min read

  Web scraping ethics and robots.txt: the lines you don't cross

  A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.

  ethicsrobots-txtlegaltos