Ethics & Law Published Jul 13, 2026 · 4 min read · 780 words

Web scraping ethics and robots.txt: the lines you don't cross

A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.

This is the least fun guide on the site and the one you should read first. The technical guides tell you how to scrape. This one tells you when you shouldn't. Getting the ethics wrong can get you banned, sued, or worse. None of this is legal advice. It's the working rules we apply.

The short version

1. If it's public, non-personal data and you scrape politely, you're usually fine.
2. If it's behind a login, a paywall, or an access control, stop and get permission.
3. If it's personal data, assume it's regulated. Don't collect it without a lawful basis.
4. Obey robots.txt unless you have a specific, defensible reason not to.
5. Don't break the site. Don't lie about who you are when honesty is required.

Read robots.txt

robots.txt is a site's statement about what automated crawlers may access. It's not a law, but it's the baseline of polite crawling, and ignoring it has real consequences.

User-agent: *
Disallow: /private/
Disallow: /search?
Allow: /

User-agent: GPTBot
Disallow: /

Reading it programmatically:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

rp.can_fetch("MyBot/1.0", "https://example.com/public")   # True
rp.can_fetch("MyBot/1.0", "https://example.com/private")  # False
rp.crawl_delay("MyBot/1.0")                                # seconds, or None

Scrapy honors this automatically with ROBOTSTXT_OBEY = True. There's rarely a good reason to turn it off.

Terms of Service

Many sites' ToS forbid scraping outright. A ToS violation is a contract issue, not a crime, but it can still get you sued and will get you banned. Read the ToS. If it says "no automated access," scraping isn't "public data, fair game." It's a breach you agreed to by using the site.

A useful test: would the site's owner be upset if they saw exactly what you're doing? If yes, that's a signal. Not a stop sign, but a signal worth weighing.

Login walls and access controls

Once you authenticate, the rules change. Content behind a login is not public. Scraping it can run into:

• The US Computer Fraud and Abuse Act (CFAA) talks about "exceeding authorized access." After Van Buren (2021) the bar is higher, but authenticated scraping against ToS is still risky.
• EU equivalents include the Computer Misuse Act (UK) and similar laws elsewhere.
• Contract breach. The ToS you accepted to get the account.

The safe default: never scrape behind a login without written permission.

Personal data

This is the sharpest line. Under GDPR (EU/UK) and CCPA/CPRA (California), personal data is personal data whether or not it's "public." Names, emails, phone numbers, photos, user profiles, location data. All of it.

• Collecting it requires a lawful basis (consent, legitimate interest, etc.).
• You must honor data subject rights (access, deletion).
• "It was public on the site" is not a defense under GDPR.

Practical rule: if your dataset could identify a person, treat it as personal data. Don't collect it unless you have a real, documented basis to.

What "public" actually means

The CAPTCHA and anti-bot line is the subtle one. A bot-defense system is, legally, often an access control. Bypassing it can be treated as circumvention. That doesn't mean every Cloudflare site is off-limits. It means the "it's public" argument is weaker there.

Be a good guest

Politeness is ethical and practical:

• Set a User-Agent with a contact URL when scraping permitted targets.
• Crawl at off-peak hours for big jobs.
• Rate-limit yourself harder than the site would.
• Cache so you never re-request.
• If the site offers an API or a data dump, use that instead. Always.

A decision checklist

Before you scrape, answer these honestly:

1. Is the data public and non-personal?
2. Does robots.txt allow your path?
3. Does the ToS permit automated access?
4. Are you behind a login or bypassing an access control?
5. Will your rate be gentle enough to go unnoticed?
6. Is there an official API or dataset you could use instead?

If any answer is "no" or "I'm not sure," slow down and figure it out before writing code.

Key takeaways

• robots.txt is the baseline. Obey it unless you have a specific, defensible reason not to.
• Login walls and paywalls change the rules. Get permission.
• Personal data is regulated even when public. Don't collect it without a lawful basis.
• Anti-bot systems can be access controls. Bypassing them is riskier than scraping plain HTML.
• Politeness is both ethical and the cheapest anti-ban strategy there is.

Found this useful? Cite it as: webscraping.space. “Web scraping ethics and robots.txt: the lines you don't cross.” https://webscraping.space/blog/web-scraping-ethics-and-robots-txt. Published 2026-07-13.