URL: https://webscraping.spaceBest viewed at 1024×768

Ethics & Law Published Jul 13, 2026 · 36 min read · 8,029 words

Web scraping ethics and robots.txt: the lines you don't cross

A clear, practical guide to scraping ethically and legally: reading robots.txt, Terms of Service, the CFAA and EU equivalents, login walls, personal data, and the difference between 'public' and 'allowed'.

This is the least fun guide on the site and the one you should read first. The technical guides tell you how to scrape. This one tells you when you shouldn't. Getting the ethics wrong can get you banned, sued, or worse. None of this is legal advice. It's the working rules we apply.

Here's the uncomfortable truth about this space: almost everything about the ethics and legality of scraping is a judgment call that depends on what you scrape, how you scrape it, and who you are. There is no bright line that says "this is fine" and "this is not." What there is, is a set of conventions, a few court decisions, a pile of privacy regulations, and a lot of unspoken norms about what it means to be a good guest on someone else's server. This guide is the map we use to navigate all of it.

The good news is that the map isn't that complicated once you internalize one distinction: public and allowed are not the same thing. Most of the mistakes people make — the ones that end in cease-and-desist letters and lawsuits — come from confusing the two. A page can be publicly reachable and still not be something the site owner has consented to having scraped. And a site owner can consent to some scraping (their API, their public dataset) while simultaneously being furious about other scraping (their entire member directory). Your job as a responsible engineer is to understand the difference and act on it.

So let's start with the mechanism that encodes the most basic version of that consent: robots.txt.

What robots.txt actually is

robots.txt is a plain-text file that a site places at the root of its domain, at /robots.txt, to tell automated crawlers which paths they may and may not access. A browser fetching https://example.com/robots.txt gets a document that looks like this:

User-agent: *
Disallow: /private/
Disallow: /search?
Allow: /

User-agent: GPTBot
Disallow: /

It is not a firewall. It is not a law. It is a sign on the door, and it is only as effective as the willingness of crawlers to read it. But for anyone building a scraper in 2026, it is the single most important document on the site you intend to crawl, and knowing its history explains a lot about why it looks the way it does.

The 1994 standard

The protocol is almost as old as the web itself. In 1994, a web engineer named Martijn Koster posted a proposal to a mailing list called www-robots-ng. At the time, the web was small enough that one person could crawl a meaningful fraction of it, and search engines had started doing exactly that. Site owners were annoyed: crawlers were fetching everything, including admin panels, search endpoints that generated infinite URLs, and files they had no business reading. Koster's proposal — called the Standard for Robot Exclusion — was a de facto convention rather than a formal standard. It had three rules, essentially: a crawler should fetch /robots.txt, respect User-agent groups, and honor Disallow lines.

For twenty-eight years, that was it. There was no RFC, no governing body, no formal specification. The 1994 standard was a mailing-list post that everyone agreed to follow because it was in everyone's interest to be polite. Allow was added later as an unofficial extension. The Sitemap directive came from sitemaps.org in 2005, well outside the original protocol. Crawl-delay was an extension popularized by Yandex and other search engines. For all those years, the "spec" was whatever the big crawlers decided to do.

RFC 9309 in 2022

That changed in September 2022, when the IETF published RFC 9309, "Robots Exclusion Protocol," largely written by engineers from Google. The RFC did not invent anything new. It formally documented the behavior that had been the de facto standard for nearly three decades: user-agent groups, the Allow and Disallow directives, the * wildcard matching any sequence of characters, the $ wildcard matching end-of-URL, and longest-match precedence rules. It explicitly listed Sitemap and Crawl-delay as extensions rather than core parts of the protocol.

Why does the formalization matter? Two reasons. First, it gives everyone a stable reference point: when two libraries parse robots.txt differently, RFC 9309 is now the thing you argue about, rather than folklore. Second, and more importantly for the ethics conversation, the RFC's language about caching and honoring the file reinforces that robots.txt is a request from the site owner. The RFC says a crawler "should" fetch it, cache it, and respect it. "Should," not "must." That word choice is doing a lot of work, and we'll come back to it.

The anatomy of a robots.txt file

Let's read one properly. The file is a sequence of groups. Each group starts with a User-agent: line and contains the rules that apply to that agent. The special value * means "every crawler that doesn't match a more specific group." The rules themselves are prefix matches against the path portion of a URL. A few details are worth internalizing:

Disallow: /private/ blocks any URL whose path starts with /private/. A trailing slash matters: /private and /private/ are different prefixes.
Allow: / re-allows everything. In the example above, Allow: / after Disallow: /private/ does not undo the disallow — most parsers apply the most specific match, so /private/ still wins for that path. But Allow: / does tell crawlers the rest of the site is fair game.
Wildcards: * matches any sequence of characters and $ anchors to the end of the URL. So Disallow: /*.pdf$ blocks every path ending in .pdf. Google and a few other crawlers support wildcards; the 1994-era crawlers mostly ignored them. RFC 9309 formally blessed * and $.
Crawl-delay: 2 asks crawlers to wait at least two seconds between requests to this site. Google has always ignored Crawl-delay (it uses its own crawl-rate controls). Bing and Yandex honor it. Many smaller crawlers honor it too, which makes it a genuinely useful politeness signal even if the big players ignore it.
Sitemap: https://example.com/sitemap.xml is not a permission directive at all. It's a pointer saying "if you want the good URLs, here's where they all live." Treat it as a gift, not a gate.

How the big players actually interpret it

It's worth knowing that "honoring robots.txt" is not one behavior. Google, Cloudflare, and the rest each have their own interpretation, and a few differences matter in practice:

Google treats robots.txt as a strong hint but not an absolute block: content disallowed by robots.txt won't be indexed, but Google may still crawl the URL to determine things like 404 status. Google ignores Crawl-delay, caps robots.txt files at 500 KiB, and uses the longest-match rule within a group. Google's crawlers re-fetch robots.txt roughly every day, honoring HTTP cache headers (Last-Modified, Expires) to avoid hammering the file.
Cloudflare — which sits in front of a huge share of the web — offers managed robots.txt generation and, more interestingly, its own AI Audit tooling (more on that below). From Cloudflare's perspective, robots.txt is one input among several for deciding what an AI crawler gets to see.
Smaller crawlers and scrapers are the wildcard. Most honor the file because it's easy; some ignore it entirely because they were written by people who never read this guide.

The limits of robots.txt

Here is the part most tutorials skip: robots.txt does not protect anything. It is a request, not a control. A determined or careless crawler ignores it trivially. It's also a signal that only works when the crawler chooses to look — which means that when you, the responsible scraper, read it and honor it, you are making a voluntary ethical choice, not complying with a technical constraint. That's precisely why it matters. The file exists because the web's crawlers agreed to a social contract in 1994, and RFC 9309 formalized it in 2022. Every time you check robots.txt before a crawl, you are renewing that contract.

robots.txt went from a 1994 mailing-list convention to a formal RFC in 2022, and near-universal adoption on large sites. The file was never designed as a security control, but it became the web's default consent mechanism.

robots.txt vs the law

Here's where people get confused, so let's be precise. Violating robots.txt is not, by itself, a crime. There is no statute anywhere that says "disobeying robots.txt is illegal." It's a private convention. But that's not the same as saying it has no legal significance. It has three distinct legal roles, and they compound:

Evidence of authorization. Under the US Computer Fraud and Abuse Act (CFAA) and its analogues, the key question is often whether you accessed a system "without authorization" or "exceeded authorized access." A robots.txt file that says Disallow: / for your bot's user-agent is, at minimum, evidence that the site owner did not authorize you to access those paths. Courts have gone back and forth on how strong that evidence is — more on that in the ToS section — but the file is the owner's written statement of what they wanted you to do.
A Terms of Service hook. Many ToS explicitly incorporate robots.txt, or ban automated access outright. When the ToS says "no scraping," your reading of robots.txt becomes part of the breach analysis: you were told twice, and you did it anyway.
The "effective access control" question. This is the subtle legal fight of the past few years. The CFAA's criminal exposure is much clearer when you bypass an actual technical access control (a login, a CAPTCHA, an IP block) than when you merely ignore a request. Courts in the US have wrestled with whether robots.txt is a "gate" in that sense. The Ninth Circuit flip-flopped on exactly this question in the hiQ v. LinkedIn litigation, which we'll cover below. The short version: robots.txt is a weaker legal signal than a login wall, but it is not nothing.

For the deep dive on case law, there's a separate post on this site covering the legal side in detail. This post is about the ethics and the mechanics, so we're keeping the law at the level of "here's the risk, here's the shape of the argument, go read the legal post for the cases."

The ethics framework: public vs allowed

Let's build the mental model that everything else hangs on.

Public means: anyone with a browser and the URL can see it. No login, no paywall, no special access. Public is a factual statement about the state of the server.

Allowed means: the site owner consents to automated access. Allowed is a statement about intent, and intent is complicated.

Almost every scraping ethics failure is a failure to distinguish these. A public profile page is public. It is not necessarily allowed — the site owner might have an API, might have a robots.txt rule, might have a ToS that says "no automated access," or might just feel violated that you scraped their entire member directory into a marketing database. All of those are the owner expressing where the "allowed" line is.

The classic metaphor is being a guest. A store that's open is public: you can walk in, look around, read the prices. But being allowed inside does not mean you're allowed to photograph every shelf, stand in the aisles for three hours, take the entire inventory catalog home with you, or come back at 3 a.m. with a forklift. The store being open is a fact about the building; how you behave inside is a fact about you. The web is the same. A site owner who publishes content publicly is making a choice about reach. They are not necessarily making a choice about bulk automated consumption, which is a very different activity with very different costs.

This distinction also explains why "it's public data" is a defense people hear a lot and a defense that only goes so far. It's a real argument — the hiQ court bought a version of it, and the "public data" argument is genuinely strong for plain, non-personal, openly linked pages scraped gently. But it's the start of the analysis, not the end. The question isn't just "can I see this?" It's "has the owner said anything about automated access?" and "is my activity the kind of thing that would make a reasonable owner upset?"

Here's the decision flow we actually use:

Every box needs a "yes" for the crawl to proceed, and even then the crawl is only allowed if you go gently. One "no" anywhere in the chain is a stop-and-ask, not a workaround.

Notice what the flow doesn't include: a box that says "is it technically possible?" Because of course it is. You can almost always bypass a robots.txt rule, a login, or a CAPTCHA if you throw enough engineering at it. "Technically possible" is the wrong question. The right question is "has the owner said yes?"

The what-is-what matrix

Here's the same judgment compressed into a table we keep on the wall. It's the "what's okay and what isn't" version of the whole guide, and it's deliberately conservative. Every cell assumes you're scraping politely, identifying yourself, and willing to stop when asked:

Data type	Public?	robots.txt applies?	ToS risk?	Verdict
Plain HTML, no login, no rules	Yes	If set	Usually	Fine, politely
Reachable but disallowed by robots.txt	Yes	Yes	Usually	No
Public API, no auth	Yes	N/A	Yes, read it	Use the API, gently
Login-gated page	No	No	Yes	No, not without permission
Paywalled content	No	No	Yes	No
Personal data, even public	Maybe	Maybe	Maybe	No without lawful basis
Behind anti-bot / CAPTCHA	Technically	No	Yes	Avoid; it's an access control

The "what is/isn't ok" matrix. "Public" is a necessary condition, never a sufficient one. The moment you need to bypass something, the verdict flips to no.

Server politeness: don't be a denial of service

Let's get concrete about the thing that actually gets scrapers banned: not ethics, not law, but sheer rudeness at the HTTP layer. This section is where "ethical" and "practical" are the same sentence, because the politeness rules are also the anti-ban rules.

Rate limits and crawl-delay

The single most important variable in whether a site survives your crawl is requests per second, and the polite number depends on the site. A large CDN-fronted site with autoscaling can absorb a few hundred requests per second without a human noticing. A mom-and-pop shop running WooCommerce on shared hosting will fall over at fifty requests per minute. The ethical number is the one the site can absorb without degrading service for its actual human users.

This is the key insight: scraping fast isn't just risky for you, it's a cost you're imposing on the site owner and on other visitors. Every request you make competes with real users for CPU, memory, bandwidth, and database connections. When you hammer a small site, you're not just breaking a rule — you're making the site slower for the people who actually asked to be there. That's why 100 requests per second against a small site is unethical even if it's perfectly legal, even if robots.txt doesn't say anything, even if the ToS is silent. You are, in effect, a distributed denial-of-service attack with good intentions.

The practical rules we apply:

Set a crawl-delay of 1 to 3 seconds per domain as a default, and go higher for small sites. Scrapy's DOWNLOAD_DELAY and AUTOTHROTTLE settings do this for you; AUTOTHROTTLE_ENABLED = True with AUTOTHROTTLE_START_DELAY = 1 and AUTOTHROTTLE_MAX_DELAY = 10 is a good baseline.
Honor robots.txt Crawl-delay if the site sets one. It's the owner telling you their threshold in writing. Google ignores it, but Google is Google — you're not Google, and the site owner is talking to you.
Cap concurrency per domain, not just globally. Two to five concurrent connections per domain is the sane range.
Add jitter. A flat, clockwork interval of exactly 2.000 seconds is a fingerprint for a bot and is indistinguishable from a naive scheduler. Jitter your delays.
Crawl at off-peak hours for large jobs. The site's timezone, not yours.

Here's a polite crawler loop that respects robots.txt's Crawl-delay and adds jitter:

import time, random, urllib.robotparser
from urllib.request import urlopen

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

# robots.txt crawl-delay, or our own 2s default, plus jitter
delay = rp.crawl_delay("MyBot/1.0") or 2.0

def polite_fetch(url):
    if not rp.can_fetch("MyBot/1.0", url):
        return None
    time.sleep(delay + random.uniform(0, 1.0))
    req = urlopen(url)  # set your UA/headers on this request
    return req.read()

for url in seed_urls:
    body = polite_fetch(url)
    if body is not None:
        save(url, body)

The + random.uniform(0, 1.0) is the jitter. It makes your request pattern look like a person with a variable pace instead of a metronome, and it measurably reduces your collision with rate-limit buckets.

Honoring Retry-After

The server has a way to tell you, in writing, exactly how rude you're being: the HTTP 429 status code, "Too Many Requests." And the HTTP spec gives the server a way to tell you how long to wait: the Retry-After header. Honor both. A 429 is not an error to retry instantly — it's a measured instruction to wait.

import time, requests

def fetch_with_retry(session, url, attempts=4):
    for i in range(attempts):
        resp = session.get(url)
        if resp.status_code == 200:
            return resp
        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", 2 ** i))
            # Clamp absurd Retry-After values; 60s is a sane ceiling
            wait = min(wait, 60)
            time.sleep(wait)
            continue
        if resp.status_code >= 500:
            time.sleep(2 ** i + random.uniform(0, 1.0))
            continue
        # 4xx other than 429 is permanent; do not retry
        break
    return None

Two details worth underlining. First, never retry a 403 or a 404 — that's not a rate-limit, that's a decision the server made, and hammering it just makes you look like a hostile scanner. Second, a Retry-After of "0" is legal, and some servers send it — treat it as "immediately but gently," not as permission to loop at full speed, because the next 429 is coming with a longer wait.

Why backoff matters morally, not just technically

Exponential backoff with jitter is the technical fix for a moral problem. When you slow down on failure, you're doing the thing a good guest does when they realize they're crowding the doorway: you step back and let the room breathe. The server's 429s are a conversation. If you treat them as noise and keep talking over them, you've crossed from "scraping" into "harassment of a computer system," and that's where both the bans and the legal theories start.

Politeness is not just ethics, it's the cheapest anti-ban strategy in existence. The polite crawler's ban probability stays in single digits even at 40 requests per second; the aggressive one is effectively banned by 20.

Terms of Service and how courts treat them

Now the legal-adjacent part, kept at the level this post needs. The dedicated legal guide on this site covers the case law in depth; here's the shape of it.

Every site you visit has Terms of Service, whether you read them or not, and the browser-wrap/click-wrap doctrine means that by using the site you've generally agreed to them. A huge fraction of ToS contain a clause that says something like "no automated access, no scraping, no bots." Some go further and say "no copying of any content." Read the ToS before you scrape. It takes thirty seconds and it tells you what the owner thinks "allowed" means.

A ToS breach is a contract issue, not a crime. But that understates it. A contract breach gets you:

Banned — the site revokes your access, and your account, IP, or fingerprint is now on a block list.
Sued — breach of contract is a real cause of action, and scraping cases often bundle it with trespass, tortious interference, and other claims.
Injunctions — in some cases courts have ordered scrapers to stop, not just awarded money damages. The order is against you, and violating a court order is contempt, which is a much more serious category than breach of contract.

hiQ v LinkedIn, in one paragraph

The most famous scraping case is hiQ Labs v. LinkedIn. hiQ scraped publicly visible LinkedIn profiles — data you could see without logging in — to sell employee analytics. LinkedIn sent a cease-and-desist and blocked hiQ. hiQ sued, and the Ninth Circuit repeatedly held that scraping public data likely does not violate the CFAA, and — in the famous bit — that LinkedIn's robots.txt rule against hiQ was not an "access control" because the data was publicly reachable. That "public data is fair game" reading was a big win for scrapers. But in 2022, on rehearing, the Ninth Circuit withdrew that reasoning and held that LinkedIn's robots.txt did effectively bar access, meaning hiQ could have violated the CFAA after all. The case settled before the Supreme Court could give a final answer.

What you should take from hiQ: the law here is genuinely unsettled, it has moved against scrapers over time, and the "it's public" argument is weaker today than it was in 2017. If you're building a business model on scraping public data, the legal risk is real, it's trending against you, and you should treat "public" as necessary but not sufficient.

When breach-of-ToS matters most

ToS violations matter most when there's something else attached to them. Courts are not going to spend a lot of effort on a pure contract claim over scraping; they are going to care when the scraping involved bypassing technical controls, causing real harm to the site, or collecting data the site has a strong commercial interest in protecting. The cases that went badly for scrapers share a pattern: the scraper knew the owner said no, kept going after being told, and caused measurable damage or commercial harm. That's the profile to avoid. The people who get sued are not the ones who scrape gently and stop when asked. They're the ones who treat a cease-and-desist as a performance review.

Let's state the rule plainly: content behind a login is not public, and scraping it is a different category of activity. Not a slightly different category — a different category, like the difference between reading a book in a library and breaking into the locked archive.

When you scrape behind a login, several things stack up against you:

You're using credentials. Yours, or someone else's. If the site's ToS says "one account, no automated use," your credentialed scraping is itself a breach — you agreed to it at signup.
The access-control argument is much stronger. Under the CFAA, "exceeds authorized access" is much easier for a plaintiff to argue when you authenticated and then did something the account wasn't meant to do. A login wall is an actual technical gate, not a request; the courts are far more comfortable treating that as a real access control.
You may be looking at other people's data. Behind a login, you're often viewing private messages, personal dashboards, account settings, or data about other users. Even if you have permission for your own account, bulk-scraping other users' data is a personal-data problem on top of everything else.

The safe default: never scrape behind a login without written permission. And get the permission from the person who can actually grant it — the site owner, not the person whose password you borrowed.

The same logic extends to CAPTCHAs, anti-bot systems, and IP blocks. A bot-defense system is, legally and practically, an access control. It's the owner saying "no automated access" in machine language instead of English. Bypassing it — solving CAPTCHAs at scale, rotating through residential proxies to dodge IP blocks — is circumvention, and it dramatically weakens every "it's public" argument you might have. That doesn't mean every Cloudflare-protected site is untouchable; it means the burden shifts. If a site is actively defending against bots, the polite and defensible move is to assume the answer is no and go find an API, an official dataset, or permission.

This is the sharpest line on the whole map, and the one most engineers get wrong, because it's counterintuitive: even public personal data is protected. "I only scraped public profiles" is not a defense under the GDPR. It might not even be the start of a defense.

What counts as personal data

Under the EU/UK General Data Protection Regulation (GDPR), personal data is any information relating to an identified or identifiable natural person. Names, email addresses, phone numbers, photos, usernames, IP addresses, location data, employment history, user IDs. If your dataset could identify a person, it's personal data. California's CCPA/CPRA uses a similar "personal information" definition that's if anything broader. The bar is low, and "it was on a public website" doesn't move it.

The lawful-basis test

The GDPR says you may only process personal data if you have a lawful basis, and the list is short: consent, contract, legal obligation, vital interests, public task, or legitimate interests. For scrapers, legitimate interests is the one that usually gets invoked, and it's the one that does the most work. The problem: legitimate interests requires a balancing test. You have to weigh your interest in the data against the data subjects' rights and freedoms. Scraping ten thousand people's email addresses to build a sales list is very hard to square with that balancing test. Scraping company names and job titles from a public professional directory to verify business contact details is a much easier case. The more sensitive the data and the less essential your use, the harder the test is to pass.

Emails, phone numbers, and the extra layer

Emails and phone numbers sit under two regulatory regimes at once. They're personal data under GDPR/CCPA, and they're also communication channels governed by anti-spam rules: CAN-SPAM in the US, the ePrivacy Directive and GDPR marketing rules in the EU. Sending unsolicited commercial email to addresses you scraped is not just bad ethics — it's illegal in most of the world, with fines that dwarf what you'd ever earn from the campaign. If you scrape contact details, the question isn't "can I collect this?" — under ePrivacy, the real question is "can I contact them?" and the answer, without consent, is usually no.

Deletion rights change the engineering

The GDPR gives data subjects the right to access, correct, and delete their data, and it gives them the right to know what you have. If you collect personal data, you inherit a set of obligations: you need a documented lawful basis, you need to be able to tell a person what you hold about them, and you need to be able to delete it on request. Here's the engineering consequence: if you can't honor a deletion request — because you never captured a contact point, never built a deletion pipeline, never wrote down where the data flows — then you shouldn't have collected the data at all. The obligation isn't abstract. It's a systems design problem, and if you're not willing to build the systems, the ethical answer is not to collect.

The practical rule we apply: if your dataset could identify a person, treat it as personal data. Assume it's regulated. Collect the minimum you need for a documented purpose, get a real lawful basis, and build the access-and-deletion machinery before you write the crawler, not after.

Scraping for AI training: the 2024-2026 landscape

The newest and most contested front in scraping ethics is model training. This is where robots.txt went from a polite convention to the center of a global fight, and it's worth understanding how we got here.

The crawlers arrived, then the blocks arrived

Starting in mid-2023, the major AI labs shipped dedicated crawlers: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl, Google-Extended and Google-Cloud-Visual from Google, Applebot-Extended from Apple, PerplexityBot from Perplexity. Each one published a user-agent, and most published statements that they honor robots.txt and respect site opt-outs. Publishers reacted fast. By 2024, a wave of news sites, forums, and content platforms were adding Disallow: / rules for the AI crawlers. Cloudflare launched its AI Audit tool in September 2024, giving site owners a dashboard showing exactly which AI bots were crawling them, how often, and offering a one-click block for all of them.

The numbers moved fast. Roughly a tenth of the largest sites blocked major AI crawlers in early 2024; by 2026, it's approaching a third, and the growth is still going. It's one of the fastest robots.txt adoption shifts since the file was invented.

The share of top sites blocking AI crawlers roughly quintupled between 2024 and 2026. Cloudflare's AI Audit, which launched in September 2024, made the blocks visible and one-click, and adoption has kept climbing since.

The publishers-vs-crawlers fights

The background to all this is a set of increasingly public fights. The New York Times sued OpenAI and Microsoft over training data in late 2023, alleging that its content — including content behind its paywall and served under anti-bot protections — was scraped and used without permission. Dozens of other publishers and platforms took similar action or, more quietly, signed licensing deals. A recurring allegation in the most serious cases is that the training data wasn't just "public web pages": it included paywalled articles fetched through browser tricks, and other content that the publishers had explicitly walled off. Whether or not any specific allegation holds up, the pattern tells you something important about the ethics landscape: the fights are not over whether a model may learn from public text — they're over the edges, and the edges are exactly where the scraping ethics rules we've been talking about live.

What "ethical AI scraping" is starting to look like

There's a rough consensus emerging about what responsible data collection for model training means, and it's remarkably aligned with the rules in this guide:

Honor robots.txt and per-crawler opt-outs. The AI crawlers publish their user-agents and say they honor the file. If a site blocks your crawler, treat the block as a stop. That's the entire social contract, applied to a new crawler.
Honor block lists. Cloudflare and others maintain lists of AI crawlers that sites can block with one toggle. If a site is behind such a list and blocks you, that's the owner speaking.
Respect the behind-auth and anti-bot lines. The most legally dangerous AI-training scrapes are the ones that went through paywalls, logins, and anti-bot systems. Don't.
Prefer licensing and opt-in deals. The mature players are moving toward negotiated data deals with publishers, not mass scraping. If you can license the data, license it. It's the difference between asking and taking.
Keep provenance. The ethical and legal question "was this data obtained with consent?" gets asked retroactively, and the way you answer it is with records. Log where data came from, what the robots.txt said at the time, what you fetched, and what you did with it.

The throughline is the same as everywhere else in this guide: the crawler that identifies itself, asks first, and stops when told is the crawler that gets to keep crawling.

The working rules we apply

Every guide ends with "be ethical," and everyone nods, and then everyone writes the same for url in urls loop they were going to write anyway. So here's the operating policy we actually apply, as a numbered list, the same one we'd hand to a new engineer on the team. It's not aspirational; it's the checklist that runs before code.

Identify yourself honestly. Set a custom User-Agent that names your bot and includes a way to contact you — a URL or an email. MyBot/1.0 (+https://example.com/contact) tells the site owner who you are and that a human is reachable. An unlabeled, browser-impersonating UA is how you become indistinguishable from an attack.
Read robots.txt, and obey it. Use urllib.robotparser, reppy, or Scrapy's built-in ROBOTSTXT_OBEY = True. Re-check periodically; robots.txt changes. Scrapy does this for you if you leave the default on — there's rarely a good reason to turn it off.
Set a crawl-delay and cap concurrency. One to three seconds per domain by default, more for small sites. Two to five concurrent connections per domain, with jitter. Honor the site's own Crawl-delay if it sets one.
Don't scrape behind a login, paywall, or anti-bot system. If it's authenticated or actively defended, the answer is no until a person with authority says otherwise, in writing.
Minimize data. Collect only what you need for a documented purpose. If it's personal data, that's a lawful-basis conversation, not a code decision.
Cache, so you never re-request. Every request you save is a request you don't make. Cache raw responses to disk, dedup URLs, and never re-crawl what you already have.
Honor opt-outs. If a site blocks you, asks you to stop, or sends a 403, stop and don't try to route around it. A 403 is an answer, not a puzzle.
Prefer the official path. If the site has an API, a data dump, or a research program, use it. "We scraped because there was no API" is a real reason. "We scraped because the API is rate-limited" is not.
Stop when asked. The person running the site gets to change their mind. A cease-and-desist, a ban, a block — these are not obstacles to route around. They're the owner closing the door, and a good guest leaves.

Reading robots.txt programmatically is the first rule, so here's the version we actually ship, including Sitemap extraction and a fallback for the crawl_delay that some parsers don't populate:

import urllib.robotparser
from urllib.parse import urljoin
from urllib.request import urlopen

class RobotsPolicy:
    def __init__(self, base_url, user_agent):
        self.user_agent = user_agent
        self.rp = urllib.robotparser.RobotFileParser()
        self.rp.set_url(urljoin(base_url, "/robots.txt"))
        try:
            self.rp.read()
        except Exception:
            # Never crash a crawl because robots.txt was flaky;
            # fall back to the most conservative policy.
            self.rp = None

    def allowed(self, url):
        if self.rp is None:
            return False
        return self.rp.can_fetch(self.user_agent, url)

    def delay(self, default=2.0):
        if self.rp is None:
            return default
        return self.rp.crawl_delay(self.user_agent) or default

    def sitemaps(self, base_url):
        # Extract Sitemap: lines — they're pointers, not permissions,
        # but they tell you where the owner WANTS you to look.
        txt = urlopen(urljoin(base_url, "/robots.txt"), timeout=5).read().decode()
        return [line.split(":", 1)[1].strip()
                for line in txt.splitlines()
                if line.lower().startswith("sitemap:")]

That's the whole first hour of any polite scrape project. Nothing in it is hard. The hard part was deciding to do it.

What happens when you get it wrong

Let's be concrete about the downside, because a lot of scraping advice treats "don't be unethical" as if it were purely a moral argument with no teeth. It has teeth. Every ban and every lawsuit on this list is something a real scraper hit.

The soft consequences. You get rate-limited, then 429ed, then 403ed, then IP-banned. Your crawler's data starts showing 403 pages and Cloudflare challenge pages, and you spend a day debugging "why is my parser returning the wrong HTML" before you realize you've been blocked. This is the most common outcome of rude scraping, and it's embarrassing more than anything. It also compounds: once your IP, ASN, or browser fingerprint is on a block list, every other site behind the same bot-defense vendor is slightly harder to crawl. Bad behavior is contagious across domains.

The medium consequences. The site identifies you — your UA has no contact info, but your crawl pattern does — and they block your entire hosting range. Or they send your ISP an abuse complaint, and your VPS gets suspended. This is where "it's public" stops mattering, because the site isn't making a legal claim, they're just declining service, which they're entirely entitled to do. A site can refuse to serve you for any reason, including "we didn't like how you crawled us."

The serious consequences. A cease-and-desist letter, and what you do next is the single most important decision in the whole affair. Some scrapers have treated those letters as a challenge and escalated: new IPs, proxy rotation, solving CAPTCHAs, scraping harder. That's the path to injunctions and to being the defendant in a reported opinion. The scraper who stops when asked, apologizes, and negotiates an API contract gets to keep their business. The scraper who keeps going writes a legal blog post about the scraper who kept going.

The reputation consequences. Scraping is a small world, and site owners talk to each other through shared block lists, bot-defense vendors, and industry groups. A scraper known for ignoring robots.txt and hammering sites finds the doors quietly closing everywhere. A scraper known for polite, identified, well-behaved crawling gets API access and data deals offered unsolicited. Reputation is a real asset, and it's built entirely out of behavior at the HTTP layer.

The deeper truth is that "getting it wrong" rarely means one catastrophic event. It means a slow grind of blocked IPs, corrupted datasets, legal letters, and a reputation that precedes you. Almost none of it is worth it. Almost every scrape that "had" to be done unethically could have been done politely with an API, a permission, or a smaller scope. The engineering is rarely the bottleneck. The asking is.

A decision checklist

Before you scrape, answer these honestly:

Is the data public and non-personal?
Does robots.txt allow your path?
Does the ToS permit automated access?
Are you behind a login or bypassing an access control?
Will your rate be gentle enough to go unnoticed?
Is there an official API or dataset you could use instead?

If any answer is "no" or "I'm not sure," slow down and figure it out before writing code. And then go one step further and answer the question the checklist can't: if the site owner were watching your crawler's logs right now, would they be okay with what they see? If the honest answer is no, you already know what to do.

Key takeaways

robots.txt is the baseline. Obey it unless you have a specific, defensible reason not to.
"Public" and "allowed" are different things. Public is a fact about the server; allowed is a fact about consent.
Login walls and paywalls change the rules. Get permission.
Personal data is regulated even when public. Don't collect it without a lawful basis.
Anti-bot systems can be access controls. Bypassing them is riskier than scraping plain HTML.
Politeness is both ethical and the cheapest anti-ban strategy there is.
Rate limits, crawl-delay, and honoring Retry-After are the mechanical expression of being a good guest.
AI crawlers have made robots.txt a battleground. Honor the opt-outs, honor the block lists, and prefer licensing over scraping.
When a site asks you to stop, stop. The people who escalate are the ones who get banned, blocked, and sued.
If there's an API, a data dump, or a permission to be had, take it. Asking is almost always cheaper than taking.

Frequently Asked Questions

Is web scraping legal?

Scraping public, non-personal data is generally legal in many jurisdictions. US case law (hiQ v. LinkedIn) has favored scraping public data. But it depends on what you scrape, how, and where you are. Scraping behind a login, violating Terms of Service, circumventing access controls, or collecting personal data can be illegal. This guide is education, not legal advice.

Do I have to obey robots.txt?

robots.txt isn't a law, but disobeying it has consequences. It can be used as evidence that you exceeded authorized access. It often violates a site's Terms of Service. And ignoring it is the fastest way to get IP-banned. Practically and ethically, obey it unless you have a specific, well-considered reason not to.

Is it okay to scrape behind a login?

Generally no, not without explicit permission. Content behind authentication is not public. Scraping it often violates Terms of Service and can constitute exceeding authorized access under laws like the US CFAA. Always get permission before scraping authenticated areas.

Can I scrape personal data like emails and phone numbers?

Scraping personal data is heavily regulated under GDPR (EU/UK), CCPA (California), and similar laws. Even public personal data is personal data under GDPR. Collecting it without a lawful basis is a violation. Don't scrape personal data unless you have a clear legal basis and a specific purpose.

Does robots.txt apply to AI training crawlers?

Most major AI crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot — say they honor robots.txt. Many publishers added blocks for them in 2024 and 2025, and services like Cloudflare can enforce block lists. But robots.txt is only a request: a crawler that ignores it is violating the site owner's stated wishes, which is both unethical and evidence against you in a dispute.

What is a crawl delay and should I set one?

A crawl-delay is the minimum number of seconds a crawler should wait between requests to a site. It's a robots.txt extension that Google ignores but Bing, Yandex, and many smaller sites honor. As a scraper you should always set your own politeness delay (1 to 3 seconds is a reasonable default) regardless of what robots.txt says.

What should I do when I get a 429 Too Many Requests?

Slow down. Read the Retry-After header and wait at least that long. Use exponential backoff with jitter for retries, and never retry 4xx errors like 404 or 403. A 429 is the server telling you you're being rude; the correct response is to be less aggressive.

Keep reading

Found this useful? Cite it as: webscraping.space. “Web scraping ethics and robots.txt: the lines you don't cross.” https://webscraping.space/blog/web-scraping-ethics-and-robots-txt. Published 2026-07-13.