Scaling Published Aug 14, 2026 · 39 min read · 8,662 words
E-commerce Scraping: The Complete 2026 Guide
The complete 2026 guide to e-commerce scraping: what data people scrape, platform anti-bot profiles, price-monitoring pipelines, cost math, working code, and the legal reality.
E-commerce is where web scraping earns its keep. Not the blogs, not the news sites — the product catalogs. Every day, retailers, brands, and marketplaces move trillions of dollars on prices that change by the hour, and the teams that win are the ones that can see those prices first. That is the commercial high-volume pillar of scraping: product listings, prices, availability, reviews, ratings, and images, pulled from Shopify stores, Amazon, Walmart, and a dozen marketplaces, on a schedule, into a database, forever.
I have built price monitors for brands and retailers, and the same machine shows up every time. Someone needs to know what a competitor charges, whether a retailer is honoring MAP pricing, or whether a product is in stock — and the answer is always the same five stages: discover the catalog, fetch the product pages, parse them into structured data, store the history, and detect changes. This guide is that machine, end to end. What data people actually scrape, the anti-bot profile of every major platform, the pipeline that turns a store into a dataset, the honest cost math, a working Python example, and the legal reality you need to respect while you do it.
The site's ground rules apply throughout: prefer an official API or dataset when one exists, respect robots.txt, rate-limit politely, and don't collect personal data you don't need. Everything here assumes public, ToS-aware, educational use.
Who is this guide for? The engineer who has been handed a price-monitoring project and needs the whole map before writing the first selector. The founder who needs to know whether the data they want is even reachable, and what it will cost. The analyst who has been scraping one store with a script and needs to know what happens when the catalog grows to a hundred thousand products. If you already know you need a scraping API, the web scraping APIs post is the faster route; this guide is the category — the data, the platforms, the pipeline, the cost, the code, and the law, in one place.
Key takeaways
- Prices are the #1 data type, by a wide margin. Roughly 4 in 10 e-commerce scraping projects are built around price data — monitoring, MAP compliance, repricing. Everything else (listings, availability, reviews) supports or feeds that core.
- Platforms split into three anti-bot tiers. Shopify stores are the easiest (often a public /products.json endpoint); Amazon and Walmart are the hardest (aggressive rate limits, CAPTCHAs, IP blocking); eBay, Etsy, and AliExpress sit in between.
- The pipeline is five stages, and the last one is the product. Catalog discovery → product pages → structured data → storage → change detection. The first four are plumbing; change detection is where the value lives.
- Check for the store's own JSON API before you render anything. Shopify exposes /products.json; many stores ship their catalog in a sitemap or an embedded JSON blob. Rendering a browser costs 10x and is the most detectable footprint on the web.
- Cost per 1k product pages ranges from pennies to dollars. Self-hosted static scraping is about $0.03/1k in compute; a rendered page via an API runs about $4.50/1k. The gap is the price of not owning the fetch-render-proxy layer.
- Change detection is a hash comparison, not a diff. Store a hash of the fields you care about, re-crawl on a schedule, and alert only when the hash changes. That's the entire discipline, and it keeps both your bill and the target's load low.
- Legal reality: public data is generally fair game; ToS violations are a contract risk. Scraping public data politely is broadly legal in the US (hiQ v. LinkedIn), but Amazon's and Walmart's ToS explicitly prohibit scraping, and the risk is real. Prefer official APIs, honor robots.txt, and don't resell content wholesale.
What people actually scrape
Every e-commerce scraping project starts with the same question: which fields? The answer is remarkably consistent across industries, and it decides your parser, your storage schema, and your re-crawl schedule. Here is the honest breakdown of what people collect, roughly in order of how often it shows up in production.
Prices lead by a wide margin, and that's not an accident. A price is the highest-value, highest-churn field on a product page: it changes more often than the title, the description, or the image, and a change is a direct business signal. For a brand, a price drop below the agreed minimum is a MAP violation. For a retailer, a competitor's price is the input to a repricing decision. For a marketplace seller, the price is the difference between winning and losing the Buy Box. Everything else on the page exists to support or explain that one number.
Listings and catalog data are the skeleton. Titles, SKUs, categories, and variant options are what make a product identifiable and joinable across sources — the key you use to match "the same product" on Amazon and Walmart. Availability is the operational signal: in stock, out of stock, backorder, and the estimated ship date. It matters most for inventory arbitrage, drop-shipping, and any workflow where "can I buy this right now" is the question. Reviews and ratings are the quality signal — text for sentiment, the aggregate score for a quick comparison. Images are the visual record, and they matter more than most people expect: an image change is a surprisingly reliable early signal of a listing being reworked or a product being replaced. Variants (size, color, configuration) carry their own prices and availability, which is where naive scrapers break — the price on the page is often the lowest variant's price, not the product's. Shipping and seller data round it out: who actually sells the item, what it costs to ship, and how long it takes.
Reviews and ratings deserve their own paragraph, because they're the data type with the most legal surface area. The aggregate rating and the review count are facts — safe to collect and compare. The review text is expression, and the reviewer's username is personal data under GDPR and similar laws. The honest line: collect the aggregate score and count freely, collect review text only when you have a real use for it, and never collect reviewer identities you don't need. Most e-commerce scraping projects that touch reviews are actually after the aggregate — the star rating that feeds a comparison or a quality signal — and that's the part that's both useful and clean.
Before you write a single selector, decide the schema — the shape every product will take once it's in your database. The schema is the contract between the scraper and everything downstream, and it's where most e-commerce projects quietly rot. A minimal product record looks like this: a stable key (SKU or canonical URL), title, brand, category, price, currency, availability, image URL, variant options, and scraped_at. The key is the part that matters most: it's what lets you join the same product across Amazon and Walmart, and it's what makes change detection possible. If you don't have a stable key, you don't have a product — you have a page. Decide the key before you decide the parser, and the rest of the pipeline falls into place.
Availability is the data type that looks boring and quietly decides whether the whole dataset is usable. A price on an out-of-stock product is a historical fact, not a buying signal — and the teams that scrape prices without availability end up repricing against products nobody can buy. The honest schema always pairs price with availability, and the change-detection hash in the example below hashes them together for exactly that reason. Images, meanwhile, are the cheapest early-warning system in e-commerce: when a listing's image changes, the product is usually being reworked, replaced, or re-listed, and that's often the first sign of a change that will show up in the price later.
The platforms and their anti-bot profiles
The platform a store runs on determines 80% of your scraping difficulty. The platform decides how the page is rendered, how aggressively it defends, whether there's a public JSON endpoint hiding in plain sight, and whether an official API exists. Here's the 2026 map.
Shopify is the easiest target in e-commerce, and it's the one most people start with. Shopify renders on the server (Liquid), so the HTML contains the data — no browser needed. Better, most Shopify stores expose a public /products.json endpoint that returns the whole catalog as JSON, and a sitemap.xml that lists every product URL. Rate limits are light, CAPTCHAs are rare, and robots.txt is usually permissive. The honest rule for Shopify: check /products.json first, then sitemap.xml, then category pages. You will rarely need anything heavier.
One practical note on the Shopify /products.json endpoint: it paginates, and the pagination is the part that trips people up. The endpoint returns 250 products per page, and the next page is linked in the response headers (a Link header with rel="next"), not in the body. A naive crawler that ignores the Link header scrapes the same 250 products forever and calls it a catalog. The fix is a loop that follows the Link header until it's absent — the same shape as following a "next" button on a category page, which is why the pagination discipline in the pipeline section applies to JSON endpoints too.
eBay and Etsy sit in the middle. Both render some content client-side, both have moderate rate limits, and both have official, well-documented APIs (eBay's Browse API, Etsy's Open API) that cover most product data. The honest rule: use the API. It's cheaper, more stable, and legal by construction. Scraping them directly works, but you're paying for the privilege of worse data.
Amazon is the hardest target in e-commerce, full stop. Aggressive rate limits, CAPTCHAs, IP blocks, TLS fingerprinting, and data that's present in the HTML but deliberately obfuscated. Amazon's Conditions of Use explicitly prohibit scraping, and the company enforces it. The honest rule: apply for the Product Advertising API (PA-API) first; if you can't get approved or need data the API doesn't cover, you're in the residential-proxy-plus-rendering tier, with strict rate discipline and a real chance of being blocked anyway.
Walmart is nearly as hard — JS-rendered, aggressive anti-bot, CAPTCHAs — and it also has an affiliate API that covers product data. AliExpress and the big-box retailers (Target, Best Buy, Home Depot) are the same tier: JS-rendered, aggressive rate limits, CAPTCHAs, and no useful public API. For all of these, the honest answer is the same: official API first, and if you must scrape, budget for residential proxies, rendering, and a serious retry-and-backoff layer.
How do you tell which platform a store runs before you scrape it? The page source tells you in seconds. Shopify stores ship /cdn/shop/ asset paths and a Liquid-rendered HTML structure; Amazon pages carry the well-known obfuscated product data and a distinctive header; Walmart and the big-box retailers are identifiable by their script bundles and CDN hosts. The web scraping without getting blocked post covers the fingerprinting side of this in detail. The operational point is simpler: identify the platform first, because the platform decides your entire approach — the endpoint to try, the render decision, the proxy budget, and the legal posture. Scraping blind is how people burn a week on a target that had a JSON endpoint all along.
The anti-bot arms race is real, and it's worth understanding before you hit your first 403. The platforms that defend hard aren't defending against you specifically — they're defending against the industrial-scale scraping that costs them money and degrades their service. A polite, low-volume crawler that respects robots.txt and rate limits is a different class of traffic than a 50-request-per-second bot farm, and the platforms' defenses are calibrated to the latter. That's not a guarantee — Amazon blocks polite crawlers too — but it's the reason the honest playbook is the same everywhere: identify the platform, prefer the API, crawl slowly, and treat a block as an identity problem rather than a request problem.
Price monitoring: the #1 use case
Why is price monitoring the #1 use case? Because it's the highest-value, highest-churn data on the internet, and it's the one scraping job with a direct, measurable ROI. A brand uses it to enforce MAP pricing — the minimum advertised price retailers agree to — and a single violation can be a six-figure problem. A retailer uses it for competitive intelligence: what does the competitor charge for the same SKU, and when? A marketplace seller uses it for repricing: the algorithm that keeps you in the Buy Box by matching or beating the lowest offer. All three are the same machine — a scheduled crawl that detects price changes — pointed at different questions.
The first decision in any price monitor is the re-crawl schedule, and it should be driven by the category, not by habit. Groceries reprice almost daily; electronics and fashion weekly; home goods monthly; books rarely. Crawl too often and you waste money and load on pages that haven't changed. Crawl too rarely and you miss the signal — the price that dropped for four hours and snapped back. The honest rule: start at a daily crawl, measure how often prices actually change for your category, and back off until you're catching roughly 95% of changes. For most categories that lands between daily and weekly.
Two price-monitoring patterns are worth naming because they change the schedule. The first is flash sales and lightning deals: prices that drop for hours and snap back. If your category does these, a daily crawl will miss most of them, and the honest answer is either a faster crawl on a small, high-value subset or an alerting integration with the platform's own deal feed. The second is MAP enforcement, where you're not tracking the market — you're tracking a specific set of authorized retailers against a contract price. That's a smaller catalog, a slower schedule, and a much higher tolerance for false positives, because a false MAP alert is a phone call to a retailer who did nothing wrong. Know which pattern you're in before you set the cadence.
The pipeline: from store to dataset
The machine is always the same five stages, and it's worth drawing once because every e-commerce scraping project is a variation on it.
Catalog discovery is the stage people skip, and it's the one that bites. You can't monitor prices you don't know exist. The best starting point is always the sitemap.xml — it lists every product URL the store wants indexed, which is exactly the catalog you want. Category pages are the fallback, and /products.json is the Shopify shortcut. The output of this stage is a list of product URLs, deduplicated and normalized.
Product pages are the fetch-and-parse stage. Fetch the page, check whether the data is in the raw HTML (it usually is), and parse the fields you decided on in the first section. Structured data is the normalization stage: every source becomes the same schema — SKU, title, price, availability, image, URL, scraped_at — so you can join Amazon and Walmart on the same key. Storage is where the history lives: SQLite for a few thousand products, Postgres for millions, with a products table and a price_changes table. Change detection is the last stage and the product: compare this run to the last run, and emit a signal only when something moved.
Catalog discovery deserves one more paragraph, because it's where the quality of the whole dataset is decided. A sitemap.xml gives you the complete, canonical catalog — every product URL the store wants indexed — and it's the right starting point for any store that publishes one. Category pages are the fallback, and they come with two traps: pagination (the "next" link that eventually 404s) and faceted navigation (the filter URLs that multiply the catalog into tens of thousands of near-duplicate pages). The discipline is to normalize URLs — strip tracking parameters, sort query strings, drop fragments — and deduplicate before you fetch, or you'll scrape the same product under a dozen URLs and your change detection will never fire. The scraping at scale post has the full URL-normalization and dedup machinery.
Handling JavaScript-rendered stores
Here's the counterintuitive truth about e-commerce: most of the data is not actually behind JavaScript. Shopify renders on the server, so the HTML contains the prices. Amazon embeds a JSON blob in the page. The stores that genuinely render client-side are the minority — but they're real, and they're usually the ones with the most aggressive anti-bot. The discipline is the same everywhere, and it's worth repeating: check the raw HTML first, look for the store's own JSON API, and render only when you must.
The Shopify /products.json trick deserves its own paragraph, because it collapses the hardest part of e-commerce scraping into one HTTP call. Most Shopify stores expose /products.json, which returns the entire catalog as JSON — title, price, availability, images, variants — with no browser, no parsing, and no anti-bot fight. It's the single best free endpoint in e-commerce scraping, and it's the first thing to check on any store that looks like Shopify. (You can usually tell from the page source or the /cdn/shop/ asset paths.)
When a store genuinely renders client-side, you have two honest options: a headless browser (Playwright) or a rendered scraping API. A browser costs roughly 10x the CPU and memory of an HTTP client and is the most detectable footprint on the web, so it's the last rung, not the default. The scraping JavaScript-rendered pages post is the full decision tree; the short version is: view source, find the JSON API, call it directly, and only then reach for a browser.
The JSON-API interception technique is the one that saves the most money in e-commerce, and it's worth spelling out. Open the page in a browser with DevTools on the Network tab, reload, and look for XHR or fetch requests that return JSON — the store's own API calls. Nine times out of ten, the product data you want is in one of those responses, and you can call that endpoint directly with a plain HTTP client, no browser, no rendering, no anti-bot fight. This is how the smartest e-commerce scrapers operate: they don't scrape the page at all, they scrape the API the page uses. It's faster, cheaper, and dramatically less detectable.
If you do reach for a browser, the Playwright discipline is the same as everywhere else: one browser context per site, a realistic viewport and User-Agent, and a hard cap on concurrency — a real Playwright run costs about 4 pages per second per browser tab, and the moment you parallelize browsers you multiply both the CPU cost and the detection surface. The headless browser scraping with Playwright post is the full guide; the e-commerce-specific advice is to use the browser to discover the JSON API, then switch to plain HTTP for the volume. The browser is a reconnaissance tool and a last resort, not a production fetcher.
Cost math: per product page
The honest cost of e-commerce scraping is per product page, and it splits into two numbers: the fetch cost and the proxy cost. A product page is roughly 100-300KB. At residential proxy prices of $4-5/GB, a gigabyte covers roughly 5,000-10,000 product pages — so bandwidth alone runs about $0.40-1.00 per 1k pages. The fetch cost depends on whether you self-host or buy.
Read the chart as two taxes. The rendered tax is the gap between the static and rendered numbers: a headless browser per page costs real CPU, and it's the single biggest line item in e-commerce scraping. The managed tax is the gap between self-hosted and API: you're paying for someone else to run the fetch-render-proxy layer and eat the maintenance. Both taxes are avoidable on the pages that don't need them — which is most of them, if you check the raw HTML first.
The math that matters: a 100,000-product catalog crawled daily is 3 million page fetches a month. At $0.25/1k that's $750; at $4.50/1k it's $13,500. The difference is entirely the rendered tax, and it's the difference between a project that's obviously worth doing and one that needs a spreadsheet. Keirolabs is the flat-price outlier at $0.25/1k with rendering and residential proxies bundled — one honest option among several, not the whole story. ScraperAPI runs about $1.10/1k static and $4.50/1k rendered; ScrapingBee is $0.20/1k basic and about $1/1k rendered; Firecrawl is about $3.20/1k. The web scraping APIs post has the full roundup and the cost calculator.
Where does the crossover sit? The honest answer is the same band as the general scraping-API math: below roughly 10,000 product pages a month, self-hosting wins on cash — requests plus BeautifulSoup is free, and the catalog is small enough that a single script handles it. Above 50,000 pages a month, a self-built stack starts to beat the per-page price — unless your targets are Amazon or Walmart, in which case the proxy, rendering, and anti-bot work you'd have to build shifts the crossover back toward buying. The e-commerce-specific version of that math is that the platform tier matters more than the volume. A million Shopify pages self-host cheaply; a hundred thousand Amazon pages do not.
One more cost lever, and it's the biggest one that isn't a price: caching. A price monitor re-crawls the same catalog on a schedule, and most of it doesn't change between runs. Cache every response — HTML or JSON — keyed by normalized URL, and re-request only what changed. On a daily crawl of a stable catalog, that cuts effective volume by 60-90%, which is a bigger saving than any provider discount. It also doubles as a retry store: a failed request can be replayed from cache until the target recovers. The cheapest page is the one you never fetch.
Change detection: where the value lives
The last stage is where the value lives. A price monitor that re-scrapes everything and stores everything is a data hoarder; a price monitor that detects changes and alerts is a product. The mechanism is embarrassingly simple: hash the fields you care about, compare to the last run, and act only on differences.
The hash is the trick. You don't diff the whole page — you hash the fields you care about (price, availability, maybe stock text) and compare hashes. If the hash is unchanged, the product is unchanged, and you skip it. If it changed, you update the row, write a price_changes record, and emit the signal — an alert, a webhook, a repricing decision. The "same" fast path is what keeps both your bill and the target's load low: on a daily crawl of a stable catalog, 90% or more of products are unchanged, and the hash lets you skip them in microseconds.
Two refinements matter in production. First, normalize before you hash: strip currency symbols, collapse whitespace, and convert "In stock" and "in stock" to the same value, or every run will look like a change. Second, store the hash with the row, not in memory — a restart shouldn't force a full re-crawl. The scraping at scale post has the full caching and dedup scaffolding; the change-detection core is the five lines in the example below.
Alerting is where change detection becomes a product, and the honest design is boring: a webhook or a message-queue event per change, with the product key, the old value, the new value, and a timestamp. The consumers decide what's interesting — a repricing engine, a MAP-compliance dashboard, a Slack channel. The trap is alert fatigue: if you alert on every price move, the signal drowns in noise, and the team stops reading. The fix is to alert on the deltas that matter — a price below a threshold, a change larger than a percentage, a product that went out of stock — and to let the raw change stream feed the analytics, not the notifications. The priority-queue feedback loop from the pipeline diagram is the same idea applied to crawling: changed items get re-checked sooner, so a fast-moving price is watched more closely than a stable one.
Which platform needs what: a decision tree
The platform section, compressed into one pass. The tree routes by what runs the store, because that single fact decides the render path, the anti-bot tier, and whether an API exists.
The tree is the whole platform section in one pass. Shopify: check the JSON endpoint, then the sitemap, then category pages — static fetch usually works, and the anti-bot is light. eBay and Etsy: official APIs exist and are well-documented; use them before scraping. Amazon and Walmart: hardest tier, official API first, and if you must scrape, residential plus rendering plus strict rate discipline. AliExpress and the big-box retailers: residential plus rendering, and expect CAPTCHAs. The one rule that overrides all of it: check for an official API before any scraping route. It beats everything on cost, stability, and legality.
A working example: crawl, extract, detect
Here's the whole machine in one runnable script. It crawls a real bookstore catalog (books.toscrape.com — a sandbox built for exactly this), extracts the fields we care about, stores them in SQLite, and detects price changes between runs. Run it once to seed the database, run it again to see the change detection fire. It's polite by construction: one request per second, a descriptive User-Agent, and no JavaScript rendering, because the data is in the HTML.
import hashlib
import sqlite3
import time
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
BASE = "https://books.toscrape.com/"
DB = "prices.db"
UA = "price-monitor/1.0 (educational, polite; contact: you@example.com)"
def discover_products(session, max_pages=5):
"""Stage 1: walk the catalog pages and collect product URLs."""
urls = []
for page in range(1, max_pages + 1):
url = f"{BASE}catalogue/page-{page}.html"
resp = session.get(url)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for a in soup.select("h3 a"):
urls.append(urljoin(BASE, a["href"]))
time.sleep(1) # polite delay between pages
return urls
def extract_product(session, url):
"""Stage 2: parse the fields we care about from a product page."""
resp = session.get(url)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
title = soup.select_one("h1").text.strip()
price = float(soup.select_one("p.price_color").text.strip("£"))
stock = soup.select_one("p.instock.availability")
available = "In stock" in stock.text if stock else False
return {"url": url, "title": title, "price": price, "available": available}
def init_db():
"""Stage 4: storage. One row per product, one row per detected change."""
conn = sqlite3.connect(DB)
conn.execute("""CREATE TABLE IF NOT EXISTS products (
url TEXT PRIMARY KEY,
title TEXT,
last_price REAL,
available INTEGER,
last_seen TEXT,
price_hash TEXT)""")
conn.execute("""CREATE TABLE IF NOT EXISTS price_changes (
url TEXT, title TEXT, old_price REAL, new_price REAL, changed_at TEXT)""")
return conn
def detect_change(conn, product):
"""Stage 5: hash the fields we care about, act only when they move."""
row = conn.execute(
"SELECT last_price, price_hash FROM products WHERE url = ?",
(product["url"],),
).fetchone()
new_hash = hashlib.sha256(
f"{product['price']}|{product['available']}".encode()
).hexdigest()
if row is None:
conn.execute(
"INSERT INTO products VALUES (?,?,?,?,?,?)",
(product["url"], product["title"], product["price"],
int(product["available"]), time.strftime("%Y-%m-%d %H:%M"), new_hash),
)
print(f"NEW {product['title']} @ £{product['price']}")
elif row[1] != new_hash:
conn.execute(
"INSERT INTO price_changes VALUES (?,?,?,?,?)",
(product["url"], product["title"], row[0], product["price"],
time.strftime("%Y-%m-%d %H:%M")),
)
conn.execute(
"UPDATE products SET last_price=?, available=?, last_seen=?, price_hash=? WHERE url=?",
(product["price"], int(product["available"]),
time.strftime("%Y-%m-%d %H:%M"), new_hash, product["url"]),
)
print(f"CHANGED {product['title']}: £{row[0]} -> £{product['price']}")
else:
print(f"same {product['title']} @ £{product['price']}")
def main():
session = requests.Session()
session.headers["User-Agent"] = UA
conn = init_db()
for url in discover_products(session):
product = extract_product(session, url)
detect_change(conn, product)
time.sleep(1) # one request per second, per domain
conn.commit()
conn.close()
if __name__ == "__main__":
main()
Three things to notice. First, the catalog discovery is a loop over category pages — in production you'd read sitemap.xml instead, but the shape is the same: collect URLs, dedupe, fetch. Second, the change detection is a hash comparison: the price and availability are hashed together, and the row is only touched when the hash moves. Third, the whole thing is polite: one request per second, a User-Agent that identifies the crawler, and no rendering. That's the entire discipline, and it's the same shape whether you're monitoring ten products or ten million — the scraping at scale post is what you add when the ten-million version stops fitting in one process.
Extending the example to production is a series of small, mechanical upgrades. Read sitemap.xml instead of hard-coded page ranges, and parse it with a streaming XML parser so a million-URL sitemap doesn't blow up memory. Add variants: a product with size and color options has a price per variant, and the hash should cover the variant list, not just the base price. Add multiple stores: the schema is the join key, and the same SKU across Amazon and Walmart becomes one product with two price histories. Add scheduling: a cron job or a queue worker that runs the crawl on the cadence your category chart says. And add a politeness log — timestamp, URL, status, bytes — because the moment you're scraping multiple stores, the per-domain rate limit is the thing that keeps you alive. Each upgrade is boring; together they're the difference between a script and a system.
The legal and ToS reality
Here's the part nobody puts in the marketing. Scraping public data is broadly legal in the US — the Ninth Circuit's hiQ v. LinkedIn ruling held that scraping publicly accessible data doesn't violate the Computer Fraud and Abuse Act. But "legal" and "allowed by the site" are different questions, and e-commerce is where that gap is widest. Amazon's Conditions of Use explicitly prohibit scraping. So do Walmart's, and most marketplaces'. A ToS violation is a contract claim, not a criminal one, and in practice the risk is account bans, IP blocks, and cease-and-desist letters — but it's a real risk, and it's yours, not your proxy provider's.
The honest operating rules, which this site has always published and which I follow in production: prefer an official API or dataset when one exists — Amazon PA-API, eBay Browse API, Walmart Affiliate API, Shopify Storefront API. Respect robots.txt and rate limits; a polite crawler is both more ethical and less likely to be blocked. Don't scrape personal data you don't need — review usernames and profile data are personal data under GDPR, and "I only need the star rating" is not a defense. Don't resell someone else's content wholesale — prices and facts aren't copyrightable, but product descriptions and images are. And when in doubt, ask a lawyer who does internet law. The web scraping legal post is the full picture; the operational summary is short: public data, polite rate limits, ToS-aware.
The hiQ v. LinkedIn case is worth understanding precisely, because it's the anchor of the US legal picture. hiQ scraped publicly accessible LinkedIn profiles; LinkedIn sent a cease-and-desist and blocked hiQ's IPs; hiQ sued. The Ninth Circuit held that scraping publicly accessible data doesn't violate the CFAA — the anti-hacking statute — because the data was public and no authorization was circumvented. What the case did not decide is the ToS question: LinkedIn's terms prohibited the scraping, and the court left that as a contract matter. That's the exact shape of e-commerce scraping: the data is public, the CFAA is generally not the risk, and the ToS is a real but civil risk. The web scraping legal post walks through the cases and the jurisdictions; the practical takeaway is that the risk profile is "contract and policy," not "criminal," and that a polite, low-volume, API-first approach is both the ethical and the defensible one.
Further reading
These go deeper on the pieces this guide covers quickly:
- Web Scraping APIs — the buy path: when an API beats self-hosting, and the full provider roundup with pricing
- The Complete Proxy Guide — residential, datacenter, and when each makes sense for e-commerce targets
- Scraping JavaScript-Rendered Pages — the render-vs-JSON-API decision, and when a browser is actually required
- Web Scraping Legal — the legal reality: public data, ToS, and the cases that define the line
- Web Scraping Without Getting Blocked — the operations playbook for staying alive on aggressive targets
Frequently Asked Questions
Is scraping Amazon legal?
Scraping publicly accessible data is broadly legal in the US — the Ninth Circuit's hiQ v. LinkedIn ruling held that scraping public data doesn't violate the CFAA. But Amazon's Conditions of Use explicitly prohibit scraping, and a ToS violation is a contract risk: account bans, IP blocks, and cease-and-desist letters. The honest path is to apply for Amazon's Product Advertising API (PA-API) first, and if you must scrape, do it politely, at low volume, and understand the risk is yours.
How do I scrape product prices?
Fetch the product page, parse the price field, and store it with a timestamp. Check the raw HTML first — most stores render prices on the server, so no browser is needed. Shopify stores often expose a /products.json endpoint that returns prices as JSON. Store each price in a history table, and on the next run compare the new price to the last one to detect changes. The working example in this guide is the full pattern.
How do I scrape Shopify stores?
Start with the store's /products.json endpoint, which returns the entire catalog as JSON — title, price, availability, images, variants. If that's disabled, read sitemap.xml for product URLs, then fetch each product page and parse the HTML. Shopify renders on the server, so the data is in the HTML and no browser is needed. Rate limits are light, but stay polite: one request per second per domain is a sane default.
How much does e-commerce scraping cost?
Per 1,000 product pages: self-hosted static scraping is about $0.03 in compute, residential proxy bandwidth about $0.40, a static scraping API about $1.10, and a rendered scraping API about $4.50. The rendered tax is the biggest line item — a headless browser per page costs real CPU. A 100,000-product catalog crawled daily is 3 million fetches a month: about $750 at $0.25/1k, about $13,500 at $4.50/1k. Check the raw HTML before you pay for rendering you don't need.
Is it legal to scrape product data from e-commerce sites?
Scraping public data politely is broadly legal in the US, but the site's terms of service matter. Amazon, Walmart, and most marketplaces explicitly prohibit scraping in their ToS, which is a contract risk even when the data is public. The operating rules: prefer an official API when one exists, respect robots.txt and rate limits, don't scrape personal data you don't need, and don't resell content wholesale. When in doubt, ask a lawyer who does internet law.
How do I scrape Walmart or other big retailers?
Walmart is JS-rendered with aggressive anti-bot — CAPTCHAs, rate limits, IP blocks. Check the affiliate API first; if you must scrape, you're in the residential-proxy-plus-rendering tier with strict rate discipline. The same applies to Target, Best Buy, and Home Depot. Expect to be blocked occasionally and build retries with backoff. The web scraping without getting blocked post is the playbook.
How often should I re-scrape prices for price monitoring?
It depends on the category. Groceries reprice almost daily; electronics and fashion weekly; home goods monthly; books rarely. Start at a daily crawl, measure how often prices actually change for your category, and back off until you're catching roughly 95% of changes. Crawling too often wastes money and load; too rarely misses the signal.
What is the best way to scrape e-commerce websites?
The best way is the one that avoids scraping: use the official API when it exists — Amazon PA-API, eBay Browse API, Walmart Affiliate API, Shopify Storefront API. When you must scrape, the pattern is: discover the catalog (sitemap.xml or /products.json), fetch product pages with a polite rate limit, parse into a consistent schema, store the history, and detect changes with a hash comparison. Render a browser only when the data isn't in the raw HTML.
Keep reading
The Complete Proxy Guide for Web Scraping (2026)
The honest 2026 proxy guide: what a proxy does, the four types compared, rotation, detection, real pricing, cost math, when you need one, and working Python code.
Web Scraping APIs in 2026: The Complete Guide
The honest 2026 guide to web scraping APIs: when to buy vs build, the five stages inside a call, real pricing for every major provider, cost math, code, and verdicts.
Building search infrastructure for 1M+ queries a month: the stuff nobody tells you
A deep, no-bullshit technical dive into building search infrastructure that handles over 1 million queries per month. Inverted index internals, posting list compression, WAND/MaxScore scoring, Go hot-path code, five-layer cache hierarchies, mmap and SSD reality, tiered segment merging, tail latency, and the non-obvious stuff that actually kills you at scale.
Found this useful? Cite it as: webscraping.space. “E-commerce Scraping: The Complete 2026 Guide.” https://webscraping.space/blog/scraping-ecommerce. Published 2026-08-14.