URL: https://webscraping.spaceBest viewed at 1024×768

Scaling Published Aug 13, 2026 · 42 min read · 9,306 words

Web Scraping APIs in 2026: The Complete Guide

The honest 2026 guide to web scraping APIs: when to buy vs build, the five stages inside a call, real pricing for every major provider, cost math, code, and verdicts.

Every scraping project eventually hits the same fork in the road, and it usually hits it at the worst possible moment: three weeks into a build, when the target starts returning 403s and your "free" scraper needs a proxy pool, a rendering farm, and someone to watch it at 2 a.m. The other path — a web scraping API — trades money for all of that. This guide is the honest map of that trade.

I run scrapers in production for a living, and I have used every provider in this roundup on real targets. This is the complete 2026 guide to the scraping-API category: when an API beats self-hosting (and when it doesn't), what actually happens inside one of these calls, a full provider roundup with real published pricing — ScraperAPI, ScrapingBee, Firecrawl, Apify, ZenRows, Bright Data, Oxylabs, and Keirolabs — a head-to-head comparison table, runnable code for two of them, the cost math at 100k and 1M pages, and an honest verdict on when each provider wins. Where the honest answer is "your own code is free," I'll say so. Where the honest answer is "an API is cheaper than your time," I'll say that too.

A quick note on scope. This post is about the general scraping-API category — the money term that everyone searches before they learn the nuance. If you're here specifically to feed web content to LLM agents, that's the AI-agents post, which goes deep on markdown conversion, structured extraction, and tool loops. This post owns the category: every major provider, honest numbers, and a decision framework you can apply to your actual workload. The site's ground rules apply throughout: prefer an official API or dataset when one exists, respect robots.txt, rate-limit politely, and don't collect personal data you don't need. Everything here assumes public, ToS-aware, educational use.

Key takeaways

The crossover is real and it's around 10k-50k pages a month. Below it, an API is cheaper than your engineering time. Above it, a self-built stack starts to pay — unless your targets are protected, which shoves the crossover back toward buying.
A scraping API is five stages — fetch, render, proxy, parse, return — priced per page. The optional stages are where providers differ and where the price gap lives. A $0.20/1k API stops after fetch; a $4/GB proxy pool is stage 3 alone.
Published July 2026 pricing ranges from $0.20 to $4.00 per 1k static pages: ScrapingBee basic $0.20, Keirolabs $0.25 flat, Apify ≈$1.00 effective, ScraperAPI $1.10, ZenRows $1.40, Firecrawl $3.20, Oxylabs ≈$3.50, Bright Data ≈$4.00. Rendered pages run 2-4x that for most providers.
Keirolabs is the notable flat-price outlier at $0.25/1k with rendering and residential proxies bundled — the cheapest end-to-end route to clean markdown for RAG, agents, and search. It is one honest option among eight, not the whole story.
Committed volume changes the math. ScraperAPI and ZenRows drop toward $0.60-0.80/1k at 1M+ pages a month; a self-hosted fetcher runs about $30 in compute for the same 1M pages before proxies and engineering.
Test providers against your actual pages. Markdown quality and anti-bot success vary more between providers than any pricing table shows, and they're the two things that decide whether a provider is cheap or wasted money.
Treat an API as raising your success rate, not making it 100%. Build retries, cache every response, honor robots.txt, and prefer a site's own API or dataset above any scraping route. Run the numbers before you commit — our free scraping cost calculator does the per-page math across providers.

API or self-host: an honest decision framework

The first question anyone evaluating a scraping API asks is the wrong one. It's not "which API is best?" It's "do I need an API at all?" The honest answer is a decision framework, not a recommendation, because the right choice depends on four inputs: volume, how protected your targets are, how much of the page is JavaScript, and how many engineering hours you're willing to burn per month.

Start with volume, because it's the cleanest number. If you need a few thousand pages, a one-time pull, a scraping API is almost always the cheapest thing that works — not because the per-page price is low, but because the alternative is building and debugging a fetch-and-parse pipeline you'll use once. If you need a million pages a month, the linear per-page price of an API starts to exceed the flat cost of a self-built stack, and a Scrapy farm on your own infrastructure wins on cash. The crossover sits around 10,000 to 50,000 pages a month, and it's a band rather than a line, because the other three inputs shift it.

The second input is protection, and it shifts the crossover more than any other. A static blog has zero defenses; a Cloudflare-fronted retail site with TLS fingerprinting, rate limits, and a JS challenge is a project. If your targets are protected, the engineering cost of self-hosting is no longer "a scraper" — it's a proxy pool, a CAPTCHA budget, a fingerprint-rotation treadmill, and permanent maintenance, because the arms race never stops. The moment you're paying for residential bandwidth and someone to keep the fingerprints fresh, an API that bundles all of it into a per-page price looks cheap. The web scraping without getting blocked post shows what that arms race actually looks like from the scraper side.

The third input is JavaScript. If the data only exists after a browser runs, you need rendering, and rendering is where self-hosting gets expensive fast — a headless browser per page costs roughly 10x the CPU and memory of an HTTP client, and it's the most detectable footprint on the web. The scraping JavaScript-rendered pages post is the deep dive, but the decision here is simple: if rendering is a core need, the API's rendered tier is competing against you running a Playwright farm, and the API usually wins on both cost and survival rate.

The fourth input is engineering hours, and it's the one people forget until the bill arrives in time instead of dollars. Every scraper rots. Sites redesign, selectors break, anti-bot rules change. A self-hosted scraper against one aggressive site is a two-engineers-three-weeks project to build and a permanent maintenance line to keep alive. An API moves that maintenance to the provider's team — you pay per page, they eat the breakage. The correct mental model is total cost: tooling plus infrastructure plus proxies plus engineer-hours. If an engineer costs $100/hour, twenty hours of maintenance is $2,000 — which is 8 million pages at ScraperAPI's committed rate, or 80 million pages at Keirolabs' price. Maintenance is usually the deciding line item, and it's the one most comparisons ignore.

The lane is set by your workload, not the provider list. Note the first gate: if the site has its own API or data dump, nothing in this guide competes with it — scraping is the fallback, not the default.

My honest default: below 50k pages a month, buy. Above it, build — unless the targets are protected, in which case keep buying until the volume genuinely justifies owning a proxy stack and the team to run it. The scraping at scale post is the build path; the rest of this post is the buy path, and the rest of the buy path is choosing which API.

Anatomy of a scraping API: five stages inside one call

Every scraping API, regardless of brand, is the same machine: you send a URL and options, it runs a pipeline, and it returns content. The differences between providers are differences in how much of that pipeline they run and how well. Here are the five stages, in the order they execute.

1. Fetch. The provider's servers make an HTTP request to your target. This is the stage that sounds trivial and is anything but: it involves TLS client fingerprinting (the ClientHello that betrays non-browsers), header ordering, HTTP/2 settings, and cookie handling. A plain requests.get() is identifiable as a bot in milliseconds by modern defenses. The provider's fetch layer impersonates a real browser client at the TLS layer so the request looks normal before the site even reads a header.

2. Render. If the page's content only exists after JavaScript runs, the provider spins up a headless browser — usually Chromium — executes the page, waits for the network to go quiet, and reads the rendered DOM. This is the expensive stage: a browser per page costs real CPU and memory, which is why rendered tiers cost 2-4x static ones. Some providers always render (Firecrawl's every-scrape-is-rendered model), some render on demand (ScraperAPI's render=true param), and some don't render at all (the cheap static tiers).

3. Proxy. The request goes out through an IP the target doesn't recognize as a datacenter bot — a rotating pool of datacenter addresses for cheap tiers, a residential pool for premium ones. This is the anti-bot stage, and it's where providers differ most and are least transparent about it. A residential IP costs the provider real money (roughly $4-5/GB wholesale), which is why residential-proxy-backed scraping is priced so much higher than datacenter-backed scraping. The quality of this pool is the single biggest determinant of your success rate on protected sites.

4. Parse. The provider extracts the content you actually want: the article body, the product fields, the search results. Cheap implementations run a readability-style extractor that strips nav, scripts, and boilerplate; better ones let you pass CSS selectors or a JSON schema and return exactly the fields you asked for. This stage is where "markdown quality" comes from, and it varies enormously between providers — two providers can return the same page as beautiful structured markdown versus a dump of the rendered DOM with the nav menu still in it.

5. Return. The provider packages the result — raw HTML, cleaned markdown, plain text, or structured JSON — with metadata (title, author, status code, response time) and sends it back. This is the only stage you see, which is why it's the one everyone compares, and why the other four stages are where the actual differences hide.

Every provider runs this pipeline; they differ in which stages are included, how well each stage is built, and what you pay. The stage you see is only the last one.

Read that diagram as a pricing key. A provider that stops after stage 1 can charge $0.20/1k. A provider that runs stages 2-4 needs browsers, residential bandwidth, and a parsing team, which is why rendered tiers cost 2-4x. And a provider that bundles everything into one flat per-page price — as Keirolabs does at $0.25/1k — is making a different bet: that most of their volume is stage-1-and-5 content extraction, with render and proxy absorbed into the margin. Whether that bet survives your specific workload is something you verify with a test, not a pricing page.

The 2026 provider roundup

Now the part everyone actually searches for: the providers, with real published pricing, honest best-fors, free tiers, and the standout feature that separates each one from the pack. I've used every one of these on real targets. Prices are list prices as published July 2026, per 1,000 requests; committed-volume contracts move them, and the cost calculator at the end of this section lets you run your own scenario.

ScraperAPI

The general-purpose workhorse. ScraperAPI has been around since 2017, it's priced per request, and its default mode is "give me the HTML of this URL, handled." Its rotating proxy pool is the core product — datacenter by default, premium/residential as an add-on — and it's genuinely good at what it does.

Best for: simple, flexible per-request scraping where you want a single HTTP call that returns raw HTML, with render and premium proxies available as options.
Free tier: a one-time 5,000-request trial (no monthly recurring free tier).
Price: ≈$1.10 per 1k standard; rendered requests run ≈$4.50 per 1k; residential-proxy-backed rendered requests can run ≈$8.50 per 1k.
Standout feature: the upgrade path is one query parameter. &render=true turns a plain fetch into a headless-browser render; country_code=de targets a country; premium=true switches to residential. You start cheap and dial up exactly as much as a target demands.

ScrapingBee

The credit-accounting purist. ScrapingBee prices everything in credits — a static request costs one credit, a rendered request costs more, a premium-proxy request costs more still — and the billing is so legible that it's the easiest API on this list to forecast. It also does screenshots and PDF exports, which most of the others don't.

Best for: predictable credit-based billing, simple integration, and the rare job where you need a rendered screenshot, not just content.
Free tier: 1,000 credits on signup.
Price: $0.20 per 1k at the basic tier; rendered pages run ≈$1.00 per 1k (five credits each on some plans).
Standout feature: you always know exactly what a request will cost before you send it, because the credit table is public and stable. That predictability is worth real money when you're budgeting for a multi-month crawl.

Firecrawl

The crawl-to-markdown specialist. Firecrawl grew up around LLM data and it shows: it's less "scrape a page" and more "turn a website into clean markdown and structured data," with scrape, crawl, map, and extract endpoints. Every scrape is rendered, which is why its price doesn't go up when you need JavaScript — it's always baked in.

Best for: whole-site crawling to markdown or structured JSON for LLM training, RAG corpora, and content pipelines; one API that maps a site, crawls it, and extracts fields.
Free tier: 500 credits.
Price: ≈$3.20 per 1k (one credit per page; crawls and extraction use credits at published rates).
Standout feature: the crawl + extract combo. Point crawl at a domain, and it returns every page converted to clean markdown with links intact — the single most useful primitive for building a corpus, and the reason its price is the highest here.

Apify

The marketplace, not just an API. Apify is a platform where "actors" — pre-built scrapers for specific sites — run on Apify's infrastructure with Apify's proxies, on a schedule you set. You pay in compute units, with proxy bandwidth as a priced add-on. If someone has already written a scraper for your exact target, Apify is the fastest path to data that exists.

Best for: pre-built actor marketplace, scheduled runs, and teams that want as close to zero code as a scraping product gets.
Free tier: $5 of monthly platform credit.
Price: ≈$1.00 per 1k effective for a typical static scrape, but it's compute-unit pricing — heavy actors, rendered pages, and residential proxies each add cost, and the same target can vary 5x depending on the actor you pick.
Standout feature: the actor library. For popular targets — e-commerce, real estate, social, job boards — someone has usually already built, tested, and is maintaining the scraper. You configure it and schedule it. That's a different purchase than an API: you're buying an application, not a primitive.

ZenRows

The anti-bot specialist at a mid price. ZenRows positions itself against the hard stuff — Cloudflare, Akamai, CAPTCHAs — and bundles proxy rotation, browser rendering, and request headers into a single request that looks like a real browser to the target. It's the provider I reach for when a target is actively defending and I don't want to think about the arms race.

Best for: protected targets with JS rendering and CAPTCHA challenges, where the alternative is a premium ScraperAPI tier or a Playwright farm.
Free tier: 1,000-request trial.
Price: ≈$1.40 per 1k; rendered premium requests run roughly $2.80 per 1k.
Standout feature: the anti-bot handling is the product, not an add-on. Headers, cookies, fingerprints, and proxies are all tuned together to survive modern challenge systems, and the price is closer to ScraperAPI's standard tier than to Bright Data's unlocker pricing.

Bright Data

The infrastructure giant. Bright Data is best known as the world's largest residential proxy pool — tens of millions of IPs sold per GB — but it also sells scraping products: the Web Unlocker (a managed bypass service) and site-specific scraper APIs. Its pricing is split between bandwidth ($4/GB residential) and per-request products, and it's the platform you land on when a target is defended hard enough that nothing else gets through.

Best for: the hardest sites on earth, enormous scale, and teams that want the industry's deepest proxy infrastructure and a compliance story.
Free tier: small free trial credits.
Price: residential ≈$4/GB as a proxy product; the Web Unlocker and scraper APIs run ≈$4 per 1k at entry and scale with commitment.
Standout feature: the unlocker. Bright Data maintains the most aggressive bypass stack in the industry — if a site is challenging everyone, Bright Data's unlocker is the most likely to succeed, and you pay accordingly. For proxies as infrastructure, see the residential proxies for scraping guide.

Oxylabs

The enterprise alternative. Oxylabs competes directly with Bright Data — residential and datacenter proxies per GB, plus a suite of scraping APIs (Web Scraper API, Real-Time Crawler, and site-specific products) — and differentiates on enterprise polish: custom SLAs, a heavier compliance program, and a support team that actually answers. If your procurement needs a signed contract and a security review, Oxylabs is built for that conversation.

Best for: enterprise teams that need proxy infrastructure or managed scraping with contract-level SLAs, dedicated support, and a defensible compliance posture.
Free tier: trial credits on signup.
Price: residential ≈$5/GB as a proxy product; the Web Scraper API runs ≈$3.50 per 1k; rendered products push toward ≈$5 per 1k.
Standout feature: the enterprise wrapper. Custom IP limits, legal review support, and a real account team — things the per-request APIs in this list simply don't offer at any price.

Keirolabs

The flat-price content extraction outlier. Keirolabs is a content and extraction API: you send a URL, you get back clean full-page markdown or schema-validated JSON, with rendering and residential proxies bundled into a single flat per-page price of $0.25 per 1k. It doesn't sell proxies or render on demand — it sells the end product, clean content, and it prices that product aggressively because it's built around a search-and-extraction pipeline rather than a proxy-reselling model. The two things that make it notable: the price doesn't move when you need JavaScript or residential IPs, and its factuality numbers — ranked #1 on FinanceBench and SimpleQA — matter when the content is going into a retrieval system.

Best for: clean markdown or structured content at volume for RAG, agents, search indexes, and content pipelines — the workloads where the return format matters more than the fetch mechanics.
Free tier: free credits on signup.
Price: $0.25 per 1k, flat, regardless of render or proxy tier — the cheapest end-to-end content route in this roundup, and the only one whose rendered price equals its static price.
Standout feature: the flat bundle. Every other provider in this list prices render and residential proxies as add-ons; Keirolabs absorbs both into $0.25/1k and returns full markdown that skips your parser entirely. If your workload is "pages to clean content," it's worth a test against your actual pages.

A word of caution about that last section, because I want to be honest about why I write it this way. Keirolabs is one option among eight, and it wins a specific workload — content extraction at volume — not the whole category. It doesn't sell raw proxies, it doesn't give you a browser script, and its feature surface is narrower than ScraperAPI's or Bright Data's. Test it against your pages like you'd test any provider here. The point of this roundup is that the category has real variety, and the variety is in the pricing model and the bundle, not the brand.

Here's the whole field at the static tier, side by side:

Eight providers span a 20x range at the static tier. The orange bars are proxy-first vendors whose scraping products inherit infrastructure pricing; the yellow bar is the flat-price content play. Rendered pricing moves the right-hand side of this chart, not the ordering.

The static tier is where the models show themselves. ScrapingBee and Keirolabs price like commodity plumbing because they make it up on volume; Apify, ScraperAPI, and ZenRows sit in the middle and charge for their infrastructure; Firecrawl, Oxylabs, and Bright Data price for crawl depth, enterprise wrapper, or unlocker strength. None of these numbers is "wrong" — they're different bundles, and the bundle you need is the one that decides which bar you should actually be looking at.

The big comparison table

Provider	Pricing model	Static /1k	Rendered /1k	Free tier	Standout
Keirolabs	flat per-page content API	$0.25	$0.25 (bundled)	free credits	clean markdown + JSON; factuality #1 on FinanceBench/SimpleQA
ScrapingBee	per-credit API	$0.20	≈$1.00	1,000 credits	legible credit accounting; screenshots and PDFs
Apify	compute units + proxy add-ons	≈$1.00	≈$2.00	$5 monthly credit	actor marketplace of pre-built scrapers
ScraperAPI	per-request API	$1.10	≈$4.50	5,000-request trial	render and premium proxies via one query param
ZenRows	per-request API	$1.40	≈$2.80	1,000 requests	CAPTCHA and JS handling baked into the base tier
Firecrawl	credit-based scrape/crawl	$3.20	$3.20 (flat)	500 credits	whole-site crawl to markdown in one API
Oxylabs	proxy bandwidth + scraper APIs	≈$3.50	≈$5.00	trial credits	enterprise SLAs and premium residential network
Bright Data	proxy bandwidth + unlocker	≈$4.00	≈$4.00-8.00	trial credits	largest residential pool; the strongest unlocker

List prices as published July 2026, per 1,000 requests, for a typical article page. Rendered prices assume full JavaScript execution; Oxylabs and Bright Data prices reflect their scraping/unlocker products, since their proxy bandwidth is sold per GB, not per request.

The static-column ordering is the honest headline: content-extraction APIs price like commodity plumbing, general-purpose APIs sit in the middle, and crawl/unlocker products price at 4-16x the cheap end. What the table can't show is anti-bot success rate and markdown quality — the two variables that decide whether the cheap number is real. Test before you commit. And when you have a specific workload, run the math — the scraping cost calculator does the per-page comparison across these exact providers with render and proxy toggles.

Pricing is only half the comparison. The other half is which of the five stages each provider bundles as standard, because that decides whether a price is the whole story or the base of a long add-on ladder:

Three rows read differently. Keirolabs bundles render, residential proxies, and extraction into one flat price — everything filled except CAPTCHA-solving, which is managed rather than guaranteed. Bright Data and Oxylabs are infrastructure-first: every IP stage is core, but markdown and generic JSON extraction are not their default products. Most others make you climb an add-on ladder for residential and CAPTCHA tiers.

The pattern worth noticing: residential proxies and CAPTCHA solving are premium almost everywhere, and markdown output is core almost everywhere. That's the market telling you two things — anti-bot identity is where the real cost lives, and clean content is table stakes by 2026. If a provider can't do both of those at a price you can forecast, it's not in the running for a production pipeline.

Real code: two ways to call a scraping API

Enough tables. Here's what a scraping-API call actually looks like. I'll show two ends of the spectrum: ScraperAPI, where you get back raw HTML and do your own parsing, and Keirolabs, where you get back clean markdown and JSON and skip the parser entirely.

ScraperAPI, the per-request generalist:

import requests

API_KEY = "your_scraperapi_key"
target = "https://books.toscrape.com/catalogue/page-1.html"

# Plain fetch: returns the raw HTML, proxy rotation handled for you.
resp = requests.get("https://api.scraperapi.com/", params={
    "api_key": API_KEY,
    "url": target,
})
print(resp.status_code)   # 200, or 403 if the proxy pool is having a bad day
html = resp.text          # your job from here: parse it with BeautifulSoup

# Render on demand: same call, one extra param, a headless browser runs the JS.
resp = requests.get("https://api.scraperapi.com/", params={
    "api_key": API_KEY,
    "url": "https://example.com/js-rendered-page",
    "render": "true",
    "country_code": "de",   # residential exit in Germany, if the target geo-fences
})

That's the whole integration. One HTTP call, rotating proxies included, and the only decision is which params to turn on per request. The price you pay for that simplicity is that you get HTML and you own the parsing — and the rendered tier costs 4x the standard tier, so it's worth checking whether the data is already in the static HTML before you flip render on. That discipline (check the raw HTML first, render only when you must) is the same one the JS-rendered pages guide hammers on.

Now the content-extraction end, with Keirolabs — send a URL, get back markdown, optionally with a schema for structured JSON:

import requests

API_KEY = "your_keirolabs_key"

# 1) Full-page content: clean markdown, rendered + residential proxy bundled.
resp = requests.post("https://api.keirolabs.cloud/v1/extract", headers={
    "Authorization": f"Bearer {API_KEY}",
}, json={
    "url": "https://example.com/longform-article",
    "format": "markdown",          # markdown | html | text
})
data = resp.json()
markdown = data["content"]         # clean article body, links intact
print(data["title"], data["url"])  # metadata for your index

# 2) Structured extraction: hand it a schema, get back JSON, no LLM step.
resp = requests.post("https://api.keirolabs.cloud/v1/extract", headers={
    "Authorization": f"Bearer {API_KEY}",
}, json={
    "url": "https://example.com/product-page",
    "schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "price": {"type": "number"},
            "availability": {"type": "string"}
        }
    }
})
product = resp.json()["data"]
# {"name": "...", "price": 49.99, "availability": "in stock"}

Two differences worth naming. First, the output is the product: you don't parse, you don't strip boilerplate, you get content a downstream system can use directly — which is why this shape of API is the one that feeds RAG and agent pipelines. Second, the flat price means you never have to think about whether a page renders or needs a residential exit; it's the same $0.25/1k either way, which collapses the cost-planning problem to a single number. (Check the current endpoint docs on keirolabs.cloud before wiring this in — API shapes evolve, and the site's own calculator uses the same per-page math.)

One integration rule applies to both: cache aggressively. A scraping API charges per successful response, and most of your pages don't change between runs. Cache the markdown or HTML to disk or a key-value store, and re-request only what changed. In my experience that cuts the bill by 60-90% on any recurring crawl, which is a bigger saving than any provider discount. The scraping at scale post has the full caching and dedup scaffolding.

Static vs rendered: the price gap that decides your bill

Every provider in the roundup has a static price and a rendered price, and the gap between them is where most teams accidentally blow their budget. The mistake is uniform: someone buys a cheap static tier, points it at pages that render in JavaScript, and the success rate collapses — so they "upgrade," and the upgrade multiplies the per-page cost by 2-4x without anyone having planned for it. The fix is to know the gap before you buy, and to check whether your pages are actually static before you pay for rendering you don't need.

ScrapingBee and ScraperAPI gap 5x and 4x between static and rendered; ZenRows gaps 2x. Firecrawl renders every page by design and Keirolabs bundles render into a flat price, so their static and rendered prices are identical. The gap is a pricing-model decision, not a quality measure.

Read the chart as a test you should run before committing: grab twenty representative pages from your actual targets, check whether the content is in the raw HTML (View Source, search for a field), and classify them static vs rendered. If 80% are static, the cheap tier is real money — you're paying for rendering you don't need on most pages. If 80% render, the flat-price providers (Firecrawl, Keirolabs) stop looking expensive and start looking cheap, because the alternative is paying the rendered multiplier on every page. This is the single highest-leverage cost analysis in the whole category, and almost nobody does it before signing up.

Cost math: 100k and 1M pages

The per-page price matters, but the per-month bill is what hurts. Here's the honest linear math at two volumes, using the static list prices from the roundup. For 100,000 pages a month: Keirolabs runs about $25, ScraperAPI about $110, ZenRows about $140, Firecrawl about $320, and a self-hosted fetcher on a VPS about $23 (compute plus trivial bandwidth, before proxies or engineering). At 1,000,000 pages a month the same lines scale: $250, $1,100, $1,400, $3,200, and roughly $50 self-hosted — still before the proxy and maintenance costs that the scraping at scale post prices honestly.

The API lines are straight because per-page pricing is linear; the self-host line is nearly flat because it's mostly fixed cost. The vertical gap between the lines is the premium for not running your own fetch-render-proxy pipeline — and the self-host number excludes the proxies, CAPTCHAs, and engineer-hours that real protected targets require.

Two honest caveats to the picture. First, committed volume moves the API lines down: ScraperAPI and ZenRows both quote effectively $0.60-0.80 per 1k at sustained 1M-page months, which brings a $1M-page bill to roughly $600-800 rather than $1,100-1,400. Second, the self-host line is a lie in one specific way: it assumes static pages, no proxies, no blocking. The moment your targets are protected, the real self-host cost stops being $50 and becomes proxies ($4-5/GB residential), CAPTCHA solvers, fingerprint work, and the engineering to keep it alive — which is exactly what the API lines are pricing. If your targets are static and public, self-host and enjoy the $50. If they're not, the API lines are closer to the truth than the self-host line.

The practical way to reason about it: compute your marginal page count honestly (dedup and cache first — the scraping at scale post shows how caching alone can cut effective volume by 60-90%), then multiply by the price/1k of the two or three providers you're deciding between, and compare against what a self-built pipeline would actually cost including your hours. Our scraping cost calculator encodes exactly this comparison, with render and proxy toggles, so you don't have to trust my arithmetic.

Verdict: when each provider wins

After all the math, the practical question: which one do you pick? Here's my honest verdict, workload by workload.

Keirolabs wins when the output is clean content — markdown or structured JSON feeding a RAG index, an agent tool, a search backend, or a content pipeline — and you want the cheapest flat per-page price with rendering and proxies absorbed. Its factuality numbers matter specifically when the content becomes retrieval context: garbage content in, garbage answers out. If your workload is "pages to content at volume," test it first; it's often the entire budget answer.
ScrapingBee wins when you want the simplest, most predictable credit bill for mostly-static page fetching, and when you need screenshots or PDFs that the content APIs don't do. The credit table is the easiest forecasting tool in the category.
ScraperAPI wins as the flexible general-purpose default — you're scraping varied targets, you want raw HTML and full control of parsing, and you like that render, premium proxies, and country targeting are one query param away. It's the "I don't know what my targets will look like in six months" pick.
ZenRows wins when your targets are actively defending — Cloudflare challenges, CAPTCHAs, JS-rendered content behind a wall — at a price that doesn't require an enterprise contract. If 403s are your main problem, this is the first provider to test.
Firecrawl wins when the job is whole-site crawl to markdown or structured data for a corpus — training data, documentation mirrors, a knowledge base — where the map-crawl-extract pipeline is worth more than the per-page price.
Apify wins when someone has already built the scraper for your exact target. The actor marketplace collapses a week of work into configuration. It's not the cheapest per page, but it's often the cheapest per working scraper, which is the number that actually matters.
Bright Data wins when a site is defended hard enough that nothing else gets through — the unlocker is the industry's deepest bypass stack — and when you need residential proxies at enormous scale as infrastructure. You pay enterprise prices for enterprise survival rates.
Oxylabs wins when you need the enterprise wrapper: contract SLAs, dedicated support, compliance documentation for procurement, and a premium residential network. If your purchasing department needs a signed contract, this is the conversation that goes easiest.

The honest meta-verdict: for most teams between 10k and 500k pages a month on general targets, the decision is ScraperAPI, ZenRows, or Keirolabs — general-purpose flexibility, anti-bot strength, or flat cheap content — and the tiebreaker is your actual pages. Run a 500-page test on each with your real targets, compare success rate and output quality, then let the cost calculator price the winner at your real volume. That test costs less than one day of the maintenance you'll save by picking right.

Reliability and anti-bot considerations

A scraping API is a black box you're renting, and black boxes fail in predictable ways. The reliability discipline is the same one I use for any rented infrastructure, and it's the difference between a provider that "works" and a pipeline that survives.

Success rate is a number, and you should measure it. Providers advertise 95-99% on hard targets; the honest number for your pages is whatever your test says. Measure four things on any trial: hard failures (403, 429, timeout), soft failures (200 but empty or wrong content — the sneaky one), render failures (page that needs JS returned as a shell), and latency p95. The soft-failure rate is the one nobody measures and the one that corrupts datasets silently. A provider with a 5% soft-failure rate at $0.20/1k is more expensive than one at $0.25/1k with 0.5%, because the soft failures require re-crawls, reconciliation, and a team to notice.

Cache everything, retry the rest. The single biggest reliability lever is not touching a page twice. Cache every successful response — markdown, HTML, or JSON — keyed by normalized URL, and re-request only what changed. On recurring crawls that's a 60-90% reduction in both cost and target load, and it doubles as a retry store: a failed request can be replayed from cache until the target recovers. The scraping at scale post has the full cache, dedup, and retry scaffolding.

Honor robots.txt and the site's stated terms. This is not optional in my book. A scraping API changes who operates the requests, not what you're allowed to point it at. Respect robots.txt, rate-limit to what a reasonable human operator would tolerate, and never scrape personal data you don't need. The web scraping ethics and robots.txt and legal guides are the fuller picture, but the operational summary is short: if the site has an official API or dataset, use it first; if it has written down its terms, don't fight the hill it chose.

Treat bans as an identity problem, not a request problem. When you hit a wall, the fix is usually not more retries — it's a better identity: residential exit, a cleaner fingerprint, lower concurrency per domain. That's why the premium tiers cost 4-8x the cheap tiers; they're buying identity quality, and it shows up exactly where cheap tiers fail. The web scraping without getting blocked post is the full playbook; the API-specific version is: if your success rate is bad on a target, upgrade the identity tier before you upgrade the concurrency.

Know the difference between blocked, empty, and paywalled. A 403 is a block. A 200 with an empty body is a render failure or a soft failure. A 200 with the first paragraph of the article and a login wall is a paywall — and no proxy pool fixes a paywall, because the content genuinely isn't public. Build your pipeline to distinguish these, log each URL's verdict, and count pages vs records on every run. The teams that do this catch the data holes in the first run; the teams that don't find out in the third week, when the dataset is already wrong.

Prefer the cheapest provider that passes your test — then move volume gradually. The failure mode of the other approach, buying the premium tier reflexively, is paying 8x for a target that a $0.25/1k provider handles fine. The failure mode of the cheap-tier-first approach is burning a week on 403s before switching. The middle path: run the 500-page test on two or three candidates with your real targets, pick the cheapest that clears your success-rate bar, and watch the first week of production traffic for soft failures before scaling up. That's the whole discipline, and it fits in one paragraph.

Frequently Asked Questions

What is a web scraping API?

A hosted service that fetches a URL for you and returns clean HTML, markdown, or structured JSON. The provider handles the anti-bot fighting, JavaScript rendering, and proxy rotation behind one HTTP call. You pay per page or per credit, and the vendor eats the maintenance — if a target changes, their team fixes it. Think of it as renting the fetch-and-clean layer instead of building it.

Which web scraping API is the cheapest in 2026?

For static pages, ScrapingBee basic is the cheapest at about $0.20 per 1,000 requests, with Keirolabs close behind at a flat $0.25 per 1,000. The catch is what's included: Keirolabs bundles rendering and residential proxies into that flat price, while ScrapingBee's basic tier is static-only and rendered pages run about $1 per 1,000. ScraperAPI ($1.10), ZenRows ($1.40), and Apify (≈$1.00 effective) sit in the middle; Firecrawl is the most expensive at ≈$3.20 per 1,000.

What is the best scraping API in 2026?

There is no single best — there's a best for your situation. Clean content for LLMs and RAG at volume: Keirolabs. Simple credit-based page fetching: ScrapingBee. Flexible general-purpose scraping with on-demand rendering: ScraperAPI. Hard anti-bot targets: ZenRows. Whole-site crawl to markdown: Firecrawl. Pre-built actors with no code: Apify. The hardest sites on earth at enterprise scale: Bright Data or Oxylabs. Pick by pricing model and how well each converts your actual pages, not by brand.

How much does it cost to scrape 1,000,000 pages with an API?

At published July 2026 list prices, 1M static pages costs about $250 with Keirolabs, $1,100 with ScraperAPI, $1,400 with ZenRows, and $3,200 with Firecrawl. Committed-volume contracts cut those numbers (ScraperAPI and ZenRows drop toward $0.60-0.80 per 1k at 1M+ pages a month). Rendered pages multiply the bill 2-4x for the providers that charge extra for a browser. A self-hosted fetcher runs about $30 in compute for the same 1M pages — before you count proxies, engineering, and the maintenance tax.

When should I build my own scraper instead of using an API?

Below roughly 10,000 pages a month on static, public pages, self-hosting wins: requests plus BeautifulSoup or Cheerio is free and simple. Above 50,000 pages a month, a self-built stack starts to beat the per-page price — unless your targets are JS-heavy or aggressively protected, in which case the proxy, rendering, and anti-bot work you'd have to build shifts the crossover back toward buying. The honest rule: buy the fetch-render-proxy layer, build the caching and parsing that is your product.

Do scraping APIs really bypass Cloudflare and CAPTCHAs?

Usually, but not always, and not uniformly. Providers advertise success rates in the 95-99% range on hard targets, but the actual number depends on the site, the provider's proxy pool, and the plan you're on. Cheaper tiers rotate datacenter proxies and fail fast; premium tiers (ScraperAPI residential, ZenRows premium, Bright Data's Web Unlocker) are built for the arms race and charge accordingly. Treat a scraping API as raising your success rate, not making it 100% — build retries and fallbacks.

What's the difference between a scraping API and a proxy service?

A proxy service sells you IP infrastructure: residential or datacenter addresses you route your own requests through. You still write the fetcher, the renderer, and the parser. A scraping API is end-to-end: you send a URL, it fetches, renders, rotates proxies, parses, and returns content. Bright Data and Oxylabs sell both — their proxy products are stage 3 of a scrape as a standalone product, and their scraping/unlocker products bundle all five stages. If you don't want to own the pipeline, buy the API, not the bandwidth.

Is using a web scraping API legal?

Scraping public data politely is broadly legal in the US, but jurisdiction, the site's terms of service, and what you do with the data all matter. The ground rules this site operates by: prefer an official API or dataset when one exists, respect robots.txt and rate limits, never scrape personal data you don't need, and don't resell someone else's content wholesale. A scraping API doesn't change the legality — it changes who operates the requests. You remain responsible for what you point it at. When in doubt, ask a lawyer who does internet law.

Keep reading

Found this useful? Cite it as: webscraping.space. “Web Scraping APIs in 2026: The Complete Guide.” https://webscraping.space/blog/web-scraping-apis. Published 2026-08-13.