Apify vs Spider vs Firecrawl vs Crawlee: Web Scrapers 2026

TL;DR + 5-branch decision tree

• Need a scraper for a well-known site (Amazon, Twitter, Instagram, Google SERPs)? Search Apify Store first — there are almost 29,000 pre-built actors and yours probably exists. Install Apify Actor MCP to call them from an agent.
• Need a hosted API that returns clean markdown for an LLM? Pick Firecrawl. Mature MCP, Apache 2.0 core, /scrape, /crawl, /map, /search, /extract — covers most agent-facing read jobs.
• Crawling at extreme volume where compute is your biggest cost? Spider’s Rust core and pay-per-GB pricing become attractive at the scale where Node crawlers spend more on EC2 than on payroll.
• Want to own the entire pipeline, run on your own workers, and skip the SaaS bill? Crawlee is the framework. Apache 2.0, Node and Python, 23k+ stars, ships with PlaywrightCrawler, PuppeteerCrawler, CheerioCrawler, request queues, and proxy rotation.
• Want to describe what you want in English instead of writing CSS selectors? ScrapeGraph wraps an LLM around a Playwright pipeline. MIT, 24.9k stars, BYO model (OpenAI, Anthropic, Groq, Gemini, or Ollama for local).

The five are not substitutes. Most production scraping stacks end up using two — typically Firecrawl or Apify for the hosted surface plus Crawlee for the long-tail bulk jobs, or Firecrawl plus ScrapeGraph when LLM extraction is the bottleneck. The expensive decision is which shape fits your workload; once that’s set, the brand inside the shape is a 30-minute test.

Five shapes of scraping in 2026

Web scraping in 2026 looks nothing like it did in 2020. The category split into five distinct shapes, and the right pick depends almost entirely on which shape matches the job — not on which tool has the longer feature list.

1. Pre-built scraper marketplace. Apify Store hosts close to 29,000 ready-made “actors” that scrape specific sites: Amazon listings, Google SERPs, Instagram profiles, Twitter timelines, LinkedIn jobs, every top-100 e-commerce site. You don’t write code — you rent someone else’s already-debugged scraper, supply input, pay per compute unit. The economic shape is “buy, not build,” and it’s the winning play when the long-tail of site-specific quirks would cost you more to solve than the per-run bill.

2. Hosted SaaS scraping API. Firecrawl is the clearest example. You POST a URL, you get back markdown or structured JSON. The vendor owns the browser, the proxies, the anti-bot mitigation, the rendering decisions. You own the key and the bill. This is the default shape for LLM-facing scraping in 2026 because the output is already in the format models expect, and the latency is acceptable for interactive agent loops. Anycrawl, Browserbase, ScrapingBee, and ZenRows live in this same shape.

3. High-performance Rust API. Spider is the most prominent example of a different bet: build the crawler in a systems language so the per-page cost is lower at very high concurrency. The hosted version emphasises raw throughput and the bandwidth-plus-compute pricing model is designed for organisations crawling billions of pages. Adjacent: many engineering teams adopt Rust-cored infrastructure once they hit the “our scrapers cost more than our engineers” threshold.

4. Build-it-yourself library. Crawlee is the canonical framework — also from the Apify team but a fundamentally different product. You install it via npm or pip, write your extraction code, run it on your own workers, and own the cost curve. The framework gives you the hard primitives (request queues, session pools, proxy rotation, browser orchestration) so you spend your time on the site-specific logic, not on the plumbing. The licence is Apache 2.0 and the trade is real: you save SaaS money, you pay in operational time. Scrapy is the Python-only sibling shape; Crawlee’s claim to fame is the headless-browser story is first-class.

5. AI extraction. ScrapeGraph is the cleanest expression of the LLM-first thesis: don’t write CSS selectors at all, describe what you want in English, let the model figure out where on the page the fields live. The library is Python, MIT-licensed, and model-agnostic (OpenAI, Anthropic, Groq, Gemini, or Ollama for fully-local extraction). Firecrawl’s /extract endpoint is the hosted-API expression of the same idea — and we cover the trade-offs between them in the dedicated section below.

These five shapes overlap at the edges. Apify ships actors that internally use Crawlee. Crawlee can call Firecrawl as one of many fetchers. ScrapeGraph can be wrapped in a Crawlee pipeline. The shapes are mental models for picking, not airtight technical categories. If you’re new to MCP itself, our What is MCP primer covers the protocol the Firecrawl and Apify servers speak.

Side-by-side matrix

Every cell is sourced from the official repo, vendor docs, or pricing page as of May 2026. Star counts and pricing move; always confirm at the source before committing.

Three observations. First, only Firecrawl and Apify have first-party MCP servers on this directory — for the others, agent integration goes through community wrappers or direct HTTP. Second, the open-vs-hosted axis is more nuanced than “OSS good, SaaS bad”; Crawlee and ScrapeGraph are libraries (you operate them), Firecrawl gives you both, and Apify and Spider are SaaS-first with OSS components. Third, AI extraction is now table stakes — four of five ship it; the question is whether you want it bundled with the scraper (Firecrawl, ScrapeGraph) or you pick the model separately (Apify per-actor, Crawlee plus your own code).

Apify — install + recipe

Apify Actorapify

Official1-Click ReadyRemote

Apify Actor offers 4,000+ cloud tools for web scraping. Scrape any website for data from e-commerce, social media, maps,

search webdeveloper tools72210

What it does best

Apify wins when the answer to “which site are you scraping?” is on the well-known list — Amazon products, Google SERPs, Instagram profiles, LinkedIn jobs, Booking.com listings, Walmart catalogs. Apify Store has close to 29,000 pre-built actors and the most popular ones are battle-tested against site changes far more often than any in-house scraper. The economic model is “rent the scraper somebody already debugged.” You supply input (a search term, a profile URL, a category), the actor returns structured JSON, you pay per compute unit consumed. The MCP integration on the apify-actor server above lets an AI agent invoke any actor in your account by name and read the results back into context.

Pick this if you...

• Need to scrape a well-known site and prefer rent-over-build for site-specific scrapers
• Want managed infrastructure with scheduling, alerts, versioning, and a dashboard out of the box
• Plan to graduate from local Crawlee scripts to scheduled cloud jobs without re-writing the scraper
• Want an AI agent to invoke a known actor by name from a Cursor or Claude Code chat without writing glue code

Recipe: scrape Google SERPs for a list of keywords via MCP

With Apify Actor MCP installed and your APIFY_TOKEN set, in any MCP client paste this prompt:

Use the Apify Actor MCP. Run the apify/google-search-scraper
actor on this input:

  {
    "queries": "best espresso machine 2026\nbest grinder 2026",
    "countryCode": "us",
    "maxPagesPerQuery": 1,
    "resultsPerPage": 10
  }

Wait for the run to complete (poll if needed), then return the
top 5 organic results for each query — title, URL, and snippet
— as a JSON array.

The agent calls call-actor with the input above, polls the run status, then fetches the dataset items when the run finishes. The output drops into the conversation as JSON you can pipe into whatever lives downstream. The same pattern works for any of the ~29k actors in the store; the shape of the call is fixed and only the input schema changes per actor.

Skip it if...

You’re scraping a site nobody has built an actor for, or your volume is high enough that paying $0.20 per compute unit becomes a meaningful line item. The first case sends you to Crawlee or Firecrawl; the second case sends you to Crawlee on your own infrastructure or to Spider for throughput. Apify Free gives you $5 of credits to test the store before committing.

Spider — install + recipe

What it does best

Spider is the answer for crawling at the scale where compute cost becomes a budget conversation. The crawler is written in Rust with async concurrency from the ground up — there are no garbage-collection pauses, no Node event-loop contention, and the per-page CPU cost is materially lower than what JS-based crawlers spend. The hosted API quotes peaks of 100k+ pages per second under best-case conditions and applies pay-per-GB bandwidth plus per-minute compute billing, which lines up cleanly with how very large crawlers actually consume resources. Built-in anti-bot rotation across proxies in 199 countries means you usually don’t bring your own proxy stack at this scale.

Pick this if you...

• Crawl hundreds of millions of pages a month and compute is in your top three line items
• Want to pay only for successful requests with no subscription floor
• Need built-in proxy rotation across many countries without managing the proxy stack yourself
• Are open to a closed-source hosted API but want the option to self-host the open-core spider-rs crawler later

Recipe: crawl a docs site and return markdown

Spider doesn’t ship a first-party MCP server, so use it via curl or its SDK. Here’s the minimal HTTP shape:

curl -X POST https://api.spider.cloud/crawl \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 100,
    "return_format": "markdown",
    "depth": 2
  }'

The response is a JSON array of pages with markdown bodies, links, and metadata. Wire this into a Crawlee job for the long-tail or call it directly from an agent via a thin custom MCP wrapper — community examples are floating around but no production-grade Spider MCP exists yet on this directory. Track the web-scraping category for new entries.

Skip it if...

You don’t need extreme throughput. For a few thousand pages a day, Firecrawl is simpler and the markdown output quality is generally cleaner. For specific well-known sites, an Apify actor is already debugged. Spider is the right pick when you’re at the scale where the Rust core matters; otherwise it’s overkill.

Firecrawl — install + recipe

Firecrawlmendableai

Official1-Click Ready

Unlock AI-ready web data with Firecrawl: scrape any website, handle dynamic content, and automate web scraping for resea

browser automation3.0k125

What it does best

Firecrawl is the default LLM-facing scraper in 2026 for one clear reason: the markdown it returns is consistently cleaner than what raw HTML-to-Markdown conversion produces. Boilerplate gets stripped, navigation gets dropped, the body gets isolated — and the agent reads what humans would call “the content” rather than a wall of nav links and footer cruft. The five endpoints (/scrape for one URL, /crawl for a site, /map for URL discovery, /search for web search, /extract for structured data) cover the vast majority of read jobs an LLM agent needs. The MCP server at firecrawl/firecrawl-mcp-server wraps all five behind a single stdio install, and the Apache 2.0 core at github.com/firecrawl/firecrawl crosses 100k stars — both hosted and self-host paths are real.

Pick this if you...

• Build an LLM agent that reads public web pages and want the cleanest markdown output without writing converters
• Need /scrape, /crawl, /map, /search, and /extract all behind one auth boundary and one MCP tool surface
• Want the option to self-host with Apache 2.0 once your volume justifies running your own proxy stack
• Are okay starting on the free 1,000-credit tier and graduating to Hobby ($16/mo) or Standard ($83/mo) as volume grows

Recipe: scrape and extract pricing from a competitor page

With Firecrawl MCP installed and your FIRECRAWL_API_KEY set, paste this in Cursor or Claude Code:

Use the Firecrawl MCP. /scrape https://example-saas.com/pricing
in markdown mode, then /extract with this schema:

  {
    "plans": [{
      "name": "string",
      "monthly_price_usd": "number | null",
      "key_features": "string[]"
    }]
  }

Return the extracted JSON only — no commentary. If the page
requires JavaScript rendering, set actions: [{ type: "wait",
milliseconds: 1500 }] before extraction.

Two tool calls: scrape gives the agent markdown plus the cleaned DOM; extract runs the schema-guided LLM extraction and returns typed JSON. The same shape works against documentation sites, news articles, product catalogs, and almost any public page. The credit cost is 1 credit per page for /scrape plus /extract’s additional LLM cost.

Skip it if...

Your target site needs to be logged in to, or your volume is high enough that 100k credits ($83/mo) per month doesn’t cover it. For interactive flows behind auth, use Playwright MCP instead. For very high volume, self-host the Apache 2.0 core or move to Crawlee.

Crawlee — install + recipe

What it does best

Crawlee is the canonical “build your own scraper” framework. It gives you persistent URL queues, session pools, proxy rotation, browser-like header generation, HTTP/2 support, and unified APIs for Cheerio, JSDOM, Playwright, and Puppeteer — so you pick the right tool per site without rewriting the framework around it. The same team builds Apify, so a Crawlee scraper deploys cleanly to Apify Cloud if you want to graduate from local execution to scheduled hosted jobs without throwing away the code. Note for clarity: Crawlee and Apify are different products from the same team — Crawlee is the library you install and run; Apify is the SaaS that hosts actors.

Pick this if you...

• Want full control over the scraping pipeline and prefer paying for compute over paying per-page API fees
• Crawl sites where no Apify actor exists and you don’t want to write the request-queue plumbing from scratch
• Need to mix headless-browser and plain-HTTP crawling in one project with shared session and proxy state
• Want optionality to deploy to Apify Cloud later without changing your scraper code

Recipe: a minimal PlaywrightCrawler

After npm install crawlee playwright:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  maxConcurrency: 5,
  maxRequestsPerCrawl: 100,
  async requestHandler({ page, request, enqueueLinks, pushData }) {
    const title = await page.title();
    const heading = await page.locator('h1').first().textContent();
    await pushData({ url: request.url, title, heading });
    await enqueueLinks({ strategy: 'same-domain' });
  },
});

await crawler.run(['https://example.com']);

That’s the entire scraper: queue, browser pool, link discovery, results storage. Run it with node index.js; results land in storage/datasets/default as JSON. Swap PlaywrightCrawler for CheerioCrawler for static HTML — same API, ten times the throughput. The Python equivalent in apify/crawlee-python uses identical concepts with Pythonic syntax.

Skip it if...

You want zero infrastructure overhead, or you only need to scrape a handful of pages a month. Firecrawl’s 1,000 free credits cover the latter without you running anything; Apify Store covers the former by renting somebody else’s actor. Crawlee shines once you’re operating real volume and the SaaS bill starts to compete with your engineering time.

ScrapeGraph — install + recipe

What it does best

ScrapeGraph throws out the “write a selector per field” model entirely. You describe what you want in a sentence, you supply an LLM key, the library extracts the page via Playwright, runs the content through the model, and returns structured data. The pitch lives or dies on developer experience — and on the 24.9k-star ScrapeGraphAI/Scrapegraph-ai repo it delivers: prototypes that took two days of selector-writing now take twenty minutes. Model-agnostic is the second pillar: OpenAI, Anthropic, Groq, Gemini, Azure, and Ollama (for fully offline extraction). The Ollama path matters more than vendors typically admit; teams scraping sensitive data use it to keep page content out of any third-party API.

Pick this if you...

• Want to skip writing CSS selectors entirely and describe the schema in natural language
• Prototype scraping pipelines where the field set changes every week
• Scrape sensitive pages and need a fully-local model via Ollama (page content never leaves your network)
• Are okay paying for LLM tokens per page in exchange for near-zero maintenance when sites redesign

Recipe: minimal SmartScraperGraph

After pip install scrapegraphai and playwright install:

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "model": "openai/gpt-4o-mini",
        "api_key": "YOUR_OPENAI_KEY",
    },
    "verbose": False,
    "headless": True,
}

smart_scraper = SmartScraperGraph(
    prompt="Extract every project: title, description, GitHub stars, language.",
    source="https://github.com/trending",
    config=graph_config,
)

result = smart_scraper.run()
print(result)

Three lines of meaningful code: the prompt, the source, the model. The library handles browser navigation, HTML preprocessing, model invocation, and JSON normalisation. Swap openai/gpt-4o-mini for ollama/llama3.1 with a local Ollama instance for fully-offline extraction.

Skip it if...

You’re scraping the same template across millions of pages — running an LLM per page is slower and more expensive than a CSS selector you write once. The next section unpacks where AI extraction wins and where it loses.

AI extraction vs CSS selectors

The single biggest 2024-to-2026 shift in scraping is the rise of LLM-based extraction. Both Firecrawl’s /extract endpoint and ScrapeGraph’s entire library bet that describing the output in English beats writing CSS selectors by hand. The bet is right for some jobs and wrong for others, and the breakdown matters more than the marketing on either side suggests.

AI extraction wins on three jobs. First, prototypes — you don’t know what fields you want yet, and rewriting selectors after each pivot wastes time. Second, sites that redesign frequently: a model tolerates layout changes that would break a brittle selector chain, because it reads semantic structure rather than DOM paths. Third, low-volume one-offs where the marginal cost of an LLM call ($0.001 to $0.01 per page) is cheaper than thirty minutes of selector authoring.

CSS selectors win on three jobs. First, cost at scale — running an LLM on every page is materially more expensive than a parsed-once selector when you’re scraping ten million pages per month from the same template. Second, determinism: a model can hallucinate a field that doesn’t exist or normalise inconsistently across runs, while a missing selector returns null in a known way. Third, latency: an LLM call adds 500-3000ms per page; a CSS selector adds zero.

Firecrawl’s /extract endpoint occupies a useful middle. The schema-guided extraction is LLM-backed but happens server-side; you pay Firecrawl credits rather than running the LLM yourself; the prompt engineering is hidden behind a JSON schema you supply. It’s the right shape when you want the description-vs-selectors experience but don’t want to operate a model. ScrapeGraph is the right shape when you want to control the model entirely — pick your provider, control prompts directly, run locally if your data sensitivity requires it.

A common production pattern: use AI extraction for the long tail (sites you scrape rarely), use CSS selectors for the high-volume targets (your top ten sources), and fall back to AI extraction whenever selectors break. Many teams ship this hybrid in production and treat “selector breakage rate” as a leading indicator that triggers re-authoring or fallback to AI extraction permanently.

The other piece: model choice matters more than the library choice for AI extraction. GPT-4o-mini and Claude 3.5 Haiku are the cheap-and-fast workhorses; Llama 3.1 via Ollama wins for sensitive-data jobs; GPT-4o and Claude 4 Opus appear when extraction quality on weird layouts matters more than per-page cost. The scraping library doesn’t care which you pick — but your bill does.

Cost shape

The five tools price for different workloads, which is worth comparing apples-to-apples-ish. All numbers are from the vendors’ public pricing pages as of May 2026; confirm at the source before designing around them.

Three observations. Apify is the only one that requires you to think in compute units — not pages — which lines up with the marketplace model where actor complexity varies wildly. Spider’s pay-per-GB plus per-minute compute is the most accurate billing shape for very high volume, because that’s where your real costs live; small workloads find it annoying. Firecrawl’s credit tiers are the most predictable for anyone budgeting from a credits-per-page mental model. Crawlee and ScrapeGraph cost zero in licence fees but require you to operate infrastructure (and an LLM provider for ScrapeGraph) — which can be very cheap or quite expensive depending on volume and how you provision proxies.

Confirm at the source: apify.com/pricing, spider.cloud/pricing, firecrawl.dev/pricing. Crawlee and ScrapeGraph have no vendor pricing — both are libraries.

Common pitfalls

Community signal

The scraping space moves fast and the public conversation is correspondingly noisy — vendor marketing claims need to be discounted, but consistent signals across HN, Reddit, GitHub Discussions, and Twitter are useful triangulation. We won’t fabricate quotes; here’s the consensus picture from late 2025 through May 2026.

Firecrawl is the clear default for LLM-facing scraping. The 100k+ star count on the Apache 2.0 repo isn’t signal-noise — it reflects heavy actual usage. Threads on HackerNews and r/LLM consistently surface Firecrawl as the first recommendation when somebody asks “what should my agent use to read web pages,” and the most common gripe is the credit cost at scale rather than output quality. The self-host path is real but requires meaningful operational investment.

Apify has the strongest position in site-specific scraping. The marketplace effect is hard to argue with — when a popular site changes, the actor that scrapes it usually has a fix shipped within hours by an author whose entire business is that one actor. Engineering teams who try to rebuild what an Apify actor does usually conclude rent-don’t-build was the right call after a month of maintenance.

Crawlee remains the framework people reach for when they’ve outgrown a hosted API and want to own the cost curve. The Python port closed the “but we’re a Python shop” objection that previously sent people to Scrapy. The 23k-star repo updates actively; the headless-browser story is uniquely strong compared to alternatives.

Spider is less mainstream but shows up consistently in conversations about large-scale crawls — typically when someone’s AWS bill from running Node-based crawlers reached a threshold that justified looking at a Rust alternative. The 2k-star spider-rs repo is smaller than the others; the hosted product is where most Spider usage actually lives.

ScrapeGraph sits in the fastest-growth bucket of the five. The 24.9k stars on a Python library that’s essentially “Playwright + an LLM” reflect how much demand there was for somebody to package the obvious idea well. The Ollama support is the most-mentioned feature in community threads — sensitive-data teams love that page content never has to leave their network. The honest critique is per-page cost at high volume, which is the inherent ceiling of the LLM-first approach.

Frequently asked questions

Sources

Apify

• apify.com/store — actor marketplace (almost 29,000 actors)
• apify.com/pricing — free, Starter, Scale, Business, Enterprise
• docs.apify.com/api/v2 — actor invocation API

Spider

• spider.cloud — hosted Rust crawling API
• spider.cloud/pricing — pay-per-GB pricing
• github.com/spider-rs/spider — open-source Rust core

Firecrawl

• firecrawl.dev — hosted SaaS
• github.com/firecrawl/firecrawl — Apache 2.0 open-source core (100k+ stars)
• github.com/mendableai/firecrawl-mcp-server — official MCP server
• firecrawl.dev/pricing — credit tiers

Crawlee

• crawlee.dev — documentation
• github.com/apify/crawlee — Apache 2.0, TypeScript (23.2k stars)
• github.com/apify/crawlee-python — Python port

ScrapeGraph

• scrapegraphai.com — project site
• github.com/ScrapeGraphAI/Scrapegraph-ai — MIT, Python (24.9k stars)

Related comparisons

• Firecrawl vs Anycrawl vs Crawlee vs Playwright (2026) — API-vs-library cut with Playwright
• Best web search MCP servers (2026)
• Chrome DevTools MCP vs Playwright MCP (2026)

Internal links

• /servers/apify-actor
• /servers/firecrawl
• /blog/what-is-mcp — protocol primer
• /best-mcp-servers — curated roundup
• /servers — browse all 3,000+