Free vs Paid Web Scraping APIs for AI Agents free scrapers work until the web fights back.

Open-source servers cover an honest majority of everyday extraction. CRW Web Scraper ships scrape, crawl, and map tools with no upstream dependency; webclaw converts any URL to clean markdown with scrape, crawl, extract, and summarize operations; ShadowCrawl takes the adversarial route in Rust, with anti-bot tactics, CDP fallback, and human-in-the-loop handoff for the hardest pages.

Run these on your own infrastructure and documentation sites, blogs, and most public pages cost you nothing but compute. Tools like Librecrawl even cover specialized jobs — self-hosted technical SEO audits with 50+ checks — entirely free.

Three things break self-hosted scraping at scale: IP reputation (your server's address gets flagged after enough requests), JavaScript rendering (headless browsers per page are expensive to operate in fleets), and active anti-bot systems that adapt faster than a side project can. Rotating residential proxies and evasion engineering are recurring costs, not one-time setup.

That's the gap commercial scraping APIs fill. Servers like zenrows-mcp and scrapegraph-mcp front exactly such services — the MCP server is free, but the upstream API meters usage; check each provider's docs for current pricing. The page-fetch infrastructure is the paid product.

Scraping is naturally denominated in pages, which makes it an ideal x402 workload: each fetch is a discrete unit with real marginal cost (proxy bandwidth, rendering compute). An agent hits the endpoint, receives an HTTP 402 quoting the price, pays in USDC, and gets the extracted content — settled on Base in about two seconds, from $0.01 per page, with a signed receipt.

That removes the worst part of scraping APIs for agents: account signup and credit pre-purchase. An agent that needs 40 defended pages for one research task can pay for exactly 40 pages and never touch a pricing page.

Self-host free servers for cooperative targets and high steady volume — your compute is cheaper than anyone's API at scale, and tools like webclaw or PageMap (which pitches token-efficient structured extraction for agents) are mature. Pay per page when targets are defended, when you need someone else's proxy fleet, or when volume is too bursty to justify infrastructure.

Loomal's Web Scraping category lists all 31 live servers with descriptions and x402 pricing where maintainers have configured it — the quickest way to see which extraction endpoints an agent can pay autonomously.