TheCrawler

About TheCrawler

Use TheCrawler with your agent

claude mcp add thecrawler -- npx -y thecrawler

{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler"
      ]
    }
  }
}

Frequently asked questions

What is TheCrawler?

TheCrawler is an MCP (Model Context Protocol) server by manchittlab in the Web Scraping category. Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

How do I connect TheCrawler to Claude, Cursor, or another MCP client?

Install TheCrawler from its npm package (thecrawler) and register it under "mcpServers" in your client's MCP configuration — for example claude_desktop_config.json or Cursor's mcp.json — then restart the client.

Is TheCrawler open source?

Yes — the source code is public at github.com/manchittlab/thecrawler.

Can AI agents pay to use TheCrawler?

Not yet through Loomal — TheCrawler is listed as a free directory entry. If its maintainer verifies ownership, they can set per-call USDC pricing that agents pay over x402, with settlement on Base.