robotstxt-ai

About robotstxt-ai

Use robotstxt-ai with your agent

claude mcp add robotstxt-ai -- npx -y robotstxt-ai-mcp

{
  "mcpServers": {
    "robotstxt-ai": {
      "command": "npx",
      "args": [
        "-y",
        "robotstxt-ai-mcp"
      ]
    }
  }
}

Frequently asked questions

What is robotstxt-ai?

robotstxt-ai is an MCP (Model Context Protocol) server by sharozdawa in the Web Scraping category. Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.

How do I connect robotstxt-ai to Claude, Cursor, or another MCP client?

Install robotstxt-ai from its npm package (robotstxt-ai-mcp) and register it under "mcpServers" in your client's MCP configuration — for example claude_desktop_config.json or Cursor's mcp.json — then restart the client.

Is robotstxt-ai open source?

Yes — the source code is public at github.com/sharozdawa/robotstxt-ai.

Can AI agents pay to use robotstxt-ai?

Not yet through Loomal — robotstxt-ai is listed as a free directory entry. If its maintainer verifies ownership, they can set per-call USDC pricing that agents pay over x402, with settlement on Base.