Seo Crawler MCP

🚀 SEO Crawler MCP - Website Crawler & SEO Analyzer for LLMs

Crawl and analyze your website to identify errors and issues that may impact your site's SEO.

I aimed to leverage my experience with the MCP protocol SDK to explore the extent to which we can enhance an AI assistant's capabilities. I decided to build a crawler to assess my site's "technical SEO" health and discovered Crawlee, which seemed like the ideal library for the crawl component of my MCP.

Interestingly, MCP typically implies some form of server connection. However, this is not the case with SEO Crawler MCP. The MCP protocol is likely more powerful than I initially realized. This is a self - contained application wrapped in the MCP SDK that handles everything locally:

• Smart request scheduling and queue management
• Automatic retry logic and error handling
• Respectful crawling with configurable delays
• Memory - efficient streaming for large sites
• Better - SQLite3 embedded database storing every crawled page's HTML, metadata, headers, link relationships, and site structure
• Custom SQL analysis engine with 25+ specialised queries detecting content issues, technical SEO problems, security vulnerabilities, and optimisation opportunities

Claude (or your preferred AI assistant) can orchestrate this entire stack through simple function calls. The crawl runs asynchronously, stores everything in SQLite, and then Claude can query that data using natural language, such as "analyse this crawl for seo opportunities" or "report on internal broken links". The MCP server then translates these requests into sophisticated SQL analysis.

You can also run crawls directly from the terminal, which is ideal for large sites or background processing. The CLI mode allows you to run a crawl, obtain the output directory, and then pass it to Claude for AI - powered analysis via the MCP tools.

Credits

The core crawling architecture is inspired by the logic and patterns from the [LibreCrawl](https://github.com/libre - crawl/core) project. We've adapted their proven crawling methodology for use within the MCP protocol while adding comprehensive SEO analysis capabilities.

🚀 Quick Start

For Beginners

If you're new to MCP servers, it's recommended to read the following first:

• [How to Add an MCP Server to Claude Desktop](https://houtini.com/how - to - add - an - mcp - server - to - claude - desktop/)
• [Claude Desktop Beginner's Guide](https://houtini.com/claude - desktop - beginners - guide/)

It's also advisable to install [Desktop Commander](https://houtini.com/desktop - commander/) first, as it's useful for working with the crawl output files. See the [Desktop Commander setup guide](https://github.com/wonderwhy - er/DesktopCommanderMCP) for details.

Quick Install (NPX)

Add the following to your Claude Desktop config file:

Windows: C:\Users\[YourName]\AppData\Roaming\Claude\claude_desktop_config.json
Mac: ~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "seo - crawler - mcp": {
      "command": "npx",
      "args": ["-y", "@houtini/seo - crawler - mcp"],
      "env": {
        "OUTPUT_DIR": "C:\\seo - audits"
      }
    }
  }
}

Restart Claude Desktop. Four tools will be available:

• seo - crawler - mcp:run_seo_audit
• seo - crawler - mcp:analyze_seo
• seo - crawler - mcp:query_seo_data
• seo - crawler - mcp:list_seo_queries

Development Install

cd C:\MCP\seo - crawler - mcp
npm install
npm run build

Then use the local path in your config:

{
  "mcpServers": {
    "seo - crawler - mcp": {
      "command": "node",
      "args": ["C:\\MCP\\seo - crawler - mcp\\build\\index.js"],
      "env": {
        "OUTPUT_DIR": "C:\\seo - audits",
        "DEBUG": "false"
      }
    }
  }
}

Environment Variables:

• OUTPUT_DIR: Directory where crawl results are saved (required)
• DEBUG: Set to "true" to enable verbose debug logging (optional, default: "false")

CLI Usage for Local Development: When running the CLI from a local build (not installed via npm), use node directly:

# Run crawl
node C:\MCP\seo - crawler - mcp\build\cli.js crawl https://example.com --max - pages = 20

# Analyze results
node C:\MCP\seo - crawler - mcp\build\cli.js analyze C:\seo - audits\example.com_2026 - 02 - 02_abc123

# List queries
node C:\MCP\seo - crawler - mcp\build\cli.js queries --category = critical

📦 Installation

The installation steps are detailed in the "Quick Start" section, including beginner - friendly guides, quick install via NPX, and development install.

💻 Usage Examples

Complete SEO Audit

The typical workflow is as follows:

1. Crawl the website

   Use seo - crawler - mcp to crawl https://example.com with maxPages = 2000
   

2. Run the analysis

   Analyse the crawl at C:/seo - audits/example.com_2026 - 02 - 01_abc123
   

3. Investigate specific issues

   Show me the broken internal links from that crawl
   

Claude will handle the rest, calling the appropriate tools, parsing the results, and presenting them in a readable format.

Security Audit

If you're specifically concerned about security headers:

1. List available security queries

   What security checks can you run on an SEO crawl?
   

2. Run security - focused analysis

   Check the security issues in crawl C:/seo - audits/example.com_2026 - 02 - 01_abc123
   

3. Deep dive on specific problems

   Show me all pages with unsafe external links
   

📚 Documentation

What Gets Detected

The analysis engine includes 25 comprehensive SEO checks across five categories:

Critical Issues (4 checks)

• Missing title tags: Pages without titles don't rank.
• Broken internal links: 404/5xx responses that hurt crawlability.
• Server errors: 5xx responses indicating site problems.
• 404 errors: Broken pages that need fixing or redirecting.

Content Quality (7 checks)

• Duplicate titles across different pages.
• Duplicate meta descriptions.
• Duplicate H1 tags.
• Missing meta descriptions.
• Missing H1 tags.
• Multiple H1 tags on single pages.
• Thin content: Pages under 300 words.

Technical SEO (5 checks)

• Redirect chains and loops.
• Orphan pages with no internal links.
• Canonical URL mismatches.
• Non - HTTPS pages still in use.
• Heading hierarchy problems (H3 before H2, etc).

Security (6 checks)

• Missing Content - Security - Policy headers.
• Missing HSTS (Strict - Transport - Security).
• Missing X - Frame - Options (clickjacking protection).
• Missing Referrer - Policy.
• Unsafe external links (target="_blank" without rel="noopener").
• Protocol - relative links (//example.com).

Optimisation (6 checks)

• Title tags too long or too short.
• Meta descriptions length issues.
• Title matches H1 (opportunity for differentiation).
• Pages with no outbound links.
• Pages with excessive external links.
• Pages missing images.

Data Storage

The crawler stores everything in SQLite databases organized by domain and date:

C:/seo - audits/example.com_2026 - 02 - 01_abc123/
├── crawl - data.db          # SQLite database
│   ├── pages              # Every page crawled
│   ├── links              # All link relationships
│   ├── errors             # Crawl errors
│   └── crawl_metadata     # Statistics
├── config.json            # Crawl settings
└── crawl - export.csv       # Optional CSV export

Performance

Typical crawl performance metrics:

Crawl Speed:

• Medium site (1,500 - 2,000 pages): ~15 minutes
• 300,000+ link relationships tracked
• Database size: ~15MB for 2,000 pages

Query Performance:

• Simple queries: under 10ms
• Complex queries: under 100ms
• Join queries: under 200ms
• Full analysis: under 600ms

The SQLite approach works well here. Everything stays local, there are no API rate limits to worry about, and the query performance is more than adequate for SEO analysis.

Limitations

There are 4 additional checks planned for v3.0:

• Core Web Vitals: Requires Playwright for real browser metrics.
• Robots.txt validation: Needs a parser library.
• Readability scoring: Requires a text analysis library.
• Mobile rendering issues: Needs device emulation.

The current 25 checks cover the most critical aspects of technical SEO that directly impact search engine crawling, indexing, and ranking.

🔧 Technical Details

Built with:

• TypeScript 5.3
• Crawlee 3.7 (HttpCrawler)
• better - sqlite3 12.6
• Cheerio 1.0 (HTML parsing)
• MCP SDK 1.0

The code uses ES modules throughout, with proper Zod validation on inputs and comprehensive error handling. The architecture is clean, with separate modules for crawling, analysis, formatting, and tool definitions.

Deployment:

• Local MCP server via Node.js
• No external dependencies
• Configurable output directory
• Concurrent crawling (5 workers)

📄 License

This project is licensed under the Apache License 2.0.

Copyright 2026 Richard Baxter

This product includes software developed by Apify and the Crawlee project. See the NOTICE file for details.

Support

GitHub: https://github.com/houtini - ai/seo - crawler - mcp
Issues: https://github.com/houtini - ai/seo - crawler - mcp/issues
Author: Richard Baxter hello@houtini.com

Tags: seo, crawler, audit, technical - seo, mcp, crawlee, sqlite, web - scraping, site - analysis