MCP Read Website Fast

🚀 @just-every/mcp-read-website-fast

Fast, token-efficient web content extraction for AI agents - converts websites to clean Markdown.

🚀 Quick Start

This MCP package @just-every/mcp-read-website-fast is designed to quickly and efficiently extract web content for AI agents, converting websites into clean Markdown. It addresses the issues of slow speed and high token consumption in existing MCP web crawlers.

✨ Features

• Fast startup: Utilizes the official MCP SDK with lazy loading for optimal performance.
• Content extraction: Employs Mozilla Readability (same as Firefox Reader View) for content extraction.
• HTML to Markdown: Converts HTML to Markdown with Turndown + GFM support.
• Smart caching: Implements smart caching with SHA - 256 hashed URLs.
• Polite crawling: Supports robots.txt and rate limiting for polite crawling.
• Concurrent fetching: Allows configurable depth crawling with concurrent fetching.
• Stream - first design: Features a stream - first design for low memory usage.
• Link preservation: Preserves links for knowledge graphs.
• Optional chunking: Provides optional chunking for downstream processing.

📦 Installation

Claude Code

claude mcp add read-website-fast -s user -- npx -y @just-every/mcp-read-website-fast

VS Code

code --add-mcp '{"name":"read-website-fast","command":"npx","args":["-y","@just-every/mcp-read-website-fast"]}'

Cursor

cursor://anysphere.cursor-deeplink/mcp/install?name=read-website-fast&config=eyJyZWFkLXdlYnNpdGUtZmFzdCI6eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqdXN0LWV2ZXJ5L21jcC1yZWFkLXdlYnNpdGUtZmFzdCJdfX0=

JetBrains IDEs

Settings → Tools → AI Assistant → Model Context Protocol (MCP) → Add Choose “As JSON” and paste:

{"command":"npx","args":["-y","@just-every/mcp-read-website-fast"]}

Or, in the chat window, type /add and fill in the same JSON—both paths land the server in a single step.

Raw JSON (works in any MCP client)

{
  "mcpServers": {
    "read-website-fast": {
      "command": "npx",
      "args": ["-y", "@just-every/mcp-read-website-fast"]
    }
  }
}

Drop this into your client’s mcp.json (e.g. .vscode/mcp.json, ~/.cursor/mcp.json, or .mcp.json for Claude).

💻 Usage Examples

Available Tools

• read_website_fast - Fetches a webpage and converts it to clean markdown
  • Parameters:
    • url (required): The HTTP/HTTPS URL to fetch
    • depth (optional): Crawl depth (0 = single page)
    • respectRobots (optional): Whether to respect robots.txt

Available Resources

• read-website-fast://status - Get cache statistics
• read-website-fast://clear-cache - Clear the cache directory

Development Usage

Install

npm install
npm run build

Single page fetch

npm run dev fetch https://example.com/article

Crawl with depth

npm run dev fetch https://example.com --depth 2 --concurrency 5

Output formats

# Markdown only (default)
npm run dev fetch https://example.com

# JSON output with metadata
npm run dev fetch https://example.com --output json

# Both URL and markdown
npm run dev fetch https://example.com --output both

CLI Options

• -d, --depth <number> - Crawl depth (0 = single page, default: 0)
• -c, --concurrency <number> - Max concurrent requests (default: 3)
• --no-robots - Ignore robots.txt
• --all-origins - Allow cross-origin crawling
• -u, --user-agent <string> - Custom user agent
• --cache-dir <path> - Cache directory (default: .cache)
• -t, --timeout <ms> - Request timeout in milliseconds (default: 30000)
• -o, --output <format> - Output format: json, markdown, or both (default: markdown)

Clear cache

npm run dev clear-cache

🔧 Technical Details

Architecture

mcp/
├── src/
│   ├── crawler/        # URL fetching, queue management, robots.txt
│   ├── parser/         # DOM parsing, Readability, Turndown conversion
│   ├── cache/          # Disk-based caching with SHA-256 keys
│   ├── utils/          # Logger, chunker utilities
│   └── index.ts        # CLI entry point

Development

# Run in development mode
npm run dev fetch https://example.com

# Build for production
npm run build

# Run tests
npm test

# Type checking
npm run typecheck

# Linting
npm run lint

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

🛠️ Troubleshooting

Cache Issues

npm run dev clear-cache

Timeout Errors

• Increase timeout with -t flag
• Check network connectivity
• Verify URL is accessible

Content Not Extracted

• Some sites block automated access
• Try custom user agent with -u flag
• Check if site requires JavaScript (not supported)

📄 License

MIT