MCP Server Spider

🚀 mcp-server-spider: A Model Context Protocol Server for Spider Crawlers

This is a Model Context Protocol (MCP) server designed for the interaction and automation of Spider crawlers. It provides tools to crawl and extract web page content.

🚀 Quick Start

The mcp-server-spider is a Model Context Protocol server tailored for Spider crawlers. It's crafted to facilitate seamless interaction and automation, offering essential tools for web content crawling and extraction.

⚠️ Important Note

Please note that mcp-server-spider is currently in the early stages of development. There may be bugs, and more features will be added in the future.

✨ Features

Tools

1. crawl

   • Crawl the specified URL and return a list of all found URLs.
   • Input parameters:
     • url: The URL address to crawl.
     • headers: Additional headers to attach to the crawl request.
     • user_agent: The user agent to set for the crawl request.
     • depth: The maximum depth of link traversal.
     • blacklist: A series of regular expression rules for blacklist filtering.
     • whitelist: A series of regular expression rules for whitelist filtering.
     • respect_robots_txt: Whether to abide by the rules in the robots.txt file.
     • accept_invalid_certs: Whether to accept invalid certificates.
   • Return result: A list of all found URLs.

2. scrape

   • Crawl the specified URL and return a list of JSON objects containing information about each crawled page (such as URL, links, and content).
   • Input parameters: The same as the crawl command.
   • Return result: A list of JSON objects containing URLs, links, and content (returned as a string).

📦 Installation

Using uv (Recommended)

When using uv, no special installation is required. We can directly use uvx to run mcp-server-spider.

Using pip

If needed, you can also install mcp-server-spider via pip:

pip install mcp-server-spider

After installation, you can start the server with the following command:

python -m mcp_server_spider