The Apify MCP Server is a tool based on the Model Context Protocol (MCP) that allows AI assistants to extract data from websites such as social media, search engines, and e-commerce through thousands of ready-to-use crawlers, scrapers, and automation tools (Apify Actors). It supports OAuth and Skyfire proxy payment and can be integrated into MCP clients such as Claude and VS Code through HTTPS endpoints or local stdio.

The Crawl4AI RAG MCP Server is an AI agent service integrating web crawler and RAG functions, supporting smart URL detection, recursive crawling, parallel processing, and vector search. It aims to provide powerful knowledge acquisition and retrieval capabilities for AI coding assistants.

The MediaCrawler MCP service upgrades the social media crawler into a standardized tool that can be directly called by AI assistants. It supports multi - platform data acquisition and has features such as externalized login, browser reuse, and structured output.

WebSearch-MCP is a service that implements the Model Context Protocol (MCP). It provides web search capabilities by integrating the WebSearch Crawler API and supports multiple AI clients to obtain web information in real-time.

PodCrawlerMCP is an MCP service that discovers podcast content through web crawlers, helping AI assistants find podcast shows and episodes based on topics.

Deployment Guide for Web Crawler MCP Server

SERP MCP Server is a Google search result crawler server based on the Model Context Protocol. It supports fingerprint rotation, location encoding, and streamlined mode, and can automatically extract data such as organic results and related searches.

A website security scanning tool based on the MCP protocol, integrating dirsearch directory scanning and firecrawl crawler technology, capable of automatically identifying the website technology stack and classifying vulnerability risk levels

A Python-based MCP web crawler project for extracting and saving website content as Markdown files, supporting batch processing and multi-threaded configuration.

An MCP service that connects the Claude desktop version with the local eGet web crawler, enabling the function of scraping web page content through the local API

An RSS crawler server based on the MCP protocol, used for crawling and managing RSS subscription content and integrating with LLM.

A development documentation server based on the MCP protocol, providing functions such as document crawling, local loading, precise search, and detail retrieval, to solve the document hallucination problem in AI development.

JMComic AI is an AI - enhanced tool for the JMComic comic crawler. It exposes functions such as searching and downloading to local AI clients through the MCP protocol, allowing users to operate comic downloads in natural language and injecting usage skills knowledge to improve the AI's decision - making ability.

The Firecrawl MCP Server is a web crawler and data extraction service based on the Firecrawl API, providing functions such as web page scraping, content search, site crawling, and structured data extraction.

This project provides a set of tools for crawling website content and generating Markdown documents. It also implements semantic search for documents through the MCP server and supports integration with tools such as Cursor.

The Crawlab MCP server is a middleware connecting AI applications and the Crawlab crawler platform, enabling natural language interaction through a standardized protocol

MCP Smart Crawler is a web content crawler based on Playwright, specifically used to extract metadata from Xiaohongshu posts and download media resources.

Dafty MCP is an independently developed open - source project that interacts with Daft.ie through web crawlers, providing functions for searching and querying details of rental information in Ireland.

A compliance risk assessment tool for website crawlers based on the MCP protocol, which provides risk detection from three dimensions: legal, social ethical, and technical, helping developers evaluate the crawler friendliness and potential risks of target websites.

This project is a semantic search tool based on Elasticsearch for semantic retrieval of Search Labs blog posts, including crawler configuration, index mapping updates, and MCP server integration functions.

SEO crawler MCP server for website crawling and SEO analysis. It stores and analyzes website data through a local SQLite database, providing comprehensive technical SEO checks and security audit functions.

An MCP server that provides Google Search and web content viewing functions, with advanced anti - crawler evasion technology

Crawl4AI is an intelligent web crawler server based on the MCP architecture. It combines the Claude AI model to provide website content analysis and processing functions, supports customizable crawling depth and AI processing tasks, and is suitable for various scenarios such as research, content creation, and data analysis.

A LinkedIn data crawler tool that supports various data extractions such as personal profiles, company information, and post searches. It provides three usage methods: Python library, command line, and MCP server, and has a built-in anti-detection mechanism.

mcp - server - webcrawl is an advanced web crawler data search and retrieval tool designed specifically for AI clients. It supports multiple crawler formats (such as WARC, wget, etc.), provides full - text search, Boolean logic queries, and resource type/status filtering functions. It can be seamlessly integrated with Claude Desktop, is installed via Python, and is suitable for tasks such as building website knowledge bases or conducting SEO/performance audits.

A pure crawler - based web search MCP service based on Node.js, supporting Bing web page and news searches without an official API, providing high - performance crawling and health monitoring functions.

An MCP server for spider crawler interaction and automation, providing web page scraping and crawling tools, currently in the early development stage.

The Firecrawl MCP Server is a web crawler service based on the Firecrawl API, providing functions such as web scraping, content search, site crawling, and structured data extraction, and supporting multiple configuration options and output formats.

The Crawl4AI MCP Server is a Python-based web crawler project that integrates Playwright browser automation, the DuckDuckGo search engine, and the MCP protocol SDK for intelligent web data collection.

A financial data crawler tool based on Playwright, supporting multi - website queries and the MCP protocol, providing table data crawling and large language model calling functions.

The Crawlab MCP service is a middleware that connects AI applications and the Crawlab crawler platform, enabling natural language interaction through a standard protocol and supporting functions such as crawler management, task control, and file operations.

A web crawling service based on the MCP protocol, providing web page content extraction functions for AI assistants, supporting bypassing anti - crawler mechanisms and returning formatted text.

This project is a Node.js - based MCP server crawler tool used to scrape and extract MCP server information from the glama.ai website. Users need to install dependencies first, run the crawler script to get server links, and then extract detailed data through the processing script.

An AI-based web crawler project that provides MCP services

Ultra-MCP-Servers is an open-source project that enables users to create and manage their own Model Context Protocol (MCP) servers. It offers a variety of tool and service integrations, such as YouTube transcription, cloud storage, web crawlers, etc., and encourages community contributions.

A web search MCP service based on pure crawler technology, supports Bing web and news searches, does not require an official API, and includes intelligent anti - detection and URL cleaning functions.

This project is a semantic search tool based on Elasticsearch for semantic retrieval of Search Labs blog posts, including crawler configuration, index mapping update, and MCP server integration functions.