The Crawl4AI RAG MCP Server is an AI agent service integrating web crawler and RAG functions, supporting smart URL detection, recursive crawling, parallel processing, and vector search. It aims to provide powerful knowledge acquisition and retrieval capabilities for AI coding assistants.

The MediaCrawler MCP service upgrades the social media crawler into a standardized tool that can be directly called by AI assistants. It supports multi - platform data acquisition and has features such as externalized login, browser reuse, and structured output.

Deployment Guide for Web Crawler MCP Server

PodCrawlerMCP is an MCP service that discovers podcast content through web crawlers, helping AI assistants find podcast shows and episodes based on topics.

WebSearch-MCP is a service that implements the Model Context Protocol (MCP). It provides web search capabilities by integrating the WebSearch Crawler API and supports multiple AI clients to obtain web information in real-time.

An MCP service that connects the Claude desktop version with the local eGet web crawler, enabling the function of scraping web page content through the local API

A Python-based MCP web crawler project for extracting and saving website content as Markdown files, supporting batch processing and multi-threaded configuration.

A website security scanning tool based on the MCP protocol, integrating dirsearch directory scanning and firecrawl crawler technology, capable of automatically identifying the website technology stack and classifying vulnerability risk levels

An RSS crawler server based on the MCP protocol, used for crawling and managing RSS subscription content and integrating with LLM.

A development documentation server based on the MCP protocol, providing functions such as document crawling, local loading, precise search, and detail retrieval, to solve the document hallucination problem in AI development.

The Firecrawl MCP Server is a web crawler and data extraction service based on the Firecrawl API, providing functions such as web page scraping, content search, site crawling, and structured data extraction.

This project provides a set of tools for crawling website content and generating Markdown documents. It also implements semantic search for documents through the MCP server and supports integration with tools such as Cursor.

MCP Smart Crawler is a web content crawler based on Playwright, specifically used to extract metadata from Xiaohongshu posts and download media resources.

The Crawlab MCP server is a middleware connecting AI applications and the Crawlab crawler platform, enabling natural language interaction through a standardized protocol

Dafty MCP is an independently developed open - source project that interacts with Daft.ie through web crawlers, providing functions for searching and querying details of rental information in Ireland.

This project is a semantic search tool based on Elasticsearch for semantic retrieval of Search Labs blog posts, including crawler configuration, index mapping updates, and MCP server integration functions.

An MCP server that provides Google Search and web content viewing functions, with advanced anti - crawler evasion technology

mcp - server - webcrawl is an advanced web crawler data search and retrieval tool designed specifically for AI clients. It supports multiple crawler formats (such as WARC, wget, etc.), provides full - text search, Boolean logic queries, and resource type/status filtering functions. It can be seamlessly integrated with Claude Desktop, is installed via Python, and is suitable for tasks such as building website knowledge bases or conducting SEO/performance audits.

Crawl4AI is an intelligent web crawler server based on the MCP architecture. It combines the Claude AI model to provide website content analysis and processing functions, supports customizable crawling depth and AI processing tasks, and is suitable for various scenarios such as research, content creation, and data analysis.

A pure crawler - based web search MCP service based on Node.js, supporting Bing web page and news searches without an official API, providing high - performance crawling and health monitoring functions.

An MCP server for spider crawler interaction and automation, providing web page scraping and crawling tools, currently in the early development stage.

The Firecrawl MCP Server is a web crawler service based on the Firecrawl API, providing functions such as web scraping, content search, site crawling, and structured data extraction, and supporting multiple configuration options and output formats.

The Crawl4AI MCP Server is a Python-based web crawler project that integrates Playwright browser automation, the DuckDuckGo search engine, and the MCP protocol SDK for intelligent web data collection.

A financial data crawler tool based on Playwright, supporting multi - website queries and the MCP protocol, providing table data crawling and large language model calling functions.

The Crawlab MCP service is a middleware that connects AI applications and the Crawlab crawler platform, enabling natural language interaction through a standard protocol and supporting functions such as crawler management, task control, and file operations.

This project is a Node.js - based MCP server crawler tool used to scrape and extract MCP server information from the glama.ai website. Users need to install dependencies first, run the crawler script to get server links, and then extract detailed data through the processing script.

A web crawling service based on the MCP protocol, providing web page content extraction functions for AI assistants, supporting bypassing anti - crawler mechanisms and returning formatted text.

An AI-based web crawler project that provides MCP services

Ultra-MCP-Servers is an open-source project that enables users to create and manage their own Model Context Protocol (MCP) servers. It offers a variety of tool and service integrations, such as YouTube transcription, cloud storage, web crawlers, etc., and encourages community contributions.

A web search MCP service based on pure crawler technology, supports Bing web and news searches, does not require an official API, and includes intelligent anti - detection and URL cleaning functions.

This project is a semantic search tool based on Elasticsearch for semantic retrieval of Search Labs blog posts, including crawler configuration, index mapping update, and MCP server integration functions.