Media Crawler MCP Service

🚀 MediaCrawler MCP Service

The MCP service that enables AI to natively use social media data. Upgrade traditional CLI crawlers to MCP standard tools, allowing direct calls from Claude / ChatGPT. One-time configuration for long-term use.

🚀 Quick Start

Environment Requirements

Python 3.8+ · Redis · Chrome/Chromium · (Optional) Node.js 16+

Steps

1. Clone the repository and install dependencies

git clone <your-repo-url>
cd media-crawler-mcp-service
poetry install
poetry run playwright install chromium

2. Configure the environment

cp .env.example .env
# Modify the port/platform switch/Redis, etc. as needed

3. Start the service

redis-server                 # If not started
poetry run python main.py    # Default port 9090

# Management interface: http://localhost:9090/admin
# Tool debugging: http://localhost:9090/admin/inspector

✨ Features

• One-time configuration for native AI calls (Claude/ChatGPT)
• Externalized login: Persistent across restarts and supports multiple platforms
• Task-level configuration isolation to eliminate global variable competition
• Reuse of Playwright browser context for second-level response
• Pydantic structured output for easy analysis and archiving
• Built-in state caching and risk control-friendly strategies

Completed and In Progress

• Completed: Bilibili search/detail/creator/comments (bili_search/bili_detail/bili_creator/bili_comments)
• In progress: Xiaohongshu / Douyin / Kuaishou / Zhihu / Tieba / Weibo, etc.

📦 Installation

The installation steps are included in the "Quick Start" section above.

💻 Usage Examples

Using in AI Assistants

You can directly let Claude/ChatGPT call MCP tools:

Example conversation:
"Help me search for Bilibili videos related to Python machine learning and analyze their popularity and creators."
→ Automatically call `bili_search` to obtain data → Analyze with indicators → Output insights

Tool Overview

The MCP Tools Inspector on the management page can view all registered tools and perform online debugging.

bili_search (Recommended, Quick Search)

{
  "keywords": "Python 机器学习",
  "page_size": 3,
  "page_num": 1
}

bili_detail (Specify Video Details)

{ "video_ids": ["115285696846946"] }

bili_creator (Creator Analysis)

{ "creator_ids": ["99801185"], "creator_mode": true }

📚 Documentation

Project Introduction

MediaCrawler MCP Service is a data acquisition toolset for "learning and research". It transforms publicly available social media information into standardized tools that can be directly called by AI assistants through MCP (Model Context Protocol). Its core capabilities include "externalized login management", "task-level configuration isolation", "browser context reuse", and "structured data output".

Why is it different?

• From scripts to standards: Transform one-time scripts into reusable MCP tools
• Completely externalized login: Visual interface + QR code/Cookie dual mode with persistent status
• True engineering: Layered decoupling, Pydantic models, status caching, and risk control-friendly
• Text format friendly: Since AI analysis is required, it returns a text-friendly format instead of a large amount of useless nested data

Management Interface and Login

1. Open the management interface http://localhost:9090/admin
2. Enter "Login Management" and select a platform (e.g., Bilibili)
3. Support "QR code login" or "Cookie login", and the status will be persisted.

Architecture and Technology Selection

🤖 AI Assistant (Claude / ChatGPT)
           │  MCP Protocol
           ▼
🎯 MediaCrawler MCP Service
  ├─ Management layer: Login/status/configuration
  ├─ Service layer: Orchestration of each platform (Bili…)
  └─ Tool layer: bili_search/detail/creator/comments
           │
           ▼
🌐 Playwright Browser (Context reuse / Risk control-friendly)
           │
           ▼
💾 Redis status caching · Local/structured storage

Comparison with traditional CLI scripts

• Call method: Scripts → MCP tools (Native AI support)
• Login management: In-script logic → Externalized page + Persistence
• Performance: Cold start every time → Resident browser reuse
• Design: Global variables → Task-level configuration isolation
• Output: Raw results → Pydantic structured

Roadmap

• Adaptation for Xiaohongshu / Douyin / Kuaishou / Zhihu / Weibo / Tieba
• Enhancement of tool debugger and visualization (Filtering, exporting, trends)
• Finer-grained rate and risk control strategies
• More storage targets (SQLite/DB persistence)

Development and Contribution

Reference Paths

• Startup entry: main.py:1
• Application factory: app/api_service.py:1
• Management interface route: app/api/endpoints/admin/admin_page_endpoint.py:1
• Tool list interface: app/api/endpoints/admin/mcp_inspector_endpoint.py:1

Typical Steps for Adding a New Platform

1. Add a login adapter: app/core/login/{platform}/...
2. Implement platform crawlers and orchestration: app/core/crawler/platforms/{platform}/...
3. Wrap as an MCP tool: app/core/mcp_tools/{platform}.py
4. Register the route: app/api/endpoints/mcp/{platform}.py

More deployment and operation options: deploy/README.md:1

Contribution Process

1. Fork and create a feature branch
2. Read the project specifications: Agent.md:1
3. Develop and test locally
4. Submit a PR and explain the changes

FAQ

Failed to start?

poetry install
poetry run playwright install chromium
redis-cli ping   # Expect PONG
APP__DEBUG=true poetry run python main.py

QR code not displayed?

BROWSER__HEADLESS=false poetry run python main.py
poetry run playwright install-deps chromium

Login status is easily lost?

• Confirm that Redis is stable and the network is normal; avoid high-frequency triggering of risk control.

Search is empty or slow?

• Use bili_search first, reduce the pagination size, and increase the request interval.

Compliance and Reasonable Use

This project is positioned as an efficiency tool for "learning and research":

• Comply with the terms of use of each platform and robots.txt
• Reasonably control the frequency to avoid putting pressure on the platform
• Respect content creators and do not use it for commercial crawling
• It is recommended to have a small single request volume and a request interval ≥ 2s

If this project helps your learning, welcome to ⭐ Star to support!