Fetch MCP

Fetcher MCP is a web page content scraping service based on the Playwright headless browser. It supports JavaScript rendering, intelligent content extraction, and parallel processing, and provides HTML/Markdown output formats. It is suitable for obtaining dynamic web page content.

Developer tools Research and data #Web scraping #Intelligent extraction #Parallel processing #Dynamic rendering .TypeScript

rating : 0 points

downloads : 633

update time : 2025-04-28

What is Fetcher MCP?

Fetcher MCP is a powerful tool for scraping web page content through the Playwright browser. It supports dynamic web page parsing, intelligent content extraction, multiple output formats, and efficient batch processing.

How to use Fetcher MCP?

Simply run the command to start scraping web page content, and it supports parameter configuration to meet different needs.

Applicable scenarios

Suitable for scenarios that require scraping dynamic web page content, extracting the main content, and batch processing web page data.

Main features

Scrape a single web pageScrape web page content by specifying the URL, supporting HTML and Markdown output.

Batch scrape multiple web pagesSupports multi-threaded concurrent scraping of multiple URLs to improve efficiency.

Intelligent content extractionAutomatically remove unnecessary content such as advertisements and navigation bars, and extract the main content.

Flexible output formatsSupports output in HTML and Markdown formats, facilitating integration into other systems.

Resource optimizationAutomatically block unnecessary resources such as images, style sheets, and fonts to save bandwidth.

Robust error handlingBuilt-in comprehensive logging and error handling mechanisms to ensure stable operation.

Advantages and limitations

Advantages

Supports JavaScript execution and can scrape dynamic web page content.

Intelligent content extraction algorithm that automatically filters out irrelevant content.

Flexible output formats (HTML/Markdown) for easy integration.

Parallel processing capability to significantly improve the efficiency of batch tasks.

Resource optimization to reduce bandwidth consumption.

Powerful error handling mechanism to ensure high reliability.

Limitations

May not be able to fully parse some highly encrypted websites.

Requires a browser environment (such as Chromium) to run.

Manual debugging may be required for particularly complex pages.

How to use

Install dependencies

First, install the required Playwright browser.

Start the MCP server

Use npx to start the Fetcher MCP server.

Scrape a single web page

Scrape web page content by specifying the URL.

Batch scrape web pages

Perform batch scraping through a configuration file or command line.

Usage examples

Scrape a single web pageScrape the specified web page and extract the main content.

Batch scrape multiple web pagesScrape multiple web pages simultaneously and merge the results.

Frequently Asked Questions

How to enable debug mode?

How to handle anti - scraping mechanisms?

How to get the complete web page content?

Related resources

Fetcher MCP GitHub repository

The official repository containing code and documentation.

Playwright official documentation

The official documentation for the Playwright browser tool.

g - search - mcp GitHub repository

Another powerful MCP server for Google search.

🚀 Fetcher MCP

A MCP server for web content scraping, leveraging Playwright's headless browser.

🚀 Quick Start

Run the Server

npm start

Install the Playwright Browser

npm run install-browser

✨ Features

JavaScript Support: Unlike traditional web crawlers, Fetcher MCP uses Playwright to execute JavaScript, enabling it to handle dynamic web content and modern web applications.
Intelligent Content Extraction: An in - built readability algorithm automatically extracts the main content from web pages, eliminating ads, navigation, and other non - essential elements.
Flexible Output Formats: Supports both HTML and Markdown output formats, facilitating integration with various downstream applications.
Parallel Processing: The fetch_urls tool enables concurrent scraping of multiple web pages, enhancing efficiency.
Rich Functionality: Includes a debug mode, custom browser installation, and detailed documentation.

📦 Installation

Configure MCP

Edit the config.json file:

{
  "port": 3000,
  "debug": false,
  "proxy": null,
  "timeout": 5000,
  "maxConcurrency": 10
}

💻 Usage Examples

Basic Usage of `fetch_url` Command

Scrape a single web page:

curl http://localhost:3000/fetch?url=example.com

Supported parameters:

url: Required, the address of the target web page.
depth: Crawling depth, default is 1.
delay: Request interval time, default is 0.
user - agent: Custom user - agent string.

Advanced Usage of `fetch_urls` Command

Scrape multiple web pages in batches:

curl http://localhost:3000/fetch/urls -X POST -H "Content-Type: application/json" -d '{"urls": ["example.com", "test.com"]}'

📚 Documentation

Installation of Dependencies

npm install

Project Build

npm run build

🔧 Technical Details

Debugging Tools

Use the MCP inspector tool for debugging:

npm run inspector

Related Projects

[g - search - mcp](https://github.com/jae - jae/g - search - mcp): A powerful MCP server that supports parallel searches using multiple keywords simultaneously. Ideal for batch data collection tasks.