Playwright Scraper

🚀 mcp-playwright-scraper

An MCP (Model Context Protocol) server for scraping web content and converting it into Markdown format.

This MCP server offers a simple tool for scraping content from web pages and transforming it into Markdown. It leverages the following technologies:

• Playwright: Enables headless browser automation, capable of handling modern web pages, including JavaScript-intensive websites.
• BeautifulSoup: Facilitates HTML parsing and cleaning.
• Pypandoc: Ensures high-quality conversion from HTML to Markdown.

✨ Features

The server implements the following tool:

• scrape_to_markdown: Scrapes content from a URL and converts it to Markdown.
  • Required parameter: url (string) - The URL to scrape.
  • Optional parameter: verify_ssl (boolean) - Whether to verify the SSL certificate (default: true).

📦 Installation

Using uv (Recommended)

When using uv, no specific installation is required. We'll directly use uvx to run mcp-playwright-scraper.

Using pip

Alternatively, you can install mcp-playwright-scraper via pip:

pip install mcp-playwright-scraper

After installation, you can run it as a script:

python -m mcp_playwright_scraper

Prerequisites

Before installing this server, ensure you install the Playwright browser dependencies:

playwright install --with-deps chromium

📚 Documentation

Configuration

In Claude Desktop

To add the MCP server to Claude Desktop, follow these steps:

1. Open the settings menu in Claude Desktop.
2. Select the "AI" tab.
3. Click the "Add Custom Model" button.
4. Fill in the following information:
   • Name: mcp-playwright-scraper
   • Type: MCP Server
   • Executable Path: your/custom/path/to/mcp-playwright-scraper/executable

In Claude Code

When using Claude Code, add the following to mcp-servers.json:

{
  "mcp-playwright-scraper": {
    "type": "MCP Server",
    "executablePath": "/path/to/mcp_playwright_scraper_executable"
  }
}

In Zed

To use it in Zed, add the following to mcp-servers.json:

{
  "mcp-playwright-scraper": {
    "type": "MCP Server",
    "executablePath": "/path/to/mcp_playwright_scraper_executable"
  }
}

In Cursor

To use it in Cursor, add the following to mcp-servers.json:

{
  "mcp-playwright-scraper": {
    "type": "MCP Server",
    "executablePath": "/path/to/mcp_playwright_scraper_executable"
  }
}

Usage

Adding the MCP Server in Claude Desktop

1. Open the settings menu.
2. Navigate to the "AI" tab.
3. Click the "+ Add Custom Model" button.
4. Fill in the following information:
   • Name: mcp-playwright-scraper
   • Type: MCP Server
   • Executable Path: /path/to/mcp_playwright_scraper_executable

Adding via Command Line

You can also add the server to mcp-servers.json using the following command:

uv add mcp-playwright-scraper --type MCP\ Server --config executablePath="/path/to/executable"

Debugging

To debug the MCP server, start the Inspector with the following command:

uv inspect run mcp-playwright-scraper

After starting, you'll see a URL in the console. Copy this URL into your browser to begin debugging.

Development

Building and Publishing

Prepare a release version:

1. Synchronize dependencies and update the lock file:

   uv sync
   

2. Build the release:

   uv build
   

3. Publish to PyPI:

   uv publish
   

Note: You need to set your PyPI credentials, which can be done via environment variables or command-line parameters:

• Token: --token or UV_PUBLISH_TOKEN
• Username/password: --username/UV_PUBLISH_USERNAME and --password/UV_PUBLISH_PASSWORD

📄 License

This MCP server is licensed under the Apache License Version 2.0. You are free to use, modify, and distribute the software, subject to the terms and conditions of the Apache License 2.0. For more details, refer to the LICENSE file or visit http://www.apache.org/licenses/LICENSE-2.0.