Mcpserver

🚀 Web Scraper MCP Server

A TypeScript web scraping server based on the Model Context Protocol (MCP), with built - in various common scraping and command tools.

✨ Features

• 🚀 Multiple Export Formats: Supports exporting scraped content in Markdown, Text, HTML, and JSON formats.
• 🎭 Puppeteer Headless Browser: Solves the rendering problem of SPA single - page applications that are not statically rendered.
• 📋 Rule Set Support: Allows setting rule sets to scrape specific parts of web pages.
• 🔧 Custom Domain Headers: Supports setting custom domain headers to bypass website login restrictions.
• 📦 Batch Scraping: Supports scraping multiple URLs simultaneously.
• 🎯 Intelligent Content Extraction: Automatically identifies and extracts the main content of web pages.

📦 Installation

# Clone the project
git clone <repository-url>
cd web-scraper-mcp

# Install dependencies
npm install

# Build the project
npm run build

💻 Usage Examples

Basic Usage

Start the Server

npm start

Available Tools

1. scrape_url - Scrape a Single Web Page

{
  "name": "scrape_url",
  "arguments": {
    "url": "https://example.com",
    "format": "markdown",
    "usePuppeteer": false,
    "ruleSet": "blog",
    "customHeaders": {
      "Authorization": "Bearer token"
    }
  }
}

Parameter Explanation:

• url (Required): The URL of the web page to be scraped.
• format (Optional): The export format, supporting markdown, text, html, json, default is markdown.
• usePuppeteer (Optional): Whether to use the Puppeteer headless browser, default is false.
• ruleSet (Optional): The name of the rule set for extracting specific content.
• customHeaders (Optional): Custom request headers.

2. create_rule_set - Create a Content Extraction Rule Set

{
  "name": "create_rule_set",
  "arguments": {
    "name": "my_blog_rule",
    "rules": {
      "title": "h1, .post-title",
      "content": ".post-content, article",
      "links": ".post-content a",
      "images": ".post-content img",
      "exclude": ["nav", "footer", ".sidebar"]
    }
  }
}

Parameter Explanation:

• name (Required): The name of the rule set.
• rules (Required): The configuration of extraction rules.
  • title: The selector for the title.
  • content: The selector for the content.
  • links: The selector for the links.
  • images: The selector for the images.
  • exclude: A list of selectors to be excluded.

3. set_domain_headers - Set Domain Request Headers

{
  "name": "set_domain_headers",
  "arguments": {
    "domain": "example.com",
    "headers": {
      "Authorization": "Bearer your-token",
      "X-Custom-Header": "custom-value"
    }
  }
}

4. batch_scrape - Batch Scrape

{
  "name": "batch_scrape",
  "arguments": {
    "urls": [
      "https://example1.com",
      "https://example2.com",
      "https://example3.com"
    ],
    "format": "markdown",
    "usePuppeteer": false,
    "ruleSet": "blog"
  }
}

Advanced Usage

Pre - defined Rule Sets

The system comes with the following pre - defined rule sets:

• blog: Suitable for most blog article pages.
• news: Suitable for news website articles.
• product: Suitable for e - commerce product pages.
• documentation: Suitable for technical documentation pages.
• forum: Suitable for forum post pages.

Export Formats

• Markdown: Converts web page content to Markdown format, including titles, content, links, and images.
• Text: Plain text format, removing all HTML tags.
• HTML: Cleaned HTML format, removing scripts and styles.
• JSON: Complete structured data, including all extracted information.

🔧 Technical Details

Project Structure

src/
├── index.ts              # Main server file
├── scraper/
│   └── webScraper.ts     # Web scraper
├── export/
│   └── exportManager.ts  # Export manager
├── rules/
│   └── ruleEngine.ts     # Rule engine
├── headers/
│   └── headerManager.ts  # Request header manager
└── utils/
    └── logger.ts         # Logging tool

Development Mode

# Watch for file changes and automatically rebuild
npm run dev

Testing

npm test

Configuration

Environment Variables

• LOG_LEVEL: Log level (debug, info, warn, error).
• PUPPETEER_TIMEOUT: Puppeteer timeout in milliseconds.
• REQUEST_TIMEOUT: HTTP request timeout in milliseconds.

Custom Configuration

You can customize the following by modifying the source code:

1. Default Request Headers: Edit src/headers/headerManager.ts.
2. Pre - defined Rule Sets: Edit src/rules/ruleEngine.ts.
3. Export Formats: Edit src/export/exportManager.ts.

Frequently Asked Questions

Q: Why does scraping some websites fail?

A: It may be because the website has an anti - scraping mechanism. Try:

1. Enable the headless browser with usePuppeteer: true.
2. Set appropriate custom request headers.
3. Use pre - defined anti - detection request headers.

Q: How to handle websites that require login?

A: Use the set_domain_headers tool to set request headers containing authentication information, such as Cookie or Authorization.

Q: What if the content of an SPA application is empty?

A: Set usePuppeteer: true to use Puppeteer to render dynamic content.

📄 License

MIT License

🤝 Contribution

Welcome to submit Issues and Pull Requests!