Scrapling Fetch MCP

🚀 Web Scraping MCP Service

This is an MCP server that helps AI assistants access text content on websites implementing anti-bot detection, bridging the gap between what you can see in your browser and what the AI can access.

🚀 Quick Start

This tool is optimized for low-volume retrieval of documents and reference materials (text/HTML only) from websites implementing anti-bot detection. It is not designed for general web scraping or data collection.

⚠️ Important Note

This project was developed in collaboration with Claude Sonnet 3.7, using LLM Context.

📦 Installation

1. Prerequisites:

   • Python 3.10+
   • uv package manager

2. Install dependencies and tools:

uv tool install scrapling
scrapling install
uv tool install scrapling-fetch-mcp

💻 Usage Examples

Set up with Claude

Add the following configuration to the MCP server configuration of your Claude client:

{
  "mcpServers": {
    "Cyber-Chitta": {
      "command": "uvx",
      "args": ["scrapling-fetch-mcp"]
    }
  }
}

Retrieve a full page

Human: Please scrape and summarize the document at https://example.com/docs

Claude: I'll help you with that. Let me scrape the document.

<mcp:function_calls>
<mcp:invoke name="s-fetch-page">
<mcp:parameter name="url">https://example.com/docs</mcp:parameter>
<mcp:parameter name="mode">basic</mcp:parameter>
</mcp:invoke>
</mcp:function_calls>

Based on the document I retrieved, here's the summary...

Extract specific content using regular expressions

Human: Please find all mentions of "API keys" in the document page.

Claude: I'll search for this specific information.

<mcp:function_calls>
<mcp:invoke name="s-fetch-pattern">
<mcp:parameter name="url">https://example.com/docs</mcp:parameter>
<mcp:parameter name="mode">basic</mcp:parameter>
<mcp:parameter name="search_pattern">API\s+keys?</mcp:parameter>
<mcp:parameter name="context_chars">150</mcp:parameter>
</mcp:invoke>
</mcp:function_calls>

I found multiple mentions of "API keys" in the document:
...

✨ Features

This package provides two different tools:

1. s-fetch-page: Full web page retrieval with pagination support
2. s-fetch-pattern: Match content using regular expressions and extract context

Feature Options

• Protection Levels:

  • basic: Fast retrieval (1 - 2 seconds), but lower success rate on heavily protected sites
  • stealth: Balanced protection (3 - 8 seconds), suitable for most websites
  • max-stealth: Maximum protection (10+ seconds) for heavily protected sites

• Content Location Options:

  • s-fetch-page: Support paginated retrieval of the entire page using start_index and max_length
  • s-fetch-pattern: Extract specific content using regular expressions (use search_pattern and context_chars)
    • Results include location information for subsequent s-fetch-page queries

💡 Usage Tip

• Start with the basic mode and only upgrade to a higher protection level when necessary.
• When dealing with large documents, use the pagination parameters of s-fetch-page.
• Use s-fetch-pattern when you need to find specific information.
• The AI will automatically adjust its approach based on the website's protection level.

🔧 Technical Details

Limitations

• Designed for text content only: Specifically targeted at documents, articles, and reference materials.
• Not suitable for high-volume scraping or data collection.
• May not handle websites that require authentication.
• Performance varies depending on website complexity.

📄 License

Apache 2.0