Bsmi021 MCP Server Webscan

🚀 MCP Web Scan Server

This is a Model Context Protocol (MCP) server for web content scraping, analysis, and information extraction. The server provides a set of tools for retrieving, analyzing, and extracting information from web pages.

🚀 Quick Start

The MCP Web Scan Server offers a variety of tools to scrape, analyze, and extract information from web pages. It runs on the stdio transport protocol, making it compatible with MCP clients such as Claire Desktop.

✨ Features

• Page Scraping: Convert web pages into Markdown format for easy analysis.
• Link Extraction: Extract and analyze all links from web pages.
• Website Crawling: Recursively crawl website content to discover more information.
• Link Checking: Identify broken links on web pages.
• Pattern Matching: Find URLs that match specific patterns.
• Sitemap Generation: Generate XML sitemaps for websites.

📦 Installation

# Clone the repository
git clone <repository-url>
cd mcp-server-webscan

# Install dependencies
npm install

# Build the project
npm run build

💻 Usage Examples

Basic Usage

Start the Server

npm start

The server runs on the stdio transport protocol, so it is compatible with MCP clients like Claire Desktop.

Available Tools

1. fetch_page

   • Function: Fetch a web page and convert it to Markdown format.
   • Parameters:
     • url (required): The URL of the page to fetch.
     • selector (optional): A CSS selector to specify the content target.

2. extract_links

   • Function: Extract all links and their text from a web page.
   • Parameters:
     • url (required): The URL of the page to analyze.
     • baseUrl (optional): The base URL for filtering links.

3. crawl_site

   • Function: Recursively crawl website content up to a specified depth.
   • Parameters:
     • url (required): The starting URL for the crawl.
     • maxDepth (optional, default: 2): The maximum crawl depth.

4. check_links

   • Function: Check all links on a page for validity.
   • Parameters:
     • url (required): The URL of the page to check links on.

5. find_patterns

   • Function: Find URLs that match a specific pattern.
   • Parameters:
     • url (required): The URL to search.
     • pattern (required): A regular expression for matching URLs.

6. generate_sitemap

   • Function: Generate a simple XML sitemap.
   • Parameters:
     • url (required): The base URL for the sitemap.
     • maxUrls (optional, default: 100): The maximum number of links to include in the sitemap.

Advanced Usage

Example Usage with Claire Desktop

1. Configure the server in Claire Desktop settings:

{
  "mcpServers": {
    "webscan": {
      "command": "node",
      "args": ["path/to/mcp-server-webscan/dist/index.js"],
      "env": {
        "NODE_ENV": "development"
      }
    }
  }
}

2. Use the tools in a conversation:

Can you scrape the content of https://example.com and convert it to Markdown format?

📚 Documentation

Development

Prerequisites

• Node.js >= 18
• npm

Project Structure

mcp-server-webscan/
├── src/
│   └── index.ts    # Main server implementation file
├── dist/           # Compiled JavaScript files
├── package.json
└── tsconfig.json

Build

npm run build

Development Mode

npm run dev

Error Handling

The server implements comprehensive error handling for:

• Invalid parameters
• Network errors
• Content parsing errors
• URL validation errors

All errors are handled according to the MCP specification.

📄 License

Thank you for using this project! If you have any questions or suggestions during use, please feel free to contact us.