MCP Server Webcrawl: Advanced Multi - format Web Crawler Integrated with Claude for AI Dev

MCP Server Webcrawl

mcp - server - webcrawl is an advanced web crawler data search and retrieval tool designed specifically for AI clients. It supports multiple crawler formats (such as WARC, wget, etc.), provides full - text search, Boolean logic queries, and resource type/status filtering functions. It can be seamlessly integrated with Claude Desktop, is installed via Python, and is suitable for tasks such as building website knowledge bases or conducting SEO/performance audits.

Search tools Research and data #Web crawler #Full - text search #AI tool #Data analysis

rating : 2.5 points

downloads : 3

update time : 2025-07-24

What is the MCP server?

The MCP server is an intelligent system specifically designed for analyzing and searching web crawler data. It helps users find, filter, and analyze web page content obtained from different crawler tools through advanced search functions.

How to use the MCP server?

The MCP server can be installed and run via the command line and supports data input from multiple web crawler tools. Users can use simple keywords or complex Boolean logic queries to retrieve specific information.

Applicable scenarios

Suitable for scenarios such as SEO audits, website performance analysis, and 404 error detection. Ideal for users who need to conduct in - depth analysis of web page content, such as website administrators, developers, and data analysts.

Main features

Multi - crawler compatibilitySupports data input from multiple web crawler tools (such as WARC, wget, InterroBot, etc.), facilitating users to integrate data from different sources.

Advanced search functionProvides functions such as Boolean logic search, field search, and wildcard matching to help users accurately locate the required information.

Content analysisSupports functions such as Markdown conversion, regular expression extraction, and XPath selectors, facilitating in - depth analysis of web page content.

Visual interfaceProvides an intuitive user interface, enabling non - technical personnel to easily use advanced search functions.

Advantages and limitations

Advantages

Supports multiple web crawler tools, facilitating data integration

Provides powerful search functions to meet complex query requirements

Easy to install and use, suitable for users with different technical levels

Limitations

Requires a certain technical background to fully utilize all functions

For very large data sets, performance may be affected

Some advanced functions may require additional configuration

How to use

Install the MCP server

Install the MCP server using pip in the command line: pip install mcp - server - webcrawl

Start the MCP server

After installation, run the MCP server to start processing data.

Import crawler data

Import your crawler data (such as WARC files) into the MCP server.

Perform a search

Use keywords, Boolean logic, or field search to find the information you need.

Usage examples

SEO auditUse the MCP server to analyze the SEO situation of a website, find potential problems, and provide improvement suggestions.

404 error detectionDetect 404 error links on the website and analyze their distribution.

Performance analysisAnalyze the speed and performance of a website and identify factors affecting the loading time.

Frequently Asked Questions

Which crawler formats does the MCP server support?

How to install the MCP server?

What environment does the MCP server require?

Can the MCP server handle large data sets?

Related resources

Official website

The official website of the MCP server, providing detailed product information and usage guides.

GitHub repository

The GitHub code repository of the MCP server, providing source code and project documentation.

Documentation center

The official documentation of the MCP server, providing detailed usage instructions and tutorials.

PyPI page

The PyPI page of the MCP server, providing installation and usage information.

🚀 mcp-server-webcrawl

Advanced search and retrieval for web crawler data. This project enables your AI client to filter and analyze web content either under your guidance or autonomously.

Website | GitHub | Docs | PyPi

🚀 Quick Start

mcp-server-webcrawl is free and open source, and requires Claude Desktop and Python (>=3.10). It can be installed on the command line via pip:

pip install mcp-server-webcrawl

For step-by-step MCP server setup, refer to the Setup Guides.

✨ Features

Ready for Claude Desktop
Compatible with multiple web crawlers
Ability to filter by type, status, etc.
Support for boolean search
Support for Markdown and snippets
Allows you to build your own website knowledge base

📦 Installation

mcp-server-webcrawl requires Claude Desktop and Python (>=3.10). You can install it using the following command:

pip install mcp-server-webcrawl

For detailed setup steps, please refer to the Setup Guides.

📚 Documentation

Supported Crawlers

mcp-server-webcrawl works with a variety of web crawlers:

Crawler/Format	Description	Platforms	Setup Guide
WARC	Standard web archive format	varies by client	Setup Guide
wget	CLI website mirroring tool	macOS/Linux	Setup Guide
InterroBot	GUI crawler and analyzer	macOS/Windows	Setup Guide
Katana	CLI security-focused crawler	macOS/Windows/Linux	Setup Guide
SiteOne	GUI crawler and analyzer	macOS/Windows/Linux	Setup Guide

Prompt Routines

mcp-server-webcrawl provides toolkits for free - style searching of web - crawl data. It can also run routines (as prompts). You can either write your own prompts or use the provided ones. These prompts are copy - and - paste and used as raw Markdown.

Prompt	Download	Category	Description
🔍 SEO Audit	`auditseo.md`	audit	Technical SEO (search engine optimization) analysis. Covers the basics, with options to dive deeper.
🔗 404 Audit	`audit404.md`	audit	Broken link detection and pattern analysis. Not only finds issues, but suggests fixes.
⚡ Performance Audit	`auditperf.md`	audit	Website speed and optimization analysis. Real talk.
📁 File Audit	`auditfiles.md`	audit	File organization and asset analysis. Discover the composition of your website.
🌐 Gopher Interface	`gopher.md`	interface	An old - fashioned search interface inspired by the Gopher clients of yesteryear.
⚙️ Search Test	`testsearch.md`	self - test	A battery of tests to check for Boolean logical inconsistencies in the search query parser and subsequent FTS5 conversion.

If you want to shortcut the site selection, paste the markdown and in the same request, type "run pasted for [site name or URL]." When pasted without additional context, you will be prompted to select from a list of crawled sites.

Boolean Search Syntax

The query engine supports field - specific (field: value) searches and complex boolean expressions. Fulltext search combines the url, content, and headers fields.

Query Example	Description
privacy	fulltext single keyword match
"privacy policy"	fulltext match exact phrase
boundar*	fulltext wildcard matches results starting with boundar (boundary, boundaries)
id: 12345	id field matches a specific resource by ID
url: example.com/somedir	url field matches results with URL containing example.com/somedir
type: html	type field matches for HTML pages only
status: 200	status field matches specific HTTP status codes (equal to 200)
status: >=400	status field matches specific HTTP status code (greater than or equal to 400)
content: h1	content field matches content (HTTP response body, often, but not always HTML)
headers: text/xml	headers field matches HTTP response headers
privacy AND policy	fulltext matches both
privacy OR policy	fulltext matches either
policy NOT privacy	fulltext matches policies not containing privacy
(login OR signin) AND form	fulltext matches fulltext login or signin with form
type: html AND status: 200	fulltext matches only HTML pages with HTTP success

Field Search Definitions

Field search allows you to specify which columns of the search index to filter, improving search precision.

Field	Description
id	database ID
url	resource URL
type	enumerated list of types (see types table)
size	file size in bytes
status	HTTP response codes
headers	HTTP response headers
content	HTTP body—HTML, CSS, JS, and more

Field Content

A subset of fields can be independently requested with results, while core fields are always included.

Field	Description
id	always available
url	always available
type	always available
status	always available
created	on request
modified	on request
size	on request
headers	on request
content	on request

Content Types

The type: field search allows filtering by broad content type groups.

Type	Description
html	webpages
iframe	iframes
img	web images
audio	web audio files
video	web video files
font	web font files
style	CSS stylesheets
script	JavaScript files
rss	RSS syndication feeds
text	plain text content
pdf	PDF files
doc	MS Word documents
other	uncategorized

Extras

The extras parameter provides additional processing options.

Extra	Description
thumbnails	Generates base64 encoded images to be viewed and analyzed by AI models. Enables image description, content analysis, and visual understanding while keeping token output minimal. Works with images, which can be filtered using `type: img` in queries. SVG is not supported.
markdown	Provides the HTML content field as concise Markdown, reducing token usage and improving readability for LLMs. Works with HTML, which can be filtered using `type: html` in queries.
regex	Extracts regular expression matches from crawled files such as HTML, CSS, JavaScript, etc. Not as precise a tool as XPath for HTML, but supports any text file as a data source. One or more regex patterns can be requested, using the `extrasRegex` argument.
snippets	Matches fulltext queries to contextual keyword usage within the content. When used without requesting the content field (or markdown extra), it can provide an efficient means of refining a search without pulling down the complete page contents. Also great for rendering old school hit - highlighted results as a list, like Google search in 1999. Works with HTML, CSS, JS, or any text - based, crawled file.
xpath	Extracts XPath selector data, used in scraping HTML content. Use XPath's text() selector for text - only, element selectors return outerHTML. Only supported with `type: html`, other types will be ignored. One or more XPath selectors (//h1, count(//h1), etc.) can be requested, using the `extrasXpath` argument.

Extras help produce token - efficient HTTP content responses. Markdown reduces the size of source HTML, snippets are small in size, and XPath can be customized. If your LLM shows an affinity for the "content" field, you can suggest using the extras feature to manage token budget.