Documentation Crawler

🚀 MCP Server Documentation

This MCP server is a powerful tool for managing and processing documentation. It supports automatic loading of JSON - formatted documents, has a built - in crawler for fetching content from official documentation sites, and provides useful tools for searching and retrieving document details.

🚀 Quick Start

📦 Installation

Install the MCP server using Node.js:

git clone [Repository URL]
cd mcp-server
npm install

💻 Usage Examples

Basic Usage

To start the MCP server, run the following command:

node server.js

By default, the server will start at http://localhost:3000.

Document Loading Process

1. Automatic Detection and Loading: When the server starts, it automatically scans and loads all JSON files in the docs directory.
2. Backup Loading Mechanism: If no document files are found in the project directory, it tries to load from the current working directory.
3. Page ID Determination: By default, the URL is used as the unique identifier for the page, and there is no need to specify an additional url field.
4. Source Name Handling: All document source names are automatically converted to lowercase to ensure consistency.

Document Format

Document files should be in JSON format with the following structure:

{
  "source": {
    "name": "taro",
    "url": "https://docs.taro.zone/docs"
  },
  "lastUpdated": "2024-03-27T12:00:00Z",
  "pages": {
    "https://docs.taro.zone/docs/components-desc": {
      "title": "Component Library Description | Taro Documentation",
      "content": "Page content..."
    }
    // More pages...
  }
}

🕷️ Crawler Functionality

The system has a built - in crawler that can fetch content from official documentation sites of various frameworks and convert it into a locally usable format. The crawler features include:

1. Multi - site Support: Supports documentation websites of any framework and library, and is fully configurable.
2. Selective Crawling: Precisely control the content to be crawled through inclusion and exclusion patterns.
3. Intelligent Content Extraction: Automatically identify page titles, body content, and relevant structural information.
4. High - concurrency Processing: Supports multi - threaded crawling simultaneously to improve efficiency.
5. Automatic Conversion: Automatically convert the fetched content into standard JSON - formatted documents.
6. Fault - Tolerant Mechanism: Has timeout handling and retry functions to ensure stability.

Crawler Configuration Example

Add the following content to config.json to configure the crawler:

{
  "crawlers": {
    "taro": {
      "url": "https://docs.taro.zone/docs",
      "include": ["components", "tutorial"],
      "exclude": ["deprecated"]
    }
  }
}

🛠️ MCP Tools

The MCP server provides the following tools:

search_docs - Document Search

• Parameters:
  • query: Search keyword (string, required)
  • source: Document source name (string, optional)
  • limit: Maximum number of results (number, optional, default is 10)
• Special Function:
  • When query is "reload", it triggers the document to be reloaded.

get_doc_detail - Get Document Details

• Parameters:
  • id: Document ID (string, required)
  • source: Document source name (string, optional)

Advanced Usage

// Search for documents
const searchRequest = {
  jsonrpc: "2.0",
  id: "search1",
  method: "tools/call",
  params: {
    name: "search_docs",
    arguments: { 
      query: "Components", 
      source: "taro",
      limit: 5
    }
  }
};

// Get document details
const detailRequest = {
  jsonrpc: "2.0",
  id: "detail1",
  method: "tools/call",
  params: {
    name: "get_doc_detail",
    arguments: { 
      id: "https://docs.taro.zone/docs/components"
    }
  }
};

⚙️ Configuration Example

Configure the MCP server in config.json:

{
  "port": 3000,
  "sources": {
    "taro": {
      "url": "https://docs.taro.zone/docs",
      "token": "your_token_here"
    }
  }
}

🧪 Testing

Create a Test Document

Create a file named example.json in the docs directory:

{
  "source": {
    "name": "test",
    "url": "http://localhost/test-docs"
  },
  "lastUpdated": "2024-03-27T12:00:00Z",
  "pages": {
    "/test": {
      "title": "Test Page",
      "content": "This is a test document."
    }
  }
}

Send a Test Request

Use the curl command to send a request:

curl http://localhost:3000 -d '{"jsonrpc":"2.0","id":"search1","method":"tools/call","params":{"name":"search_docs", "arguments":{"query":"test"}}}'

❌ Error Handling

Common Errors

• 404 Not Found: The requested resource was not found.
• 500 Internal Server Error: An internal server error occurred.

Log Viewing

Check the log files in the logs directory for more information.

📝 Release Notes

v1.0.0

• Initial version released
• Added basic functions and document loading support

v1.1.0

• Added multi - threaded crawling function
• Optimized document search performance
• Fixed known bugs