Mult Fetch MCP Server

🚀 Multithreaded Web Crawling - Project Documentation

mult-fetch is a multithreaded web crawling library that supports switching between Node.js and browser modes. It offers powerful configuration options and extensibility, making it suitable for complex web page crawling tasks.

✨ Features

• Multithreaded Operation: Supports starting multiple crawling threads simultaneously.
• Intelligent Mode Switching: By default, it uses Node.js fetch. If it fails, it automatically switches to BrowserFetcher (based on Puppeteer).
• Rich Configuration Options: Provides detailed configuration items such as proxy, debugging, and browser settings.
• Content Processing Capability: It has a built - in content extraction function that can filter out ads and useless information.

🚀 Quick Start

📦 Installation

npm install mult-fetch chrome-driver

💻 Usage Examples

Basic Usage

const fetch = require('mult-fetch');
 
// Create a default configuration object
const defaultConfig = {
  // Whether to enable browser mode, default is false
  useBrowser: false,
  // Whether to automatically detect and switch modes, default is true
  autoDetectMode: true,
  // Timeout for waiting for the page to load, default is 5000ms
  waitForTimeout: 5000,
  // Proxy configuration, in descending order of priority:
  proxy: {
    server: 'http://localhost:1080' // Example of a SOCKS5 proxy
  }
};

// Create an instance
const fetcher = new fetch.MultiFetcher(defaultConfig);

// Execute the crawling task
fetcher.fetch('https://example.com', (err, resp, body) => {
  if (err) {
    console.error('[FETCH]', err.message);
    return;
  }

  // Process the response content
  const $ = cheerio.load(body);
  // Extract the article content
  const content = $('article').text();
  console.log('Content:', content);
});

📚 Documentation

Configuration Options

Network Configuration

// Proxy priority:
1. Proxy specified in the command line (if any)
2. The proxy parameter explicitly set in the request
3. Obtained from environment variables when useSystemProxy is true
4. Read from the git config when useSystemProxy is true

Note: When proxy is explicitly set, useSystemProxy will be automatically set to false.

Browser Mode Configuration

// Browser-related configuration:
{
  // Whether to wait for a specific selector to appear, default is 'body'
  waitForSelector: string | null,
  // Timeout for waiting, default is 5000ms
  waitForTimeout: number,
  // Whether to scroll to the bottom, default is false
  scrollToBottom: boolean,
  // Whether to save cookies, default is true
  saveCookies: boolean,
  // Whether to close the browser after crawling is completed, default is false
  closeBrowser: boolean
}

Content Processing Options

{
  // Whether to enable the content extraction function, default is false
  extractContent: boolean,
  // Whether to include metadata when extracting content, default is false
  includeMetadata: boolean,
  // When extractContent is true, the number of retries on failure, default is 3 times
  extractRetries: number
}

Debug Information

Log Output Format

[LOG]    - Normal log information
[INFO]   - System prompt information
[DEBUG]  - Debug-related information (debug mode needs to be enabled)
[ERROR]  - Error information
[FETCH] - Crawling-related operations

Log Prefix Explanation

• [MAIN]: Logs from the main process
• [CHILD]: Logs from child processes
• [FETCH]: Logs from a single crawling task
• [EXTRACT]: Logs from the content extraction module

Environment Variable Configuration

# Enable debug mode
DEBUG=mult-fetch:*

# Set the default number of threads
MAX_THREADS=10

# Configure the proxy address
FETCH_PROXY=http://localhost:1080

License

This project follows the MIT open - source license. For more details, please refer to the LICENSE file.

Appendix

Error Code Explanation

• ECONNREFUSED: Connection refused
• ETIMEDOUT: Request timed out
• ENOTFOUND: Domain name not found
• EBADURL: Illegal URL format

Event Listening

// Listen for global error events
process.on('uncaughtException', (err) => {
  console.error('Caught an error:', err);
});

// Listen for the process exit event
process.on('exit', () => {
  console.log('Process is exiting...');
});

Internal Utility Functions

// Get information about all running threads
fetcher.getThreadsInfo();

// Force terminate all threads
fetcher.terminate();

More Documentation

Please refer to the official documentation: mult-fetch.github.io