MCP Web Scrapper

🚀 Quick Start with MCP Server Scraper

This project enables you to scrape and extract MCP web servers from glama.ai. Follow the steps below to set up and run the scraper.

📦 Prerequisites

• Node.js
• npm

📦 Installation

1. Clone the repository:

   git clone https://github.com/Neche-Stephen/mcp_web_scrapper
   cd mcp_web_scrapper
   

2. Install dependencies:

   npm install
   

🗃 Check Existing Data

Before running a new scrape:

1. Check if the servers-json folder exists.

2. If it exists and contains files, it means data has been scraped previously.

3. To run a new batch, delete the servers-json folder:

   rm -rf servers-json
   

Step 1: Scrape MCP Server Links

Run the scraper to collect all MCP server links:

node scraper.js

• The scraper will launch a headless browser and start collecting data.
• Please be patient – this may take a few minutes.
• After the scraping is complete, you'll see the message "Closing browser".

Output: A file named server-links.json will be generated in the servers-json folder, containing all the scraped links.

Step 2: Process and Extract Server Data

Now, get detailed information from each server link:

node process-links.js

• The script reads server-links.json and visits each link to scrape the necessary data.
• A progress report will be displayed in the terminal.
• After the scraping is complete, you'll see a success message.

All the scraped data will be stored in different files in the servers-json directory, with the file names being the corresponding MCP server names.