MCP Leetcode Crawler

🚀 LeetCode Crawler Tool

A comprehensive tool for scraping interview questions from the LeetCode discussion forum, with a special focus on questions from Google interviews. This tool extracts valuable information from discussions, organizes it by month, and supports exporting to various formats, including CSV and Google Sheets.

🚀 Quick Start

The LeetCode Crawler Tool is designed to scrape interview questions from the LeetCode discussion forum. It can filter questions by company label, extract detailed information, and organize the results by month. The tool also supports exporting data to CSV files and Google Sheets.

✨ Features

• Scrape interview questions from the LeetCode discussion forum.
• Filter by company label (default: Google).
• Extract detailed information, including question links.
• Group results by month.
• Save to CSV files for easy analysis.
• Export to Google Sheets with formatted tables.
• Support flexible use via the command-line interface.

📦 Installation

1. Clone this repository:

   git clone https://github.com/yourusername/mcp-leetcode-crawler.git
   cd mcp-leetcode-crawler
   

2. Install the required dependencies:

   pip install -r requirements.txt
   

💻 Usage Examples

Basic Usage

Run the crawler tool with default settings:

python crawler.py

This will:

1. Scrape Google interview questions from the LeetCode discussion forum.
2. Save all posts to a single CSV file (leetcode_interview_questions.csv).
3. Group results by month and save them to separate CSV files in the output directory.

Command-Line Interface

For more flexible control, use the CLI:

python cli.py --company google --pages 20 --output results.csv --output-dir monthly_data

Available options:

• --company: Company label to filter questions (default: google).
• --pages: Number of pages to scrape (default: 10).
• --output: Path to the output CSV file (default: leetcode_interview_questions.csv).
• --output-dir: Directory to output monthly files (default: output).
• --verbose: Enable detailed logging.
• --category: Category to scrape (default: interview-question).

Export to Google Sheets

To export data to Google Sheets:

1. Set up Google Sheets API credentials:
   • Go to Google Cloud Console.
   • Create a new project.
   • Enable the Google Sheets API.
   • Create OAuth 2.0 credentials (Desktop app).
   • Download the credentials file as credentials.json to the project directory.
2. Run the export tool script:

   # Export a single CSV file
   python google_sheets_exporter.py --csv leetcode_interview_questions.csv --name "Google Interview Questions"
   
   # Export all monthly data
   python google_sheets_exporter.py --monthly --name "LeetCode Monthly Interview Questions"
   

Automated Run

Use the provided shell scripts:

# Basic run (CSV only)
./run.sh

# Run with Google Sheets export
./run_with_sheets.sh

📚 Documentation

Project Structure

• crawler.py - Main crawler implementation.
• cli.py - Command-line interface.
• google_sheets_exporter.py - Tool to export data to Google Sheets.
• run.sh - Basic execution script.
• run_with_sheets.sh - Execution script including Google Sheets export.
• requirements.txt - Python dependencies.

Customization

You can modify the scripts to:

• Change the company label (e.g., from "google" to "facebook", "amazon", etc.).
• Adjust the number of pages to scrape.
• Modify the output format.
• Change the way data is grouped and organized.

Future Improvements

• [ ] Add support for multiple company labels simultaneously.
• [ ] Implement automated scheduling for regular data updates.
• [ ] Add a web interface for easy interaction.
• [ ] Integrate other data storage solutions (e.g., MongoDB).
• [ ] Add data visualization features.
• [ ] Implement full-text search functionality.

📄 License

Please refer to the license file in the project repository.