AI Cursor Scraping Assistant

🚀 AI-Cursor-Scraping-Assistant

A powerful tool that leverages Cursor AI and MCP (Model Context Protocol) to effortlessly generate web crawlers for various types of websites. This project helps you quickly analyze websites and generate suitable Scrapy frameworks or Camoufox templates with minimal effort.

🚀 Quick Start

AI-Cursor-Scraping-Assistant combines artificial intelligence and web scraping technology, aiming to improve the efficiency and intelligence level of web data collection. The tool mainly consists of two parts: Cursor AI and the MCP protocol. Cursor AI is responsible for analyzing and processing web page content, while the MCP protocol is used to generate efficient crawling strategies.

✨ Features

• Cursor AI: Analyzes and processes web page content.
• MCP Protocol: Generates efficient crawling strategies.

📦 Installation

Before using AI-Cursor-Scraping-Assistant, ensure your environment meets the following requirements:

1. Install Python 3.8 or a higher version.
2. Make sure the necessary dependencies are installed, including:
   • Scrapy Framework: For web crawler development.
   • Camoufox Template: For generating efficient crawling strategies.

Please follow these steps to install the project:

# Clone the repository
git clone https://github.com/your-username/AI-Cursor-Scraping-Assistant.git

# Navigate to the project directory
cd AI-Cursor-Scraping-Assistant

# Install dependencies
pip install scrapy camoufox

📚 Documentation

MCP Server Configuration

1. Open the MCPfiles/xpath_server.py file.
2. Update the environment variables according to the comments:

import os
os.environ['CAMOUFOX_FILE_PATH'] = 'Path/to/Camoufox_template.py'

Web Crawler Templates

• Scrapy Framework: For basic web crawler development.
• Camoufox Template: For generating efficient crawling strategies.

Detailed Rule Explanations

Website Analysis Rules

The website-analysis.mdc file contains rules on how to analyze and process website content, including the following steps:

1. Analyze the web page structure.
2. Extract the required data.
3. Process the extracted data.

Scrapy Framework Best Practices

The scrapy.mdc file provides best practices when using the Scrapy framework, including:

• Ensure the crawler code is concise and easy to understand.
• Use middleware and extensions to enhance functionality.
• Regularly update the crawling strategy to adapt to website changes.

Step-by-Step Guide to Creating Web Crawlers

The scepy-step-by-step-process.mdc file provides detailed steps for creating web crawlers, including:

1. Determine the crawling target.
2. Design the crawler logic.
3. Implement and test the crawler.

Web Crawler Templates

The scraper-models.mdc file provides different types of web crawler templates, including:

• E-commerce Websites: For scraping product information.
• News Websites: For scraping news data.

Prerequisites

The prerequisites.mdc file lists the settings that need to be completed before using the tool, including:

1. Install the necessary dependencies.
2. Configure environment variables.

Project Structure

AI-Cursor-Scraping-Assistant/
├── MCPfiles/
│   ├── xpath_server.py     # MCP server file for generating XPath selectors
│   └── Camoufox_template.py # Camoufox template file for creating efficient crawling strategies
├── cursor-rules/
│   ├── website-analysis.mdc    # Website analysis rules
│   ├── scrapy.mdc              # Scrapy framework best practices
│   ├── scrapy-step-by-step-process.mdc # Step-by-step guide to creating web crawlers
│   ├── scraper-models.mdc      # Different types of web crawler templates
│   └── prerequisites.mdc       # Settings to be completed before using the tool
└── README.md                  # Project description file

Future Improvement Plans

Proxy Integration

• Add on-demand proxy support.
• Implement proxy rotation strategies.
• Support more proxy providers.
• Handle proxy authentication.
• Integrate popular proxy services.

Improved XPath Generation and Validation

• Add a validation mechanism for XPath selectors.
• Implement a feedback loop to optimize selectors.
• Manage control flow to rewrite selectors.
• Automatically fix problematic selectors.
• Handle edge cases such as dynamic content and AJAX loading.

Other Planned Features

• Support more types of web crawlers (e.g., news websites, social media).
• Integrate more anti-bot bypass technologies.
• Enhance JSON data extraction capabilities.
• Support more complex navigation patterns.
• Optimize multi-page scraping performance.

References

This project is based on articles from The Web Scraping Club:

• Claude & Cursor AI
• MCP Protocol

📄 License

Please refer to the project's LICENSE file for detailed information.