Browser Scraping & Search

🚀 Browser Automation Agent

A powerful browser automation tool built with MCP (Model Control Program), integrating web scraping and LLM intelligence. This agent can search Google, visit web pages, and intelligently scrape content from various types of websites such as GitHub, Stack Overflow, and documentation sites.

🚀 Quick Start

This Browser Automation Agent is a powerful tool that combines web scraping and LLM intelligence. It allows you to search Google, visit web pages, and scrape content from different websites. Here's a quick guide on how to get started.

✨ Features

• 🔍 Google Search Integration: Find and retrieve the top search results for any query.
• 🕸️ Intelligent Web Scraping: Customize scraping strategies based on different website types:
  • 📂 GitHub repositories
  • 💬 Stack Overflow Q&A
  • 📚 Documentation pages
  • 🌐 Generic websites
• 🧠 AI-driven Processing: Use Mistral AI to understand and process the scraped content.
• 🥷 Simple Mode: Implement browser fingerprint protection to avoid detection.
• 💾 Content Saving: Automatically save screenshots and text content of the scraped pages.

🔧 Technical Details

The project uses a client-server architecture based on MCP:

• 🖥️ Server: Handles browser automation and web scraping tasks.
• 👤 Client: Provides an AI interface using Mistral AI and LangGraph.
• 📡 Communication: The client and server interact through standard I/O.

📦 Installation

1. Clone the repository:

git clone https://github.com/yourusername/browser-automation-agent.git
cd browser-automation-agent

2. Install the dependencies:

pip install -r requirements.txt

3. Install the Playwright browser:

playwright install

4. Create a .env file in the project root directory and add your Mistral AI API key:

MISTRAL_API_KEY=your_api_key_here

💻 Usage Examples

Basic Usage

Run the server

python main.py

Run the client

python client.py

Example interaction

When both the server and client are running:

1. Enter your query when prompted.
2. The agent will perform the following actions:
   • 🔍 Search for relevant information on Google.
   • 🧭 Navigate to the top result.
   • 📊 Scrape content based on the website type.
   • 📸 Save the page screenshot and text content to a file.
   • 📤 Return the processed information.

Advanced Usage

The agent provides several useful functions for different scraping needs:

get_top_google_url

🔍 Search Google and return the URL of the top result for a given query.

browse_and_scrape

🌐 Visit a URL and scrape content based on the website type.

scrape_github

📂 Specifically extract README content and code blocks from GitHub repositories.

scrape_stackoverflow

💬 Extract questions, answers, comments, and code blocks from Stack Overflow pages.

scrape_documentation

📚 Optimized to extract documentation content and code examples.

scrape_generic

🌐 Extract paragraph text and code blocks from generic websites.

📚 Documentation

File Structure

browser-automation-agent/
├── main.py            # MCP server implementation
├── client.py          # Mistral AI client implementation
├── requirements.txt   # Project dependencies
├── .env               # Environment variables (API keys)
└── README.md          # Project documentation

Content Saving

• browser automation: Batch process LinkedIn requests via PhantomBuster and send connection invitations.
• network research: Obtain company information and salary data from Indeed, Glassdoor, and LinkedIn.
• data analysis: Analyze contact lists in CSV files, identify potential customers, and classify them.
• crm integration: Import the scraped data directly into Salesforce or other CRM systems.

⚠️ Important Note

• Ensure that all operations comply with the robots.txt rules of the target website and relevant laws and regulations.
• Keep your API keys secure to avoid leakage.
• Regularly update the tool to ensure compatibility with the latest website structures.

If you have any questions or feedback, please feel free to contact the support team.