Websearch Tools

🚀 WebSearch - Advanced Web Search and Content Extraction Tool

WebSearch is a powerful web search and content extraction tool built with Python. It leverages the Firecrawl API to offer advanced web crawling, search, and content analysis capabilities.

🚀 Quick Start

WebSearch is a powerful web search and content extraction tool built with Python. It leverages the Firecrawl API to offer advanced web crawling, search, and content analysis capabilities.

✨ Features

• Intelligent Web Search: Supports intelligent search with customizable parameters.
• Content Extraction: Extracts specific information from web pages through natural language prompts.
• Website Crawling: Crawls websites with configurable depth and limits.
• Web Scraping: Scrapes content with support for multiple output formats.
• MCP Integration: Serves as a Model Context Protocol (MCP) server for seamless integration.

📋 Prerequisites

• Python 3.8 or higher
• uv package manager
• Firecrawl API key
• OpenAI API key (optional, for enhanced features)
• Tavily API key (optional)

📦 Installation

1. Clone the repository:

   git clone https://github.com/yourusername/WebSearch.git
   cd WebSearch
   

2. Install dependencies:

   pip install -r requirements.txt
   

3. Configure environment variables:

   • Create a .env file (if it doesn't exist).
   • Add the following content:

     OPENAI_API_KEY=your_openai_api_key_here
     FIRECRAWL_API_KEY=your_firecrawl_api_key_here
     

4. Run the project:

   python main.py
   

💻 Usage Examples

Basic Usage

from websearch import WebSearch

# Initialize the search object
searcher = WebSearch()

# Execute a search query
results = searcher.search("Python Tutorial")

# Process the results
for result in results:
    print(result.url)

Advanced Usage

• Customize crawling parameters:

  # Example: Limit the number of search results
  results = searcher.search("Machine Learning", max_results=10)
  

📚 Documentation

Feature List

1. Supports multiple web search strategies.
2. Provides powerful content parsing and extraction capabilities.
3. Integrates AI-accelerated search optimization.
4. Offers a flexible result filtering and sorting mechanism.

🏷️ Tags

• Python
• Web Scraping
• AI
• API
• Open Source Project

⚠️ Important Note

When using this tool, please comply with relevant laws, regulations, and the terms of use of websites.