Websearch Tools

🚀 WebSearch Search Tool

WebSearch is an advanced web search and content extraction tool based on Python, integrating the powerful capabilities of Firecrawl and OpenAI. This project aims to provide users with efficient and intelligent web data collection and analysis capabilities.

🚀 Quick Start

WebSearch is a Python - based tool that combines Firecrawl and OpenAI to offer advanced web search and content extraction. Below are the steps to get started.

✨ Features

• Powerful Web Crawling: Utilize Firecrawl's high - performance API for web page scraping.
• Intelligent Content Parsing: Implement content understanding and extraction through OpenAI's AI technology.
• Flexible Interface Design: Support output in multiple data formats (such as JSON, XML).
• Easy Integration: Provide detailed documentation and examples for users to quickly get started.

📦 Installation

Prerequisites

1. Python Version Requirement:
   • Python 3.6 or higher is required.
2. Package Manager:
   • Ensure that the uv package manager is installed (for dependency management).
3. API Keys:
   • OpenAI API Key: Required for AI functionality. Obtain it from the OpenAI platform.
   • Firecrawl API Key: Required for web crawling functionality. Obtain it from the Firecrawl official website.

Installation Steps

1. Clone the Repository

git clone https://github.com/yourusername/WebSearch.git
cd WebSearch

2. Install Dependencies

Use the uv package manager to install all dependencies:

uv install requirements.txt

3. Configure Environment Variables

Create and edit the .env file (the file should be located in the project root directory):

OPENAI_API_KEY=your_openai_key_here
FIRECRAWL_API_KEY=your_firecrawl_key_here

💻 Usage Examples

Integrate the Tool into Claude for Desktop

1. Open the claudius.json file.
2. Add the following configuration under the root directory:

{
    "webSearch": {
        "apiKey": "your_firecrawl_api_key",
        "aiIntegration": true,
        "proxyEnabled": false
    }
}

3. Restart the Claude software to apply the changes.

Available Tools

1. Web Search:
   • Input: Keywords or query statements.
   • Output: A list of search results in JSON format, including titles, links, and summaries.
2. Content Extraction:
   • Input: Web page URL.
   • Output: Structured data (such as text, image links).
3. Data Analysis:
   • Input: A list of batch URLs.
   • Output: Statistical reports or visual charts.

📚 Documentation

API Reference

Web Search Interface

def search(query: str, count: int = 10) -> dict:
    """
    Perform a web search and return the results.
    
    Args:
        query (str): Search keywords.
        count (int): Number of results, default is 10.
        
    Returns:
        dict: JSON - formatted data containing search results.
    """

Content Extraction Interface

def extract(url: str) -> dict:
    """
    Extract the content of the specified web page.
    
    Args:
        url (str): The URL of the web page to extract content from.
        
    Returns:
        dict: JSON - formatted results containing structured data.
    """

Configuration Guide

Environment Variable Settings

Create a .env file and add the following content:

# OpenAI API Key - for AI functionality
OPENAI_API_KEY=your_openai_api_key_here

# Firecrawl API Key - for web crawling functionality
FIRECRAWL_API_KEY=your_firecrawl_api_key_here

Obtain API Keys

1. OpenAI API Key:
   • Visit the OpenAI platform.
   • After logging in, go to the API Keys interface and create a new Secret Key.
2. Firecrawl API Key:
   • Visit the Firecrawl official website.
   • After registering or logging in, generate a new key on the dashboard.

🔧 Troubleshooting

Error Types

• Ensure that the API keys in the .env file are filled in correctly.
• Check the network connection to ensure that external services can be accessed.

Common Issues

• If a ConnectionError occurs, it may be caused by proxy settings or network restrictions.
• If an AuthenticationError occurs, please re - check if the keys are valid.

📄 Contact Information

• Project URL: GitHub Repository
• Email: contact@websearchproject.com
• Discussion Group: Join the WebSearch User Group

Thank you for using the WebSearch tool! If you need help, please contact our support team at any time.