Search Engine With Rag And MCP

🚀 Search Engine with RAG and MCP

A powerful search engine that combines LangChain, Model Context Protocol (MCP), Retrieval Augmented Generation (RAG), and Ollama to build an agent-based AI system capable of web searching, information retrieval, and providing relevant answers.

🚀 Quick Start

This search engine integrates multiple advanced technologies to offer efficient web search and information retrieval services. You can quickly start using it by following the installation and usage guides below.

✨ Features

• Web search functionality using the Exa API.
• Web page content retrieval using FireCrawl.
• RAG (Retrieval Augmented Generation) for extracting more relevant information.
• MCP (Model Context Protocol) server for standardized tool invocation.
• Support for local LLMs via Ollama and cloud-based LLMs via OpenAI.
• Flexible architecture supporting direct search, agent-based search, or server mode.
• Comprehensive error handling and graceful degradation.
• Python 3.13+ with type hints.
• Asynchronous processing for efficient execution of network operations.

🔧 Technical Details

Architecture

This project integrates several key components:

1. Search Module: Performs web searches using the Exa API and retrieves content using FireCrawl.
2. RAG Module: Embeds documents, chunks them, and stores them in a FAISS vector library.
3. MCP Server: Provides a standardized protocol for tool invocation.
4. Agent: A LangChain-based agent that utilizes search and RAG capabilities.

Project Structure

search-engine-with-rag-and-mcp/
├── LICENSE              # MIT License
├── README.md            # Project documentation
├── data/                # Data directory
├── docs/                # Documentation
│   └── env_template.md  # Environment variable documentation template
├── logs/                # Log file directory (automatically created)
├── src/                 # Main package (source code)
│   ├── __init__.py      
│   ├── core/            # Core functionality
│   │   ├── __init__.py
│   │   ├── main.py      # Entry point
│   │   ├── search.py    # Web search module
│   │   ├── rag.py       # RAG implementation
│   │   ├── agent.py     # LangChain agent
│   │   └── mcp_server.py # MCP server implementation
│   └── utils/           # Utility package
        └── __init__.py
└── requirements.txt      # Dependency management

📦 Installation

1. Clone the repository:

git clone https://github.com/your-username/search-engine-with-rag-and-mcp.git
cd search-engine-with-rag-and-mcp

2. Install dependencies:

pip install -r requirements.txt

💻 Usage Examples

Basic Usage

1. Direct Search Mode

Run the search engine server:

python src/main.py serve --mode direct

Use the search engine to perform a query:

python src/main.py search --query "your-query-here"

2. Agent-based Search Mode

Run the agent server:

python src/main.py serve --mode agent

Invoke the agent for searching via the API:

curl -X POST http://localhost:8000/agent/search -d '{"query": "your-query-here"}'

3. MCP Server Mode

Run the MCP server:

python src/main.py serve --mode mcp

Invoke the MCP service via the HTTP protocol:

curl -X POST http://localhost:8001/mcp/execute -d '{"tool": "search", "input": "your-query-here"}'

Development

The project adheres to the following best practices:

• Code Formatting: Use black for automatic formatting to ensure consistent coding styles.
• Type Checking: Utilize mypy for static type checking to avoid runtime errors.
• Linting: Apply flake8 and pylint for code quality control.
• Unit Testing: Use pytest to write and execute unit tests to ensure functional correctness.

📄 License

This project is licensed under the MIT License. Please refer to the LICENSE file for detailed information.

Acknowledgments

Thanks to the contributions of the following projects:

• LangChain for providing the agent framework.
• Model Context Protocol for providing standardized tool invocation.
• Ollama for local LLM capabilities.
• Exa for web search functionality.
• FireCrawl for content retrieval functionality.