Search Engine With Rag And MCP

🚀 Search Engine with RAG and MCP

A powerful search engine that combines LangChain, Model Context Protocol (MCP), Retrieval Augmented Generation (RAG), and Ollama to build an agent-based AI system. It can perform web searches, retrieve information, and provide relevant answers.

✨ Features

• Web search functionality using Exa API.
• Web page content retrieval using FireCrawl.
• RAG (Retrieval Augmented Generation) for more relevant information extraction.
• MCP (Model Context Protocol) server for standardized tool invocation.
• Support for local LLMs via Ollama and cloud-based LLMs via OpenAI.
• Flexible architecture supporting direct search, agent-based search, or server mode.
• Comprehensive error handling and graceful degradation.
• Python 3.13+ with type hints.
• Asynchronous processing for efficient web operations.

🔧 Technical Details

This project integrates several key components:

1. Search Module: Conducts web searches using Exa API and retrieves content using FireCrawl.
2. RAG Module: Embeds documents, chunks them, and stores them in a FAISS vector library.
3. MCP Server: Provides a standardized protocol for tool invocation.
4. Agent: A LangChain-based agent that utilizes search and RAG capabilities.

📦 Project Structure

search-engine-with-rag-and-mcp/
├── LICENSE              # MIT License
├── README.md            # Project documentation
├── data/                # Data directory
├── docs/                # Documentation
│   └── env_template.md  # Environment variable documentation template
├── logs/                # Log file directory (automatically created)
├── src/                 # Main package (source code)
│   ├── __init__.py      
│   ├── core/            # Core functionality
│   │   ├── __init__.py
│   │   ├── main.py      # Entry point
│   │   ├── search.py    # Web search module
│   │   ├── rag.py       # RAG implementation
│   │   ├── agent.py     # LangChain agent
│   │   └── mcp_server.py # MCP server implementation
│   └── utils/           # Utility package
        └── __init__.py
└── requirements.txt      # Dependency management

📦 Installation

1. Clone the repository:

git clone https://github.com/your-username/search-engine-with-rag-and-mcp.git
cd search-engine-with-rag-and-mcp

2. Install dependencies:

pip install -r requirements.txt

💻 Usage Examples

Basic Usage

1. Direct Search Mode

Run the search engine server:

python src/main.py serve --mode direct

Use the search engine to query:

python src/main.py search --query "your-query-here"

2. Agent-based Search Mode

Run the agent server:

python src/main.py serve --mode agent

Search through the API call to the agent:

curl -X POST http://localhost:8000/agent/search -d '{"query": "your-query-here"}'

3. MCP Server Mode

Run the MCP server:

python src/main.py serve --mode mcp

Invoke the MCP service via HTTP protocol:

curl -X POST http://localhost:8001/mcp/execute -d '{"tool": "search", "input": "your-query-here"}'

📚 Documentation

The project follows the following best practices:

• Code Formatting: Use black for automatic formatting to ensure consistent coding styles.
• Type Checking: Leverage mypy for static type checking to avoid runtime errors.
• Linting: Apply flake8 and pylint for code quality control.
• Unit Testing: Use pytest to write and execute unit tests to ensure functional correctness.

📄 License

This project is licensed under the MIT License. For detailed information, please refer to the LICENSE file.

🙏 Acknowledgments

Thanks to the contributions of the following projects:

• LangChain for providing the agent framework.
• Model Context Protocol for providing standardized tool invocation.
• Ollama for local LLM capabilities.
• Exa for web search functionality.
• FireCrawl for content retrieval functionality.