Crawl4ai RAG

🚀 Crawl4AI RAG MCP Server

Crawl4AI RAG MCP Server is a comprehensive tool integrating web crawling, document storage, and vector retrieval. It aims to help users efficiently obtain information from the Internet and further process it through Retrieval Augmented Generation (RAG) technology.

🚀 Quick Start

Crawl4AI RAG MCP Server combines web crawling, document storage, and vector retrieval to offer an all - in - one solution for information acquisition and processing.

✨ Features

1. Web Crawling:

   • Use multi - threading technology to crawl multiple web pages simultaneously.
   • Support websites with dynamic content loading.
   • Provide anti - crawling mechanisms to avoid being blocked by target websites.

2. Document Storage:

   • Automatically parse and store the crawled content in a PostgreSQL - compatible database.
   • Support large - scale data storage and management.

3. Vector Retrieval:

   • Use Milvus or other vector databases to perform vectorization on text content.
   • Provide efficient similarity retrieval functions to quickly find relevant documents.

📦 Installation

Install Dependencies

pip install crawl4ai openai python - dotenv supabase - client milvus

Database Setup

1. Install PostgreSQL:

   • Download and install PostgreSQL: https://www.postgresql.org/download/
   • Configure PostgreSQL users and databases to ensure their security.

2. Install Milvus:

   • Install Milvus according to the official documentation: [https://milvus.io/docs/v1.2.0/install - milvus.md](https://milvus.io/docs/v1.2.0/install - milvus.md)
   • Configure the Milvus service to ensure its normal operation.

Environment Configuration

Create a .env file and add the following content:

# PostgreSQL Configuration
POSTGRES_USER=your_postgres_user
POSTGRES_PASSWORD=your_postgres_password
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=crawl4ai_db

# Milvus Configuration
MILVUS_HOST=http://localhost:19530
MILVUS_COLLECTION_NAME=crawl4ai_collection

Start the Server

python -m crawl4ai.server

💻 Usage Examples

Web Crawling

from crawl4ai import Crawler

crawler = Crawler()
urls = ["http://example.com", "http://target.com"]
results = crawler.scrape(urls)
print(results)

Document Storage

from crawl4ai import DatabaseManager

db_manager = DatabaseManager()
await db_manager.store_documents(documents)

Vector Retrieval

from crawl4ai import VectorStore

vector_store = VectorStore()
query = "查询内容"
results = vector_store.query_vector(query)
print(results)

📚 Documentation

Expansion and Optimization

1. Custom Crawler:

   • Extend the crawling logic according to specific requirements to handle the structures of different websites.

2. Optimize Storage Efficiency:

   • Use compression technology to reduce storage space.
   • Implement a data archiving strategy to manage historical data.

3. Improve Retrieval Performance:

   • Configure Milvus' index parameters to improve query speed.
   • Retrain the model regularly to maintain retrieval accuracy.

Security Precautions

1. Database Security:

   • Protect PostgreSQL with strong passwords and SSL connections.
   • Configure the firewall to restrict external access.

2. Web Crawling Rules:

   • Comply with the rules in the robots.txt file.
   • Set an appropriate request frequency to avoid putting pressure on target websites.

3. Data Privacy:

   • Ensure that the processed data complies with relevant laws and regulations.
   • Take encryption measures to protect sensitive information.