MCP Rag Go

🚀 MCP RAG Proof of Concept — Oil & Gas Prototype

This project is a Proof of Concept for Multi Context Protocol (MCP) and Retrieval-Augmented Generation (RAG) in the oil and gas industry. This application connects various internal domain sources (drilling, production, HSSE, purchase order, timeseries, and RAG search) into a single router, enabling automatic routing of client queries to the appropriate service or repository.

🚀 Quick Start

Prerequisites

• Go 1.22+
• MySQL 8+
• Node.js 18+ (for the frontend)

Set Up the Database

make build
make gen-data
make demo-data
make ingest-docs

Run the Service

# Aminer mysql
[http://localhost:8080/chat](http://localhost:18080/?)

Run the Frontend

# Access http://localhost:5173

✨ Features

MCP Router & Tools

• get_production, get_timeseries, get_drilling_events, get_po_status, get_po_vendor_compare, get_po_vendor_summary, summarize_npt_events, answer_with_docs, get_po_top_amount (Top-N by amount).

RAG Hybrid

The /rag/search_v2 endpoint directly uses MySQL (BM25 + cosine) on doc_chunks.embedding (JSON).

Answer With Docs

Answers quote documents with citations (DOC-XXXX#pY).

Chat SSE

The /chat/stream endpoint follows this process: plan (LLM) → NormalizePlan → execute routes (MCP/RAG) → stream answers.

Plan Normalizer

• Rewrites the rag route to the rag_search_v2 tool (payload {query, top_k, alpha}).
• Falls back from detect_anomalies to RAG if the payload is incorrect but the question has a document-based nuance.
• Automatically corrects the case of Top-N PO by amount by using get_po_top_amount.

Observability

• /metrics, /healthz, and structured logs.

📦 Installation

Prerequisites

• Go 1.22+
• MySQL 8+
• Node.js 18+ (for the frontend)

Set Up the Database

make build
make gen-data
make demo-data
make ingest-docs

Run the Service

# Aminer mysql
[http://localhost:8080/chat](http://localhost:18080/?)

Run the Frontend

# Access http://localhost:5173

💻 Usage Examples

Basic Usage

To start using the application, follow the installation steps above. Then, you can use the following endpoints:

• Health & Metrics: GET /healthz, GET /readyz, GET /metrics
• Chat SSE: GET|POST /chat/stream?q=...
• MCP Router (HTTP-internal): POST /mcp/route
• RAG Hybrid: GET|POST /rag/search_v2 with the body {"query":"...","top_k":10,"alpha":0.6}
• Domain HTTP (mirror MCP): /api/timeseries, /api/drilling-events, /api/po/status, /api/production, /api/work-orders/search, /api/npt/summarize, /api/po/vendor-compare, /api/answer-with-docs, etc.

Advanced Usage

For more complex scenarios, you can use the application's advanced features:

• Plan Normalization: The application automatically handles various planning normalization tasks, such as rewriting RAG routes, handling anomaly detection fallbacks, and correcting Top-N PO by amount routes.
• Chat SSE Workflow: The /chat/stream endpoint follows a complex workflow that includes planning, normalization, route execution, and answer synthesis. You can customize this workflow based on your specific requirements.

📚 Documentation

What is MCP?

Multi Context Protocol (MCP) is a protocol for connecting LLMs with various tools or services behind the scenes. With MCP, user questions can be automatically mapped to a specific domain (e.g., drilling events, purchase orders, or RAG search) and then executed in the most appropriate way.

What is RAG?

Retrieval-Augmented Generation (RAG) is a technique that combines the capabilities of LLMs with document-based searches. Instead of relying solely on the model, RAG extracts the most relevant text passages from the database (e.g., doc_chunks) and uses them to generate more accurate and cited answers.

Problem

In the oil and gas industry, operational data is scattered across multiple domains and separate systems:

• Drilling → drilling reports & NPT (Non Productive Time).
• Production → daily production data & timeseries.
• HSSE → incident records & safety reports.
• Purchase Order → purchase status & vendors.
• Technical documents → PDF/Word files scattered across various departments.

As a result, to answer a simple question (e.g., "How many HSSE incidents were there at rig B07 this month?"), users must:

• Contact multiple teams/departments.
• Manually search for report files.
• Wait for data validation and summarization.

This process can take days and is prone to inconsistencies between sources.

Solution

This MCP + RAG project offers a modern approach:

• MCP Router → acts as a smart gateway that routes user questions to the appropriate domain/data repository (drilling, production, HSSE, PO, timeseries).
• RAG Search → combines keyword search (BM25) and semantic search (cosine similarity) to retrieve the most relevant document passages.
• LLM Integration → understands and answers users' natural language queries with context, complete with citations from source documents.
• Chat SSE → answers are sent in a streaming manner, making the interaction interactive, similar to chatting with an AI assistant.

Impact on Oil and Gas Companies

• ⏱ Faster → cross-domain answers in seconds.
• 📊 More accurate → answers based on documents and citations, easy to verify.
• 🔄 More efficient → reduces the coordination burden between departments.
• 🧩 More flexible → easily expandable with additional tools/domains according to business needs.

Architecture and Chat SSE Flow

1. Planner (LLM JSON-mode) reads tools from schemas/mcp/*.schema.json.
2. NormalizePlan:
   • Routes kind:"rag" to rag_search_v2.
   • Corrects incorrect routes (e.g., top-amount PO).
   • Adds supporting RAG if relevant.
3. ExecuteRoutes:
   • Runs MCP tools in-process.
   • Runs RAG via the closure ragFn (hybrid /rag/search_v2 → fallback embeddings repo).
4. Synthesizer:
   • The LLM streams answers based on sources (or fallback extractive).

Plan Normalization (Planner)

The following aspects are automatically handled:

• RAG rewrite: All routes of kind:"rag" (including the old tool "rag") are changed to the rag_search_v2 tool with the body {query, top_k, alpha}.
• Detect anomalies → RAG: If the detect_anomalies payload is invalid and the question seems “document-based”, it is redirected to RAG.
• Top-N PO by amount: Detects phrases like “top/highest amount PO” and adds/corrects the route get_po_top_amount {limit, ...}.

Example Questions

1. What was the production of WELL_A12 on 2025-09-05?
2. Compare the total value of POs with the status delivered between SLB and Weatherford from 2025-09-20 to 2025-10-06.
3. Provide a summary of the number of POs per latest status.
4. Compare the total value of POs between Halliburton, NOV, and Weatherford from 2025-09-01 to 2025-10-11.
5. Emergency Response Plan.
6. Production Forecast Q4 2025.
7. Retrieve the gas production of WELL_E05 on 2025-09-01.
8. What was the production of WELL_B07 in September 2025?
9. Display the gas production of WELL_C03 on 2025-09-04.
10. Retrieve the latest production value for WELL_A12.
11. Display all NPT events at WELL_B07 during September 2025.
12. What were the drilling events at WELL_D02 from September 7–13, 2025?
13. List of NPT events at WELL_E05 on September 3, 2025?
14. Retrieve all drilling events at WELL_C03 between September 24–30, 2025.
15. What events occurred at WELL_F10 on September 10, 2025?
16. Display the flow rate trend of FLOW_A12 from 00:00–00:23 on September 1, 2025.
17. What was the flow rate with quality=0 at FLOW_A12 from September 11–12, 2025?
18. What was the comparison trend of the 5-minute average at FLOW_A12 at the beginning of September 1, 2025?
19. Was there a significant decrease in the flow rate between 00:05–00:10 and 00:15–00:20 on September 14, 2025, at FLOW_A12?
20. Retrieve the oil rate of OIL_D01 for the entire day of September 18, 2025 (00:00–23:59).
21. Display the trend of OIL_B07 from September 15–16, 2025.
22. Retrieve the oil rate of FLOW_A12 between 06:00–12:00 on 2025-09-01.
23. Compare the flow rate of OIL_D01 with OIL_D02 on September 18, 2025.
24. Retrieve the timeseries for FLOW_C03, FLOW_E05, FLOW_F10 on September 3, 2025, and then display the daily average.

Troubleshooting

1) /chat/stream cannot find documents (retrieved_chunks is empty)

• Check the hybrid endpoint:

curl "http://localhost:8080/rag/search_v2?q=Production%20Forecast%20Q4%202025"

Ensure the response is in the following format:

{
    "query":"...",
    "alpha":0.6,
    "count":10,
    "retrieved_chunks":[
        {"doc_id":"DOC-1005","title":"...","url":"...","page_no":3,"snippet":"...","score":0.123}
    ]
}

• If the structure is correct but the SSE is still empty, ensure that NormalizePlan is active and ragFn in chat_sse_handler.go reads retrieved_chunks[*].page_no (not page).

2) Decode error when MCP plans RAG

• Ensure that the consumer decodes the object with the retrieved_chunks field (not the slice directly).
• Example of correct decoding (Go):

var resp struct {
    RetrievedChunks []struct {
        DocID string `json:"doc_id"`
        ...
    } `json:"retrieved_chunks"`
}

3) LLM/Embeddings are not available

• The system still works: RAG hybrid + fallback extractive for answer_with_docs. Add OPENAI_API_KEY for more natural answers.

4) Observability

• Check /healthz, /metrics, and the logs in logs/.

🔧 Technical Details

Environment Configuration

Example .env variables:

# Database
DB_DSN="user:pass@tcp(localhost:3306)/mcp_oilgas?parseTime=true"

# JWT / Admin
ADMIN_JWT_SECRET="supersecret"

# LLM (optional)
OPENAI_API_KEY="sk-..."

# Planner
MCP_SCHEMAS_DIR="schemas/mcp"
PLAN_MAX_ROUTES=8

Without OPENAI_API_KEY, the system still functions (RAG hybrid & fallback extractive for answer_with_docs).

Important Endpoints

• Health & Metrics
  • GET /healthz, GET /readyz, GET /metrics
• Chat SSE
  • GET|POST /chat/stream?q=... (SSE events: plan, sources, delta, done, etc.)
• MCP Router (HTTP-internal)
  • POST /mcp/route (receives a plan or question to automatically select the appropriate tool)
• RAG Hybrid
  • GET|POST /rag/search_v2 → body {"query":"...","top_k":10,"alpha":0.6} (Response contains retrieved_chunks)
• Domain HTTP (mirror MCP)
  • /api/timeseries, /api/drilling-events, /api/po/status, /api/production, /api/work-orders/search, /api/npt/summarize, /api/po/vendor-compare, /api/answer-with-docs, etc.

Chat SSE Architecture and Flow

1. Planner (LLM JSON-mode) reads tools from schemas/mcp/*.schema.json.
2. NormalizePlan:
   • Routes kind:"rag" to rag_search_v2.
   • Corrects incorrect routes (e.g., top-amount PO).
   • Adds supporting RAG if relevant.
3. ExecuteRoutes:
   • Runs MCP tools in-process.
   • Runs RAG via the closure ragFn (hybrid /rag/search_v2 → fallback embeddings repo).
4. Synthesizer:
   • The LLM streams answers based on sources (or fallback extractive).

Plan Normalization (Planner)

The following aspects are automatically handled:

• RAG rewrite: All routes of kind:"rag" (including the old tool "rag") are changed to the rag_search_v2 tool with the body {query, top_k, alpha}.
• Detect anomalies → RAG: If the detect_anomalies payload is invalid and the question seems “document-based”, it is redirected to RAG.
• Top-N PO by amount: Detects phrases like “top/highest amount PO” and adds/corrects the route get_po_top_amount {limit, ...}.

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

Why Use Go (Golang)?

The MCP + RAG project is built with Go for several technical and business reasons that are highly relevant to the oil and gas industry:

• 🚀 High Performance & Efficiency: Compiled to a fast, stable binary with minimal overhead. Ideal for cross-domain queries with low latency.
• ⚡ Concurrency with Goroutines: Data from multiple sources (SCADA, IoT, ERP) can be fetched in parallel.
• 🛡️ Stable & Proven: Used in Kubernetes, Docker, Hashicorp Consul.
• 📦 Easy Deployment (Single Binary): No dependency hell, easy to migrate to on-premise/private cloud.
• 🔄 Supports Real-Time Streaming (SSE): Thousands of connections remain lightweight with net/http and goroutines.

Credits

Developed by Kukuh Tripamungkas Wicaksono (Kukuh TW) as a PoC architecture for MCP + RAG in the oil and gas domain.