Kafka MCP Server

The Kafka MCP Server is an Apache Kafka Model Context Protocol (MCP) server implemented in Go. Through a standardized interface, it enables LLM models to perform Kafka operations. It provides functions such as producing/consuming messages, managing topics and consumer groups, and monitoring cluster health. It supports multiple security authentication methods and can be integrated with AI applications such as Cursor and Claude.

Developer tools Database #Kafka Tool #AI Integration #Message Queue #Protocol Service .Go

rating : 2.5 points

downloads : 5.1K

update time : 2025-04-29

Open Site

What is the Kafka MCP Server?

The Kafka MCP Server is a tool for interacting with Apache Kafka. Through the standardized Model Context Protocol (MCP), it enables Large Language Models (LLMs) to easily perform common Kafka operations, such as message production and consumption, topic management, and cluster health checks.

How to use the Kafka MCP Server?

Simply install the server and configure the environment variables to start using it. By integrating it into your AI applications, you can easily call Kafka operations, such as sending messages or viewing the status of consumer groups.

Use cases

Suitable for scenarios that require automated Kafka operations, such as data stream monitoring, real - time log analysis, and large - scale message processing.

Main Features

Produce Messages

Send messages to a specified Kafka topic.

Consume Messages

Pull messages from one or more topics.

List Brokers

Display the addresses of currently connected Kafka brokers.

Describe Topic

Get detailed metadata of a specific topic.

List Consumer Groups

Display active consumer groups.

Describe Consumer Group

Show the status, members, and partition offsets of a specific consumer group.

Describe Configurations

Retrieve the configuration items of a specific resource (topic or broker).

Cluster Overview

Provide an overall health summary of the Kafka cluster.

Advantages

Simplify Kafka operations through the standard MCP protocol.

Support multiple authentication mechanisms (SASL and TLS).

Easy to integrate with other AI tools.

Comprehensive error handling and feedback.

Limitations

Appropriate permissions are required to access the Kafka cluster.

Some advanced features may require additional configuration.

Systems with high low - latency requirements may need performance optimization.

How to Use

Install the Kafka MCP Server

Clone the project code and build the server locally.

Configure Environment Variables

Set the Kafka server address, client ID, and other necessary options.

Start the Server

Run the server to listen for MCP requests.

Usage Examples

Send an Order Update Message

Demonstrate how to send a message with an order ID to a Kafka topic.

List Consumer Groups

Show how to list all active consumer groups.

Frequently Asked Questions

How to enable TLS encryption in a production environment?

What if my consumer group is lagging?

Related Resources

Official Documentation

The GitHub repository of the Kafka MCP Server.

Installation Guide

Detailed installation and configuration instructions.

Open - Source License

MIT License.

Installation

Copy the following command to your Client for configuration

{
  "mcpServers": {
    "kafka": {
      "command": "kafka-mcp-server",
      "env": {
        "KAFKA_BROKERS": "localhost:9092",
        "KAFKA_CLIENT_ID": "kafka-mcp-server",
        "MCP_TRANSPORT": "stdio",
        "KAFKA_SASL_MECHANISM": "",
        "KAFKA_SASL_USER": "",
        "KAFKA_SASL_PASSWORD": "",
        "KAFKA_TLS_ENABLE": "false",
        "KAFKA_TLS_INSECURE_SKIP_VERIFY": "false"
      }
    }
  }
}

{
  "mcpServers": {
    "kafka": {
      "command": "kafka-mcp-server",
      "env": {
        "KAFKA_BROKERS": "kafka-broker-1:9092,kafka-broker-2:9092",
        "KAFKA_CLIENT_ID": "kafka-mcp-server",
        "MCP_TRANSPORT": "stdio",
        "KAFKA_SASL_MECHANISM": "scram-sha-512",
        "KAFKA_SASL_USER": "kafka-user",
        "KAFKA_SASL_PASSWORD": "kafka-password",
        "KAFKA_TLS_ENABLE": "true",
        "KAFKA_TLS_INSECURE_SKIP_VERIFY": "false"
      }
    }
  }
}

{
  "mcpServers": {
    "kafka": {
      "command": "docker",
      "args": [
        "run", 
        "--rm", 
        "-i", 
        "-e", "KAFKA_BROKERS=kafka-broker:9092", 
        "-e", "KAFKA_CLIENT_ID=kafka-mcp-server",
        "-e", "MCP_TRANSPORT=stdio",
        "-e", "KAFKA_SASL_MECHANISM=", 
        "-e", "KAFKA_SASL_USER=", 
        "-e", "KAFKA_SASL_PASSWORD=", 
        "-e", "KAFKA_TLS_ENABLE=false",
        "ghcr.io/tuannvm/kafka-mcp-server:latest"
      ],
      "env": {}
    }
  }
}

{
  "mcpServers": {
    "kafka": {
      "command": "kafka-mcp-server",
      "args": [],
      "env": {
        "KAFKA_BROKERS": "localhost:9092",
        "KAFKA_CLIENT_ID": "kafka-mcp-server",
        "MCP_TRANSPORT": "stdio",
        "KAFKA_SASL_MECHANISM": "",
        "KAFKA_SASL_USER": "",
        "KAFKA_SASL_PASSWORD": "",
        "KAFKA_TLS_ENABLE": "false"
      }
    }
  }
}

{
  "mcpServers": {
    "kafka-http": {
      "url": "http://localhost:9097/sse"
    }
  }
}

Note: Your key is sensitive information, do not share it with anyone.

🚀 Kafka Distributed Computing Platform (MCP) User Guide

Kafka Distributed Computing Platform (Kafka MCP for short) is a powerful toolset designed to simplify interactions with Apache Kafka clusters. This platform offers a variety of command - line tools and REST APIs for operations such as consumer group description, topic offset management, and producer/consumer logging.

🚀 Quick Start

Let's take a quick look at how to use the main tools provided by Kafka MCP.

✨ Features

Consumer Group Management

Consumer Group Description: Retrieve the consumer ID of a specified consumer group and its associated client ID.
Offset Query: Check the current consumption position of a specific topic and partition within a consumer group.
Reset Offset: Allow manual adjustment of the consumer group's offset for precise consumption control.

Topic Management

Topic Creation: Quickly create new topics by defining parameters such as the number of partitions and replication factor.
Topic Deletion: Safely delete unnecessary topics to free up cluster resources.
ISR Monitoring: Real - time view of the _isr (In - Sync Replicas) status of topic partitions to ensure data consistency.

Message Tracking

Message Consumption: Consume messages from a specified topic in real - time, and support flexible retrieval by time range, batch size, etc.
Production Log Generation: Conveniently send test or debugging messages to a specified topic.
Offset Adjustment: Manually adjust the consumer offset when necessary for precise stream processing.

📦 Installation

Environment Requirements

Java 8 or higher
Kafka version 2.13+

Installation Steps

Download and extract the Kafka MCP distribution package:

wget https://example.com/kafka-mcp.tar.gz
tar -xzvf kafka-mcp.tar.gz
cd kafka-mcp

Configure environment variables (optional):

export PATH=$PATH:/path/to/kafka-mcp/bin

Start the MCP service (if using the REST API version):
```
./start_mcp_server.sh
```

💻 Usage Examples

Basic Usage

Tool Overview

Here are some basic commands to interact with Kafka MCP:

1. List all consumer groups

# Get a list of all Kafka consumer groups
list_consumer_groups --bootstrap-server <broker_address>

2. Describe a specific consumer group

# View detailed information about a specified consumer group, including consumer ID and offset
describe_consumer_group --group-id <consumer_group_id> \
                       --bootstrap-server <broker_address> \
                       --include-protocol-info \
                       --include-consumer-info \
                       --include-offsets

3. Get the topic list

# List all topics in the Kafka cluster
list_topics --bootstrap-server <broker_address>

4. Describe topic details

# View the partition assignment, replicas, and ISR status of a specified topic
describe_topic --topic-name <topic_name> \
               --bootstrap-server <broker_address>

5. Manage topic offsets

# Consume offsets from all partitions of a specific topic
consume_offsets --group-id <consumer_group_id> \
               --topic-name <topic_name> \
               --bootstrap-server <broker_address>

# Provide the current offset of a specified partition
seek_consumer_partition --group-id <consumer_group_id> \
                       --topic-name <topic_name> \
                       --partition <partition_number> \
                       --offset <target_offset> \
                       --bootstrap-server <broker_address>

6. Monitor producer and consumer logs

# Consume the latest messages from a specified topic
consume_logs --topic-name <topic_name> \
             --bootstrap-server <broker_address>

# Generate simulated producer logs
produce_logs --topic-name <topic_name> \
             --message "your_message_here" \
             --bootstrap-server <broker_address>

Advanced Usage

Use Cases

Here are some practical use cases of Kafka MCP:

Case 1: Consumer group offset query

# View the offset of a specified consumer group
consume_offsets --group-id my_consumer_group \
               --bootstrap-server localhost:9092

Case 2: Topic description

# Get detailed information about a specific topic
describe_topic --topic-name user_activity \
               --bootstrap-server localhost:9092

📚 Documentation

Configuration Reference

System Configuration

Property	Details
`--bootstrap-server`	Specify the IP and port of the Kafka service in the format `host:port`.
`--group-id`	Used to identify a group of consumer instances.
`--topic-name`	The specific topic for the operation.

Environment Variables

# Set the default broker address
export KAFKA_BOOTSTRAP_SERVER=localhost:9092

🔧 Technical Details

Security Precautions

Authentication and Authorization: In a production environment, it is recommended to enable SASL/PLAINTEXT or TLS encrypted communication to ensure the security of API calls.
Access Control: Limit the permissions of different users or applications through Kafka's ACL (Access Control List) mechanism.
Log Monitoring: Regularly review operation logs to identify abnormal behavior and respond promptly.