Kafka MCP Server

The Kafka MCP Server is an Apache Kafka Model Context Protocol (MCP) server implemented in Go. Through a standardized interface, it enables LLM models to perform Kafka operations. It provides functions such as producing/consuming messages, managing topics and consumer groups, and monitoring cluster health. It supports multiple security authentication methods and can be integrated with AI applications such as Cursor and Claude.

Developer tools Database #Kafka Tool #AI Integration #Message Queue #Protocol Service .Go

rating : 2.5 points

downloads : 33

update time : 2025-04-29

What is the Kafka MCP Server?

The Kafka MCP Server is a tool for interacting with Apache Kafka. Through the standardized Model Context Protocol (MCP), it enables Large Language Models (LLMs) to easily perform common Kafka operations, such as message production and consumption, topic management, and cluster health checks.

How to use the Kafka MCP Server?

Simply install the server and configure the environment variables to start using it. By integrating it into your AI applications, you can easily call Kafka operations, such as sending messages or viewing the status of consumer groups.

Use cases

Suitable for scenarios that require automated Kafka operations, such as data stream monitoring, real - time log analysis, and large - scale message processing.

Main Features

Produce MessagesSend messages to a specified Kafka topic.

Consume MessagesPull messages from one or more topics.

List BrokersDisplay the addresses of currently connected Kafka brokers.

Describe TopicGet detailed metadata of a specific topic.

List Consumer GroupsDisplay active consumer groups.

Describe Consumer GroupShow the status, members, and partition offsets of a specific consumer group.

Describe ConfigurationsRetrieve the configuration items of a specific resource (topic or broker).

Cluster OverviewProvide an overall health summary of the Kafka cluster.

Advantages and Limitations

Advantages

Simplify Kafka operations through the standard MCP protocol.

Support multiple authentication mechanisms (SASL and TLS).

Easy to integrate with other AI tools.

Comprehensive error handling and feedback.

Limitations

Appropriate permissions are required to access the Kafka cluster.

Some advanced features may require additional configuration.

Systems with high low - latency requirements may need performance optimization.

How to Use

Install the Kafka MCP Server

Clone the project code and build the server locally.

Configure Environment Variables

Set the Kafka server address, client ID, and other necessary options.

Start the Server

Run the server to listen for MCP requests.

Usage Examples

Send an Order Update MessageDemonstrate how to send a message with an order ID to a Kafka topic.

List Consumer GroupsShow how to list all active consumer groups.

Frequently Asked Questions

How to enable TLS encryption in a production environment?

What if my consumer group is lagging?

Related Resources

Official Documentation

The GitHub repository of the Kafka MCP Server.

Installation Guide

Detailed installation and configuration instructions.

Open - Source License

MIT License.

Installation

Copy the following command to your Client for configuration

{
  "mcpServers": {
    "kafka": {
      "command": "kafka-mcp-server",
      "env": {
        "KAFKA_BROKERS": "localhost:9092",
        "KAFKA_CLIENT_ID": "kafka-mcp-server",
        "MCP_TRANSPORT": "stdio",
        "KAFKA_SASL_MECHANISM": "",
        "KAFKA_SASL_USER": "",
        "KAFKA_SASL_PASSWORD": "",
        "KAFKA_TLS_ENABLE": "false",
        "KAFKA_TLS_INSECURE_SKIP_VERIFY": "false"
      }
    }
  }
}

{
  "mcpServers": {
    "kafka": {
      "command": "kafka-mcp-server",
      "env": {
        "KAFKA_BROKERS": "kafka-broker-1:9092,kafka-broker-2:9092",
        "KAFKA_CLIENT_ID": "kafka-mcp-server",
        "MCP_TRANSPORT": "stdio",
        "KAFKA_SASL_MECHANISM": "scram-sha-512",
        "KAFKA_SASL_USER": "kafka-user",
        "KAFKA_SASL_PASSWORD": "kafka-password",
        "KAFKA_TLS_ENABLE": "true",
        "KAFKA_TLS_INSECURE_SKIP_VERIFY": "false"
      }
    }
  }
}

{
  "mcpServers": {
    "kafka": {
      "command": "docker",
      "args": [
        "run", 
        "--rm", 
        "-i", 
        "-e", "KAFKA_BROKERS=kafka-broker:9092", 
        "-e", "KAFKA_CLIENT_ID=kafka-mcp-server",
        "-e", "MCP_TRANSPORT=stdio",
        "-e", "KAFKA_SASL_MECHANISM=", 
        "-e", "KAFKA_SASL_USER=", 
        "-e", "KAFKA_SASL_PASSWORD=", 
        "-e", "KAFKA_TLS_ENABLE=false",
        "ghcr.io/tuannvm/kafka-mcp-server:latest"
      ],
      "env": {}
    }
  }
}

{
  "mcpServers": {
    "kafka": {
      "command": "kafka-mcp-server",
      "args": [],
      "env": {
        "KAFKA_BROKERS": "localhost:9092",
        "KAFKA_CLIENT_ID": "kafka-mcp-server",
        "MCP_TRANSPORT": "stdio",
        "KAFKA_SASL_MECHANISM": "",
        "KAFKA_SASL_USER": "",
        "KAFKA_SASL_PASSWORD": "",
        "KAFKA_TLS_ENABLE": "false"
      }
    }
  }
}

{
  "mcpServers": {
    "kafka-http": {
      "url": "http://localhost:9097/sse"
    }
  }
}

Note: Your key is sensitive information, do not share it with anyone.

🚀 Kafka Distributed Computing Platform (MCP) User Guide

Kafka Distributed Computing Platform (MCP) simplifies interactions with Apache Kafka clusters, offering a range of tools and APIs for efficient management.

🚀 Quick Start

Kafka Distributed Computing Platform (Kafka MCP for short) is a powerful toolset designed to simplify interactions with Apache Kafka clusters. The platform provides various command - line tools and REST APIs for operations such as consumer group description, topic offset management, and producer/consumer logging.

✨ Features

Consumer Group Management

Consumer Group Description: Retrieve the consumer ID and its associated client ID of a specified consumer group.
Offset Query: Check the current consumption position of a specific topic and partition within a consumer group.
Offset Reset: Allow manual adjustment of the consumer group's offset for precise consumption control.

Topic Management

Topic Creation: Quickly create new topics by defining parameters such as the number of partitions and replication factor.
Topic Deletion: Safely delete unnecessary topics to free up cluster resources.
ISR Monitoring: Real - time monitoring of the _isr (In - Sync Replicas) status of topic partitions to ensure data consistency.

Message Tracking

Message Consumption: Consume messages from a specified topic in real - time, with flexible retrieval options by time range, batch size, etc.
Production Log Generation: Conveniently send test or debugging messages to a specified topic.
Offset Adjustment: Manually adjust the consumer offset when necessary for precise stream processing.

📦 Installation

Environment Requirements

Java 8 or higher
Kafka version 2.13+

Installation Steps

Download and extract the Kafka MCP distribution package:

wget https://example.com/kafka-mcp.tar.gz
tar -xzvf kafka-mcp.tar.gz
cd kafka-mcp

Configure environment variables (optional):

export PATH=$PATH:/path/to/kafka-mcp/bin

Start the MCP service (if using the REST API version):
```
./start_mcp_server.sh
```

💻 Usage Examples

Basic Usage

Tool Overview

List all consumer groups

# Get a list of all Kafka consumer groups
list_consumer_groups --bootstrap-server <broker_address>

Describe a specific consumer group

# View detailed information about a specified consumer group, including consumer ID and offset
describe_consumer_group --group-id <consumer_group_id> \
                       --bootstrap-server <broker_address> \
                       --include-protocol-info \
                       --include-consumer-info \
                       --include-offsets

Get a list of topics

# List all topics in the Kafka cluster
list_topics --bootstrap-server <broker_address>

Describe topic details

# View the partition assignment, replicas, and ISR status of a specified topic
describe_topic --topic-name <topic_name> \
               --bootstrap-server <broker_address>

Manage topic offsets

# Consume offsets of all partitions of a specific topic
consume_offsets --group-id <consumer_group_id> \
               --topic-name <topic_name> \
               --bootstrap-server <broker_address>

# Provide the current offset of a specified partition
seek_consumer_partition --group-id <consumer_group_id> \
                       --topic-name <topic_name> \
                       --partition <partition_number> \
                       --offset <target_offset> \
                       --bootstrap-server <broker_address>

Monitor producer and consumer logs

# Consume the latest messages from a specified topic
consume_logs --topic-name <topic_name> \
             --bootstrap-server <broker_address>

# Generate simulated producer logs
produce_logs --topic-name <topic_name> \
             --message "your_message_here" \
             --bootstrap-server <broker_address>

Advanced Usage

Use Case 1: Consumer Group Offset Query

# View the offset of a specified consumer group
consume_offsets --group-id my_consumer_group \
               --bootstrap-server localhost:9092

Use Case 2: Topic Description

# Get detailed information about a specific topic
describe_topic --topic-name user_activity \
               --bootstrap-server localhost:9092

📚 Documentation

Configuration Reference

System Configuration

Property	Details
`--bootstrap-server`	Specify the IP and port of the Kafka service in the format `host:port`.
`--group-id`	Used to identify a group of consumer instances.
`--topic-name`	The specific topic for the operation.

Environment Variables

# Set the default broker address
export KAFKA_BOOTSTRAP_SERVER=localhost:9092

🔧 Technical Details

Security Precautions

Authentication and Authorization: In a production environment, it is recommended to enable SASL/PLAINTEXT or TLS encrypted communication to ensure the security of API calls.
Access Control: Use Kafka's ACL (Access Control List) mechanism to limit the permission scope of different users or applications.
Log Monitoring: Regularly review operation logs to identify abnormal behavior and respond promptly.