# Comprehensive Guide to Monitoring Kafka Metrics with OpenTelemetry Collector
> Learn how to monitor Apache Kafka metrics effectively using OpenTelemetry Collector Contrib and OpenObserve. This detailed guide covers Kafka installation, setting up OpenTelemetry for real-time Kafka monitoring, configuring dashboards, and tracking key Kafka metrics like brokers, consumer groups, partition offsets, replicas, and message rates. Discover the benefits of monitoring Kafka with OpenObserve, including automated alerts, performance insights, and data integrity checks. Optimize your Kafka infrastructure with proactive monitoring and real-time observability-boost system reliability, detect issues early, and scale efficiently.
Source: https://openobserve.ai/blog/how-to-monitor-kafka-with-otel/
Published: 2025-03-12
Authors: Chaitanya Sistla
Category: How To
Tags: DevOps, Integrations, OpenTelemetry
---
Apache Kafka is a widely used distributed event streaming platform that requires robust monitoring for optimal performance and reliability. OpenTelemetry (OTel) provides an open-source observability framework, and the `otelcol-contrib` (OpenTelemetry Collector Contrib) is an extended version that includes various receivers and exporters. In this guide, we will set up Kafka monitoring using OpenTelemetry Collector Contrib and export the metrics to OpenObserve for visualization.
## Why Monitor Kafka?
Monitoring Kafka is essential for maintaining system health, ensuring data integrity, and optimizing performance. Here are key reasons why Kafka monitoring is critical:
- **Early Detection of Issues**: Monitoring helps detect broker failures, consumer lag, and under-replicated partitions before they impact performance.
- **Performance Optimization**: By tracking message throughput, consumer group activity, and partition distribution, teams can optimize resource utilization.
- **Data Integrity and Reliability**: Observing offsets, replication status, and in-sync replicas ensures that no data is lost or corrupted.
- **Capacity Planning**: By analyzing metrics over time, teams can anticipate scaling needs and prevent bottlenecks.
- **Regulatory Compliance and Auditing**: Some industries require detailed logging and monitoring to comply with data governance policies.
By integrating OpenTelemetry Collector Contrib, we gain deeper insights into Kafka performance, message throughput, consumer lag, and topic health.
## Step 1 - Install Kafka
To begin, install Kafka on your system:
For a more detailed setup, you can follow kafka documentation.
```bash
sudo apt update && sudo apt upgrade -y
sudo apt install openjdk-11-jdk -y
java -version
wget https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz
tar -xzf kafka_2.13-3.9.0.tgz
cd kafka_2.13-3.9.0
```
Start ZooKeeper and Kafka services:
```bash
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
```
### Verify Kafka Installation
To verify if Kafka is running correctly, execute the following command:
```bash
bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092
```
If Kafka is correctly set up, you should see a list of supported API versions.
## Step 2 - Create Test Kafka Topics
Create multiple test Kafka topics to simulate different use cases:
```bash
bin/kafka-topics.sh --create --topic quickstart-events-1 --partitions 3 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events-2 --partitions 10 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events-3 --partitions 16 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic openobserve --partitions 2 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
```
### List Available Kafka Topics
To confirm the topics were created successfully, run:
```bash
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
```
## Step 3 - Setup OpenTelemetry Collector Contrib
Download and install the OpenTelemetry Collector Contrib package:
```bash
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.1/otelcol-contrib_0.116.1_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.116.1_linux_amd64.deb
```
### Configure OpenTelemetry Collector Contrib
Edit the configuration file. For a complete walkthrough of the config file's structure, receivers, processors, and exporters, see our [OpenTelemetry Collector Configuration Guide](https://openobserve.ai/blog/opentelemetry-collector-configuration-guide/):
```bash
sudo vi /etc/otelcol-contrib/config.yaml
```
Add the following content to configure the Kafka metrics receiver and OpenObserve exporter:
```yaml
receivers:
kafkametrics:
brokers: localhost:9092
protocol_version: 2.0.0
scrapers:
- brokers
- topics
- consumers
- producer
- messages
exporters:
otlphttp/openobserve:
endpoint: OPENOBSERVE_ENDPOINT
headers:
Authorization: OPENOBSERVE_TOKEN
stream-name: OPENOBSERVE_STREAM
service:
pipelines:
metrics:
receivers: [kafkametrics]
exporters: [otlphttp/openobserve]
```
Replace OPENOBSERVE_ENDPOINT, OPENOBSERVE_TOKEN and OPENOBSERVE_STREAM with your OpenObserve credentials, which you can find in your OpenObserve dashboard under **Data Sources** -> **Custom** -> **Metrics** -> **Otel Collector**. 
### Start OpenTelemetry Collector Contrib
```bash
sudo systemctl start otelcol-contrib
sudo systemctl status otelcol-contrib
```
### Verify OpenTelemetry Metrics Collection
To confirm that OpenTelemetry Collector Contrib is successfully collecting Kafka metrics, check the logs:
```bash
journalctl -u otelcol-contrib --no-pager -n 50
```
If everything is configured correctly, you should see log messages indicating that metrics are being collected and exported. 
## Step 4 - Download and Use Prebuilt Dashboards
To visualize Kafka metrics, Download and configure a prebuilt dashboard supporting the following metrics:
### With and Without OpenObserve
| Aspect | Without OpenObserve | With OpenObserve |
| ------------------------- | -------------------------------------------- | ------------------------------------------------------- |
| **Issue Detection** | Manual, reactive troubleshooting | Automated alerts and proactive monitoring |
| **Performance Insights** | Limited visibility into Kafka internals | Comprehensive dashboards for real-time analysis |
| **Consumer Lag Tracking** | Requires custom scripts and manual queries | Prebuilt metrics and visualizations |
| **Scaling Decisions** | Trial and error based on rough estimates | Data-driven scaling strategies with historical insights |
| **Data Integrity Checks** | Hard to track in-sync replicas and offsets | Clear visibility into partitions, replicas, and offsets |
| **Audit and Compliance** | Requires external logging and complex setups | Centralized monitoring for compliance tracking |
## Conclusion
By following this guide, you have successfully installed and configured Kafka, set up OpenTelemetry Collector Contrib, and exported Kafka metrics to OpenObserve. Monitoring Kafka with OpenTelemetry helps in tracking performance issues, ensuring system reliability, and optimizing event streaming operations.
With real-time Kafka monitoring in place, you can:
- Detect consumer lag and take corrective actions.
- Ensure broker health and stability.
- Monitor message throughput for better resource allocation.
- Identify partition replica inconsistencies.
Stay observant of Kafka metrics to maintain a healthy event-driven architecture and enhance your distributed streaming ecosystem.
> #### Get Started with OpenObserve Today!
>
> Sign up for a [14 day trial](https://cloud.openobserve.ai).
>
> Check out our GitHub repository for self-hosting and contribution opportunities.