Comprehensive Guide to Monitoring Kafka Metrics with OpenTelemetry Collector

Apache Kafka is a widely used distributed event streaming platform that requires robust monitoring for optimal performance and reliability. OpenTelemetry (OTel) provides an open-source observability framework, and the otelcol-contrib (OpenTelemetry Collector Contrib) is an extended version that includes various receivers and exporters. In this guide, we will set up Kafka monitoring using OpenTelemetry Collector Contrib and export the metrics to OpenObserve for visualization.

Why Monitor Kafka?

Monitoring Kafka is essential for maintaining system health, ensuring data integrity, and optimizing performance. Here are key reasons why Kafka monitoring is critical:

• Early Detection of Issues: Monitoring helps detect broker failures, consumer lag, and under-replicated partitions before they impact performance.
• Performance Optimization: By tracking message throughput, consumer group activity, and partition distribution, teams can optimize resource utilization.
• Data Integrity and Reliability: Observing offsets, replication status, and in-sync replicas ensures that no data is lost or corrupted.
• Capacity Planning: By analyzing metrics over time, teams can anticipate scaling needs and prevent bottlenecks.
• Regulatory Compliance and Auditing: Some industries require detailed logging and monitoring to comply with data governance policies.

By integrating OpenTelemetry Collector Contrib, we gain deeper insights into Kafka performance, message throughput, consumer lag, and topic health.

Step 1 - Install Kafka

To begin, install Kafka on your system:

For a more detailed setup, you can follow kafka documentation.

sudo apt update && sudo apt upgrade -y
sudo apt install openjdk-11-jdk -y
java -version
wget https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz
tar -xzf kafka_2.13-3.9.0.tgz
cd kafka_2.13-3.9.0

Start ZooKeeper and Kafka services:

bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &

Verify Kafka Installation

To verify if Kafka is running correctly, execute the following command:

bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092

If Kafka is correctly set up, you should see a list of supported API versions.

Step 2 - Create Test Kafka Topics

Create multiple test Kafka topics to simulate different use cases:

bin/kafka-topics.sh --create --topic quickstart-events-1 --partitions 3 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events-2 --partitions 10 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events-3 --partitions 16 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic openobserve --partitions 2 --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

List Available Kafka Topics

To confirm the topics were created successfully, run:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 3 - Setup OpenTelemetry Collector Contrib

Download and install the OpenTelemetry Collector Contrib package:

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.1/otelcol-contrib_0.116.1_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.116.1_linux_amd64.deb

Configure OpenTelemetry Collector Contrib

Edit the configuration file:

sudo vi /etc/otelcol-contrib/config.yaml

Add the following content to configure the Kafka metrics receiver and OpenObserve exporter:

receivers:
  kafkametrics:
    brokers: localhost:9092
    protocol_version: 2.0.0
    scrapers:
      - brokers
      - topics
      - consumers
      - producer
      - messages

exporters:
  otlphttp/openobserve:
    endpoint: OPENOBSERVE_ENDPOINT
    headers:
      Authorization: OPENOBSERVE_TOKEN
      stream-name: OPENOBSERVE_STREAM

service:
  pipelines:
    metrics:
      receivers: [kafkametrics]
      exporters: [otlphttp/openobserve]

Replace OPENOBSERVE_ENDPOINT, OPENOBSERVE_TOKEN and OPENOBSERVE_STREAM with your OpenObserve credentials, which you can find in your OpenObserve dashboard under Data Sources -> Custom -> Metrics -> Otel Collector.

Start OpenTelemetry Collector Contrib

sudo systemctl start otelcol-contrib
sudo systemctl status otelcol-contrib

Verify OpenTelemetry Metrics Collection

To confirm that OpenTelemetry Collector Contrib is successfully collecting Kafka metrics, check the logs:

journalctl -u otelcol-contrib --no-pager -n 50

If everything is configured correctly, you should see log messages indicating that metrics are being collected and exported.

Step 4 - Download and Use Prebuilt Dashboards

To visualize Kafka metrics, Download and configure a prebuilt dashboard supporting the following metrics:

With and Without OpenObserve

Aspect	Without OpenObserve	With OpenObserve
Issue Detection	Manual, reactive troubleshooting	Automated alerts and proactive monitoring
Performance Insights	Limited visibility into Kafka internals	Comprehensive dashboards for real-time analysis
Consumer Lag Tracking	Requires custom scripts and manual queries	Prebuilt metrics and visualizations
Scaling Decisions	Trial and error based on rough estimates	Data-driven scaling strategies with historical insights
Data Integrity Checks	Hard to track in-sync replicas and offsets	Clear visibility into partitions, replicas, and offsets
Audit and Compliance	Requires external logging and complex setups	Centralized monitoring for compliance tracking

Conclusion

By following this guide, you have successfully installed and configured Kafka, set up OpenTelemetry Collector Contrib, and exported Kafka metrics to OpenObserve. Monitoring Kafka with OpenTelemetry helps in tracking performance issues, ensuring system reliability, and optimizing event streaming operations.

With real-time Kafka monitoring in place, you can:

• Detect consumer lag and take corrective actions.
• Ensure broker health and stability.
• Monitor message throughput for better resource allocation.
• Identify partition replica inconsistencies.

Stay observant of Kafka metrics to maintain a healthy event-driven architecture and enhance your distributed streaming ecosystem.

Get Started with OpenObserve Today!

Sign up for a 14 day trial.

Check out our GitHub repository for self-hosting and contribution opportunities.