Docker Container Monitoring with OpenTelemetry and OpenObserve

Docker containers have revolutionized application deployment by providing consistent, isolated environments for running applications. However, as containerized applications grow in complexity, monitoring these containers becomes crucial for ensuring optimal performance, resource utilization, and troubleshooting issues.

In this comprehensive guide, we'll explore how to monitor Docker container metrics using OpenTelemetry and visualize them in OpenObserve. We'll leverage the Docker Stats Receiver from the OpenTelemetry Collector Contrib distribution to collect detailed metrics from your Docker containers without modifying your applications.

Understanding Docker Container Metrics

Docker containers share the host system's resources, making it essential to monitor their resource consumption. Key metrics to track include:

• CPU usage: How much processing power each container is consuming
• Memory consumption: RAM usage and limits for each container
• Network I/O: Data transferred in and out of containers
• Block I/O: Disk read/write operations
• Process information: Number of processes running in containers

Monitoring these metrics helps identify resource bottlenecks, optimize container performance, and ensure efficient resource allocation across your containerized applications.

Docker Stats Receiver

The Docker Stats Receiver is a component of the OpenTelemetry Collector Contrib distribution that queries the local Docker daemon's container stats API for all running containers at a configured interval. It collects comprehensive metrics about container resource usage.

Note: The Docker Stats Receiver requires Docker API version 1.25+ and is primarily supported on Linux systems. While it can work on macOS with Docker Desktop, you might need to adjust the Docker socket path.

Default Metrics

The Docker Stats Receiver collects several metrics by default, including:

Metric	Description	Metric Type	Value Type
container.cpu.usage.total	Total CPU time consumed	Sum	Int
container.cpu.usage.kernelmode	CPU time in kernel mode	Sum	Int
container.cpu.usage.usermode	CPU time in user mode	Sum	Int
container.cpu.utilization	Percentage of CPU used by the container	Gauge	Double
container.memory.usage.total	Memory usage of the container (excluding cache)	Sum	Int
container.memory.usage.limit	Memory limit of the container	Sum	Int
container.memory.percent	Percentage of memory used	Gauge	Double
container.network.io.usage.rx_bytes	Bytes received by the container	Sum	Int
container.network.io.usage.tx_bytes	Bytes sent by the container	Sum	Int
container.network.io.usage.rx_dropped	Incoming packets dropped	Sum	Int
container.network.io.usage.tx_dropped	Outgoing packets dropped	Sum	Int
container.blockio.io_service_bytes_recursive	Number of bytes transferred to/from disk	Sum	Int

Additional metrics that can be enabled include:

Metric	Description	Enabled by Default
container.cpu.usage.percpu	CPU usage per CPU core	No
container.pids.count	Number of processes running in the container	No
container.uptime	Container uptime in seconds	No
container.restarts	Number of container restarts	No
container.network.io.usage.rx_errors	Receive errors	No
container.network.io.usage.tx_errors	Transmit errors	No
container.network.io.usage.rx_packets	Packets received	No
container.network.io.usage.tx_packets	Packets transmitted	No

Getting Started

Prerequisites

Before we begin setting up Docker container monitoring, ensure you have:

1. Docker installed and running on your system

   • Docker Desktop for Linux
   • Docker Desktop for Mac (macOS)
   • Docker Desktop for Windows

2. Access to an OpenObserve instance (cloud or self-hosted)

3. Basic understanding of OpenTelemetry concepts

Verify Docker Containers

First, let's verify that you have Docker containers running on your system that we can monitor:

docker ps

This command will list all running containers. If you don't have any containers running, you can start some with:

# Start an Nginx web server
docker run -d --name nginx-test -p 8080:80 nginx:latest

# Start a MySQL database
docker run -d --name mysql-test -e MYSQL_ROOT_PASSWORD=mysecretpassword -p 3306:3306 mysql:latest

Install the OpenTelemetry Collector

We'll use the OpenTelemetry Collector Contrib distribution as it includes the Docker Stats Receiver.

1. Visit the OpenTelemetry Collector Contrib Releases page.

2. Download the latest release for your machine. For macOS with Apple Silicon, use:

curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.0/otelcol-contrib_0.116.0_darwin_arm64.tar.gz

3. Extract the downloaded file:

tar -xvf otelcol-contrib_0.116.0_darwin_arm64.tar.gz

4. Move the binary to a directory in your PATH:

sudo mv otelcol-contrib /usr/local/bin/

5. Verify the installation:

otelcol-contrib --version

Find Your Docker Socket Path

The Docker Stats Receiver needs to connect to the Docker daemon through a socket. The socket path varies depending on your operating system:

For Linux:

ls -la /var/run/docker.sock

For macOS with Docker Desktop:

# Check the standard Linux path (sometimes works with Docker Desktop)
ls -la /var/run/docker.sock

# If that doesn't exist, check these common locations
ls -la ~/Library/Containers/com.docker.docker/Data/docker.sock
ls -la ~/.docker/run/docker.sock

Note: On macOS, /var/run/docker.sock is often a symbolic link to the actual socket location. You can follow this link to find the actual path.

Configure the OpenTelemetry Collector

Create a configuration file named config.yaml with the following content:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  docker_stats:
    endpoint: unix:///var/run/docker.sock  # Adjust this path based on your system
    collection_interval: 30s
    timeout: 10s
    api_version: "1.25"  # Note the quotes around the version number
    metrics:
      container.uptime:
        enabled: true
      container.restarts:
        enabled: true
      container.network.io.usage.rx_errors:
        enabled: true
      container.network.io.usage.tx_errors:
        enabled: true
      container.network.io.usage.rx_packets:
        enabled: true
      container.network.io.usage.tx_packets:
        enabled: true
      container.cpu.usage.percpu:
        enabled: true
      container.pids.count:
        enabled: true

processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["dns", "os"]
  resourcedetection/docker:
    detectors: [env, docker]
    timeout: 2s
    override: false

exporters:
  otlphttp/openobserve:
    endpoint: http://localhost:5080/api/default  # Replace with your OpenObserve endpoint
    headers:
      Authorization: Basic YOUR_AUTH_TOKEN  # Replace with your OpenObserve auth token
      stream-name: default
  debug:
    verbosity: detailed

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    metrics:
      receivers: [otlp, docker_stats]
      processors: [resourcedetection/docker, resourcedetection/system, batch]
      exporters: [otlphttp/openobserve, debug]

1. Replace unix:///var/run/docker.sock with your actual Docker socket path.

2. Replace YOUR_API_ENDPOINT and YOUR_AUTH_TOKEN with your OpenObserve credentials, which you can find in your OpenObserve dashboard under Data Sources -> Custom -> Metrics -> Otel Collector.

Run the OpenTelemetry Collector

To run the OpenTelemetry Collector with the specified configuration, execute the following command:

otelcol-contrib --config config.yaml

You should see logs indicating that the collector has started and is collecting metrics from Docker containers. The metrics will begin flowing into your OpenObserve instance within a few seconds.

Monitoring Docker Metrics with OpenObserve

Exploring Docker Metrics

Navigate to the Metrics section in your OpenObserve dashboard to see the incoming Docker container metrics. You'll be able to see metrics like CPU utilization, memory usage, network I/O, and disk operations for all your monitored containers.

Pre-built Docker Monitoring Dashboard

To make monitoring easier, we've created a comprehensive Docker Container Metrics dashboard that you can import directly into your OpenObserve instance. This dashboard provides visualizations for all the key Docker metrics.

You can download this pre-built dashboard from our community dashboards repository and import it into your OpenObserve instance for instant visibility into your Docker container performance without having to build visualizations from scratch.

Advanced Configuration Options

Filtering Containers

You might not want to monitor all containers. The Docker Stats Receiver allows you to exclude specific containers using the excluded_images configuration:

docker_stats:
  # ... other settings
  excluded_images:
    - undesired-container
    - /.*undesired.*/
    - another-*-container

This configuration:

• Excludes containers with the exact name "undesired-container"
• Excludes containers matching the regex ".undesired."
• Excludes containers matching the glob pattern "another-*-container"

Mapping Container Labels to Metric Labels

You can map Docker container labels to metric labels for better organization:

docker_stats:
  # ... other settings
  container_labels_to_metric_labels:
    com.docker.compose.service: service_name
    com.docker.compose.project: project_name
  env_vars_to_metric_labels:
    APP_ENV: environment

This maps:

• The Docker label com.docker.compose.service to a metric label called service_name
• The Docker label com.docker.compose.project to a metric label called project_name
• The environment variable APP_ENV to a metric label called environment

Troubleshooting

If you encounter issues with your Docker metrics collection, here are some quick fixes:

1. Docker Socket Access: Ensure the user running the collector has read permissions for the Docker socket.

2. API Version Compatibility: Verify your Docker API version with docker version --format '{{.Server.APIVersion}}' and adjust your configuration accordingly.

3. macOS Socket Path: On macOS, check the actual socket path with ls -la /var/run/docker.sock and follow any symbolic links.

4. Collector Logs: Examine the collector logs for error messages that might indicate configuration issues.

Conclusion and Next Steps

Monitoring Docker containers with OpenTelemetry and OpenObserve provides essential visibility into your containerized applications. By collecting metrics on CPU usage, memory consumption, network activity, and disk operations, you can quickly identify performance bottlenecks and resource constraints.

This guide has shown you how to set up a comprehensive Docker monitoring solution without modifying your applications or adding instrumentation. The Docker Stats Receiver from the OpenTelemetry Collector makes it easy to gather detailed performance data from all your containers.

Now that you have your Docker metrics flowing into OpenObserve, consider these next steps to enhance your monitoring capabilities:

1. Set Up Real-Time Alerts: Configure alerts in OpenObserve to notify you when containers exceed resource thresholds or experience issues. For example, you can create alerts for high CPU utilization or memory consumption.

2. Correlate with Application Metrics: Combine container metrics with application-level telemetry for a complete view of your system's performance.

3. Implement Log Collection: Add log collection to your monitoring setup to correlate metrics with log events for better troubleshooting.

For more information on Docker monitoring best practices, check out the Docker Stats Receiver Documentation and the OpenObserve Documentation.

Happy monitoring! 🚀