Complete Linux Monitoring with OpenObserve: Event Logs, Host Metrics, and More

Linux powers everything from web servers and databases to containers and IoT devices. While its stability is legendary, effective monitoring remains essential for maintaining performance, security, and reliability. Linux generates a wealth of telemetry data through system logs and performance metrics that—when properly collected and analyzed—provides invaluable insights into system health and behavior.

In this comprehensive guide, we'll explore how to implement complete Linux monitoring using OpenObserve. We'll cover two powerful approaches: the streamlined OpenObserve agent for quick deployment and the customizable OpenTelemetry Collector for advanced configurations. Whether you're troubleshooting system issues, tracking security events, or monitoring performance metrics, this guide will help you gain full visibility into your Linux environment.

Understanding Linux Monitoring Data

Linux systems generate several types of monitoring data that are essential for maintaining system health and security:

Linux System Logs

Linux system logs are records of system activities, errors, warnings, and informational messages generated by the Linux kernel, applications, and services. Think of them as your system's diary – they note everything that happens behind the scenes.

The most important log files and their locations include:

• /var/log/syslog or /var/log/messages: General system events (the primary system log)
• /var/log/auth.log or /var/log/secure: Authentication and security events
• /var/log/kern.log: Kernel messages and hardware-related events
• /var/log/dmesg: Boot-time messages
• /var/log/cron: Scheduled task (cron job) logs
• /var/log/faillog: Failed login attempts
• /var/log/[application]: Application-specific logs (e.g., Apache, MySQL)

Note: Log file locations may vary slightly between different Linux distributions. For example, CentOS and RHEL use /var/log/messages while Ubuntu and Debian use /var/log/syslog.

Running ls -la /var/log shows you all available logs:

Each log entry typically contains:

• Timestamp: When the event occurred
• Hostname: Which system generated the event
• Application/service name: What generated the event
• Process ID (PID): Which process wrote the log
• Message: The actual event details

Here's a sample log entry from syslog:

Apr 15 14:23:01 ubuntu-server systemd[1]: Started Daily apt download activities.

Modern Linux distributions use one of two main logging systems:

1. Traditional Syslog: Text-based log files managed by rsyslog or syslog-ng daemons
2. Systemd Journal: Binary structured logs managed by systemd-journald, accessed via the journalctl command

Important: Understanding your Linux distribution's logging system is crucial for effective monitoring. Most modern distributions use systemd and journald, but many still maintain traditional syslog files for compatibility.

Linux Performance Metrics

Beyond logs, Linux provides detailed performance metrics through various interfaces:

• CPU Usage: User time, system time, idle time, and load averages
• Memory: Used/free memory, swap usage, and buffer/cache statistics
• Disk I/O: Read/write operations, throughput, and utilization
• Network: Bytes sent/received, packet statistics, and connection states
• Process Metrics: Resource usage by individual processes

Monitoring these metrics helps identify performance bottlenecks, capacity issues, and resource constraints before they impact users.

Setting Up Linux Monitoring with OpenObserve

OpenObserve offers two powerful approaches for Linux monitoring:

1. OpenObserve Agent: A streamlined, all-in-one solution for quick deployment
2. OpenTelemetry Collector: A customizable approach for advanced configurations

Let's explore both options, starting with the recommended and simplest approach.

Option 1: Using the OpenObserve Agent (Recommended)

The OpenObserve agent provides a simple, one-command installation that automatically collects both Linux system logs and performance metrics.

Prerequisites

Before we begin, ensure you have:

1. A Linux machine (Ubuntu, Debian, CentOS, RHEL, etc.)
2. Root or sudo access to install and configure services
3. Access to an OpenObserve instance (cloud or self-hosted)

Installation

To install the OpenObserve agent:

1. Log in to your OpenObserve instance
2. Navigate to Data Sources → Recommended → Linux
3. Copy the provided bash command
4. Open a terminal and execute the command:

curl -O https://raw.githubusercontent.com/openobserve/agents/main/linux/install.sh && chmod +x install.sh && sudo ./install.sh https://your-openobserve-instance.com/api/default/ YOUR_API_KEY

The command will automatically be populated with your specific OpenObserve endpoint and API key from the UI, so you can simply copy and paste it.

That's it! The agent will automatically:

• Install as a systemd service that starts automatically
• Collect system logs from journald or syslog
• Collect host metrics (CPU, memory, disk, network)
• Forward all data to your OpenObserve instance

What's Being Collected

The OpenObserve agent collects:

System Logs:

• Journald logs (on systemd-based distributions)
• Syslog (on traditional syslog-based systems)
• Authentication events
• Kernel messages

Performance Metrics:

• CPU usage (total and per core)
• Memory usage and availability
• Disk I/O and space utilization
• Network traffic and errors
• Process-level metrics

Note: The agent automatically detects your Linux distribution and configures itself accordingly, making it work seamlessly across different environments.

Option 2: Using the OpenTelemetry Collector

For more advanced configurations or if you're already using OpenTelemetry in your environment, you can use the OpenTelemetry Collector with the appropriate receivers.

Step 1: Setting Up the OpenTelemetry Collector

First, let's set up the OpenTelemetry Collector:

# Create a directory for the collector
sudo mkdir -p /opt/otel-collector
cd /opt/otel-collector

# Download the latest collector contrib distribution
curl -L https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.115.0/otelcol-contrib_0.115.0_linux_amd64.tar.gz -o otelcol-contrib.tar.gz

# Extract the archive
tar -xzf otelcol-contrib.tar.gz

Important: Always check for the latest version of the OpenTelemetry Collector on the official releases page.

Step 2: Configure the OpenTelemetry Collector

Create a file named config.yaml in the /opt/otel-collector directory with the following content:

receivers:
  filelog:
    include:
      - /var/log/syslog
      - /var/log/auth.log
      - /var/log/kern.log
      - /var/log/messages
      - /var/log/secure
    start_at: end
    include_file_path: true
    include_file_name: true
    operators:
      - type: regex_parser
        regex: '^(?P<time>[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+(?P<host>[^\s]+)\s+(?P<service>[^\s\[]+)(\[(?P<pid>\d+)\])?:\s+(?P<message>.*)$'
        timestamp:
          parse_from: time
          layout: Jan 02 15:04:05
  
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      disk:
      filesystem:
      network:
      load:
      paging:
      process:
        mute_process_name_error: true
        metrics:
          process.cpu.utilization:
            enabled: true
          process.memory.utilization:
            enabled: true

processors:
  batch:
    send_batch_size: 1024
    timeout: 10s
  resourcedetection:
    detectors: [system, env]
    system:
      hostname_sources: ["os"]

exporters:
  otlphttp/openobserve:
    endpoint: "https://your-openobserve-instance.com/api/default"
    headers:
      Authorization: "Basic YOUR_API_KEY"
      stream-name: "linux-logs"
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/openobserve, debug]
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [otlphttp/openobserve, debug]
  telemetry:
    logs:
      level: "info"

Replace https://your-openobserve-instance.com/api/default with your OpenObserve endpoint and YOUR_API_KEY with your actual API key.

Step 3: Run the OpenTelemetry Collector

Run the collector using the following command:

sudo ./otelcol-contrib --config config.yaml

Advanced Configuration Options

Here are some essential configurations to enhance your Linux monitoring:

Security-Focused Monitoring

For enhanced security monitoring:

filelog:
  include:
    - /var/log/auth.log
    - /var/log/secure
    - /var/log/audit/audit.log
  operators:
    # For SSH authentication events
    - type: router
      routes:
        - expr: 'includes(file.name, "auth.log") or includes(file.name, "secure")'
          output: auth-parser
    - type: regex_parser
      id: auth-parser
      regex: '(?P<time>[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+(?P<host>[^\s]+)\s+sshd\[(?P<pid>\d+)\]:\s+(?P<message>.*)'
      timestamp:
        parse_from: time
        layout: Jan 02 15:04:05

This configuration focuses on authentication logs to help detect unauthorized access attempts and potential security breaches.

Using Journald Receiver

If you prefer to collect logs directly from journald instead of log files:

journald:
  units:
    - ssh.service
    - systemd-logind.service
  priority: info

Important: The journald receiver requires systemd and only works on Linux systems using systemd as their init system.

Troubleshooting

If you encounter issues with your Linux monitoring setup, here are the most common problems and their solutions:

Agent Installation Issues

• Permission Denied: Run the installation command with sudo
• Service Not Starting: Check status with sudo systemctl status openobserve-agent and look for error messages
• Network Issues: Verify connectivity with curl -v https://your-openobserve-instance.com

Log Collection Issues

• Missing Logs: Check file permissions and ensure the agent has read access to log files
• Journald Access: For systemd-based systems, ensure the agent has proper journal access
• Log Format Problems: If logs appear malformed, check timezone settings and log formats

Metrics Collection Issues

• Missing Metrics: Verify the agent has permissions to access system metrics
• Performance Impact: If the agent uses too many resources, adjust collection intervals

OpenTelemetry Collector Issues

• Configuration Errors: Validate your config.yaml syntax and check collector logs
• Export Failures: Verify your endpoint URL and API key are correct
• Resource Constraints: Adjust batch settings if the collector terminates unexpectedly

For detailed troubleshooting, check the agent logs in the console.

Visualizing Linux Monitoring Data in OpenObserve

Once your data is flowing into OpenObserve, you can view and analyze it through the intuitive interface.

Exploring Linux System Logs

Navigate to the Logs Page:

You can filter logs by severity, service, or any other field to focus on specific events.

Analyzing Performance Metrics

The Metrics page allows you to visualize and analyze system performance metrics:

You can create custom charts to track CPU usage, memory consumption, disk I/O, and network activity over time.

Pre-built Dashboard

OpenObserve provides a pre-built dashboard for Linux Host Metrics that you can import directly:

Conclusion

Effective Linux monitoring is essential for maintaining reliable, secure, and high-performing systems. With OpenObserve, you now have a powerful platform for collecting, analyzing, and visualizing your Linux logs and metrics.

By following this guide, you've set up comprehensive monitoring that provides deep visibility into your Linux environment. The combination of system logs and performance metrics gives you a complete picture of your systems' health and behavior.

To maximize the value of your monitoring setup:

• Implement log rotation to prevent logs from consuming too much disk space
• Correlate logs with metrics to quickly identify the root cause of issues
• Enhance security monitoring by tracking privileged commands and critical file changes
• Configure appropriate retention periods to meet compliance requirements

Whether you're managing a single server or a large fleet of Linux machines, this monitoring approach scales with your needs while providing the insights required for effective system administration.

For more information and advanced configurations, check out the OpenObserve documentation and start transforming your Linux monitoring today.

Happy monitoring! 🚀