Ingesting Data from Cribl to OpenObserve: A Complete Guide

What is Cribl?

Cribl (https://cribl.io/) is a data engine designed to optimize and manage the flow of observability and security data. It provides users with the ability to:

• Route data to multiple destinations, reducing vendor lock-in
• Optimize data ingestion by filtering, enriching, and transforming data before sending it to storage
• Reduce data costs by shaping data and controlling retention

What is OpenObserve?

OpenObserve is an open-source, high-performance observability platform designed for real-time log and trace analytics. Unlike traditional log storage solutions, OpenObserve provides:

• Ingestion and query optimization: Store, index, and search logs at scale
• Distributed and scalable architecture: Built for cloud-native environments
• Advanced visualization and alerting: Powerful dashboards, search capabilities, and real-time alerts
• Cost efficiency: Significantly reduces observability costs compared to legacy solutions

By integrating Cribl with OpenObserve, organizations can efficiently route, filter, and analyze logs and traces in a cost-effective manner.

In this guide, we will explore how to integrate Cribl with OpenObserve to seamlessly ingest and monitor logs and traces. The tutorial will cover:

• Setting up a simple internal data source in Cribl
• Configuring Cribl to send data to OpenObserve via a webhook destination
• Visualizing and monitoring ingested data in OpenObserve

How to Ingest Data from Cribl to OpenObserve

The following steps outline the integration process:

1. Configuring a Simple Internal Source in Cribl

To get started with Cribl, we need to create an internal data source that generates test logs for forwarding to OpenObserve.

Step 1: Access Cribl UI

• Navigate to the Cribl Stream UI and log in.
• Go to Worker group -> Routing -> QuickConnect
• Select Sources from the left-hand menu.
• Click on Add Source and choose System and Internal as the source type and select Cribl Internal

Step 2: Configure the Internal Source

• Provide a name for the source (e.g., cribl).
• Define a data generation pattern or use default sample logs.
• Click Save & Start to activate the source.

Once this is set up, Cribl will begin generating test logs for processing.

2. Configuring a Webhook Destination in Cribl

Now, we configure Cribl to send data to OpenObserve via a webhook.

Step 1: Create a Webhook Destination

• In Cribl UI, navigate to Destinations.
• Click Add Destination and select Webhook.

Step 2: Set Up OpenObserve as the Destination

• Provide a name for the destination (e.g., OpenObserve_Webhook).
• Set the webhook URL to OpenObserve’s ingestion API endpoint:

  http://<openobserve-server>/api/default/cribl/_json
  

• Select POST as the HTTP method.
• Configure Authentication:
  • Authentication type: Basic
  • Username: O2_INGESTION_USER
  • Password: O2_INGESTION_PASSWORD
• Save the destination and activate it.

3. Routing Data from the Internal Source to OpenObserve

Now that we have both an internal source and a webhook destination, we need to create a route that connects them.

You can connect the source and destination via Passthru and save the connection which is ready to send the logs. You can test this by generating sample logs in the webhook section and you should see a successful message if all the configurations were successful.

Monitoring Logs and Traces in OpenObserve

After sending data from Cribl, we can now analyze it in OpenObserve.

Step 1: Query Logs in OpenObserve

• Log into OpenObserve UI.
• Navigate to Logs and select cribl from streams:
• This should display logs received from Cribl.

Step 2: Query Traces in OpenObserve

• Navigate to Traces and select cribl from streams:
• This should display traces received from Cribl.

Conclusion

By integrating Cribl with OpenObserve, organizations can streamline log and trace ingestion while reducing costs and improving analytics. Cribl provides powerful data processing and optimization capabilities, while OpenObserve offers scalable storage, visualization, and alerting features.

With this setup:

• Data flows seamlessly from Cribl to OpenObserve.
• Logs are enriched and transformed before ingestion.
• Monitoring and alerting ensure proactive observability.

Happy monitoring! 🚀