Question 1

What types of pipelines does OpenObserve support?

Accepted Answer

OpenObserve supports two main types of pipelines: real-time and scheduled. Real-time pipelines process data as it arrives, transforming and enriching it on the fly. These pipelines can include multiple processing steps using functions, conditions, and stream operations. Scheduled pipelines run at defined intervals, allowing for batch processing and periodic data transformations.

Question 2

How do functions work in pipelines?

Accepted Answer

Functions in OpenObserve use Vector Remap Language (VRL) for data transformation. Each function contains VRL code that can parse, transform, and enrich data. Functions can access the incoming data fields, perform conditional processing, and modify or create new fields. For example, a function might parse JSON logs, extract specific fields, and enrich them with geographical information using enrichment tables.

Question 3

What are enrichment tables and how are they used?

Accepted Answer

Enrichment tables are CSV-based lookup tables that allow you to add additional context to your data. You can upload CSV files containing reference data, which can then be queried within pipeline functions using the getenrichmenttablerecord function. Common use cases include IP to location mapping, user agent parsing, and adding business context to technical data.

Question 4

How does data parsing work in pipelines?

Accepted Answer

Data parsing in OpenObserve pipelines is handled through VRL functions. The platform supports parsing various formats including JSON, structured logs, and custom formats. Functions can include conditional logic to apply different parsing rules based on the data source or content. Error handling is built into the parsing functions, allowing graceful handling of malformed data.

Question 5

What stream operations are available?

Accepted Answer

Pipeline streams can be configured with various operations including source selection, transformation steps, and destination routing. The platform supports both logs and metrics streams. You can create complex workflows with multiple processing steps, conditions, and parallel processing paths. Stream operations maintain data consistency while allowing for flexible transformation chains.

Question 6

How do pipeline conditions work?

Accepted Answer

Conditions in pipelines allow for selective processing based on data attributes. Using VRL expressions, you can create sophisticated routing and processing logic. Conditions can check field values, apply pattern matching, and implement complex business rules. This enables targeted processing of specific data streams or message types.

Question 7

How are pipeline changes managed?

Accepted Answer

Pipeline configurations are managed through the OpenObserve interface, where you can create, edit, and delete pipelines. Changes to pipeline functions can be tested before deployment using sample telemetry data.

Observability Pipelines

Protect Sensitive Data

Accelerate Data Insights

Improve Data Quality

OpenObserve Observability Pipelines

Pipeline Types

Real-time

Scheduled

Data Transformation

VRL Functions

Data Parsing

Data Enrichment

Enrichment Tables

Dynamic Lookups

Pipeline Components

Function Nodes

Stream Operations