Edge Delta OTTL Transform Node

Overview

OTTL (Observability Telemetry Transformation Language) statements are used for transforming telemetry data. OTTL allows specifying transformations and operations on the data, using a defined language structure and functions. You can use it for altering field values, upserting (update or insert) new fields, and applying conditional logic to modify the data set based on certain criteria.

The OTTL node enables you to transform data using OTTL statements.

Note: advanced transformations may have an impact on agent performance.

OTTL Statements

See the OpenTelemetry OTTL documentation for more details about the OTTL.

Upserting (Update or Insert) New Fields with a Static Value

Syntax: set(target, value)

Here, the new_field attribute is added with a static value “defaultValue” if it doesn’t exist, or updated if it does.

set(attributes["new_field"], "defaultValue")

Converting the Case

Syntax: set(target, value) Syntax: ConvertCase(target, toCase)

This statement converts the value of existing_field to upper case and assigns it to a new attribute capitalized_field.

set(attributes["capitalized_field"], ConvertCase(attributes["existing_field"], "upper"))

Decoding a Base64 String

Syntax: set(target, value) Syntax: Decode(value, encoding)

This example decodes a base64 string and assigns the result to a base64 attribute.

set(attributes["base64"], Decode("aGVsbG8gd29ybGQ=", "base64"))

Applying Conditional Logic to Modify the Data Set

Syntax: syntax: set(target, value) where <condition>

This statement sets the alert field to Critical only if the status field is error, applying conditional logic for field modification.

set(attributes["alert"], "Critical") where attributes["status"] == "error"

Appending Tags

Syntax: append(target, value)

This example appends a new tag to the tags attribute list.

append(attributes["tags"], "prod")

Deleting a Specific Key

Syntax: delete_key(target, key)

This example removes a specific key from attributes.

delete_key(attributes, "http.request.header.authorization")

Deleting Matching Keys

Syntax: delete_matching_keys(target, pattern)

Use this to remove keys that match a particular regex pattern, like sensitive keys.

delete_matching_keys(attributes, "(?i).*password.*")

Keeping Only Matching Keys

Syntax: keep_matching_keys(target, pattern)

This keeps keys that match a given pattern, filtering out all other keys.

keep_matching_keys(attributes, "(?i).*version.*")

Flattening Nested Structures:

Syntax: flatten(target)

This is used to flatten nested attribute structures.

flatten(attributes)

Setting a Duration from a String

Syntax: set(target, value) Syntax: Duration(value) Syntax: where

This converts a string representation of a duration into a duration object if the original value is a string.

set(attributes["duration"], Duration(attributes["duration_str"])) where IsString(attributes["duration_str"])

Example Configuration

This configuration checks if the uuid attribute is empty. If it is, it sets the uuid attribute to a default value of 00000000-0000-0000-0000-000000000000. It adds a new attribute log_source with the value ClickHouse. Finally it converts the level attribute (severity level) to uppercase and assigns it to a new attribute normalized_level.

nodes:
- name: Transform ClickHouse
  type: ottl_transform
  statements: |-
    set(attributes["uuid"], "00000000-0000-0000-0000-000000000000") where attributes["uuid"] == ""
    set(attributes["log_source"], "ClickHouse")
    set(attributes["normalized_level"], ConvertCase(attributes["level"], "upper"))    

Input

{
  "body": "2024.10.22 03:42:45.068446 [ 1786 ] {} <Debug> analytics.module (QueryEngine): Loaded queue",
  "resource": {
...
  },
  "attributes": {
    "level": "Debug",
    "message": "analytics.module (QueryEngine): Loaded queue",
    "pid": "1786",
    "timestamp": "2024.10.22 03:42:45.068446",
    "uuid": ""
  }
}

Output

{
  "body": "2024.10.22 03:42:45.068446 [ 1786 ] {} <Debug> analytics.module (QueryEngine): Loaded queue",
  "resource": {
...
  },
  "attributes": {
    "level": "Debug",
    "log_source": "ClickHouse",
    "message": "analytics.module (QueryEngine): Loaded queue",
    "normalized_level": "DEBUG",
    "pid": "1786",
    "timestamp": "2024.10.22 03:42:45.068446",
    "uuid": "00000000-0000-0000-0000-000000000000"
  }
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ottl_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

statements

The statements parameter defines a collection of OTTL expressions that specify how each data item should be transformed. Each statement is written using OTTL syntax as a block scalar in YAML format, allowing multiple transformation instructions to be included within a single statements parameter. The pipe character | signifies that the subsequent content is a multi-line string.

nodes:
  - name: <name>
    type: ottl_transform
    statements: |
      <OTTL expression>
      <OTTL expression>
      <OTTL expression>      

Note: When defining multiple transformation statements, ensure that each statement appropriately considers data dependencies and transformation order.