Edge Delta OTTL Transform Node

Transform logs using OTTL statements.

  5 minute read  

Beta

Overview

OTTL (Observability Telemetry Transformation Language) statements are used for transforming telemetry data. OTTL allows specifying transformations and operations on the data, using a defined language structure and functions. You can use it for altering field values, upserting (update or insert) new fields, and applying conditional logic to modify the data set based on certain criteria.

The OTTL node enables you to transform data using OTTL statements.

Note: advanced transformations may have an impact on agent performance.

OTTL Statements

See the OpenTelemetry OTTL documentation for more details about the OTTL.

Note: Inserting regex as a string requires correct escaping of special characters. See Regex as a String.

Upserting (Update or Insert) New Fields with a Static Value

Syntax: set(target, value)

Here, the new_field attribute is added with a static value “defaultValue” if it doesn’t exist, or updated if it does.

set(attributes["new_field"], "defaultValue")

Converting the Case

Syntax: set(target, value) Syntax: ConvertCase(target, toCase)

This statement converts the value of existing_field to upper case and assigns it to a new attribute capitalized_field.

set(attributes["capitalized_field"], ConvertCase(attributes["existing_field"], "upper"))

Decoding a Base64 String

Syntax: set(target, value) Syntax: Decode(value, encoding)

This example decodes a base64 string and assigns the result to a base64 attribute.

set(attributes["base64"], Decode("aGVsbG8gd29ybGQ=", "base64"))

Applying Conditional Logic to Modify the Data Set

Syntax: syntax: set(target, value) where <condition>

This statement sets the alert field to Critical only if the status field is error, applying conditional logic for field modification.

set(attributes["alert"], "Critical") where attributes["status"] == "error"

Appending Tags

Syntax: append(target, value)

This example appends a new tag to the tags attribute list.

append(attributes["tags"], "prod")

Deleting a Specific Key

Syntax: delete_key(target, key)

This example removes a specific key from attributes.

delete_key(attributes, "http.request.header.authorization")

Deleting Matching Keys

Syntax: delete_matching_keys(target, pattern)

Use this to remove keys that match a particular regex pattern, like sensitive keys.

delete_matching_keys(attributes, "(?i).*password.*")

Make sure you properly escape the regex pattern as a string. See Regex as a String.

Keeping Only Matching Keys

Syntax: keep_matching_keys(target, pattern)

This keeps keys that match a given pattern, filtering out all other keys.

keep_matching_keys(attributes, "(?i).*version.*")

Make sure you properly escape the regex pattern as a string. See Regex as a String.

Flattening Nested Structures:

Syntax: flatten(target)

This is used to flatten nested attribute structures.

flatten(attributes)

Setting a Duration from a String

Syntax: set(target, value) Syntax: Duration(value) Syntax: where

This converts a string representation of a duration into a duration object if the original value is a string.

set(attributes["duration"], Duration(attributes["duration_str"])) where IsString(attributes["duration_str"])

Edge Delta Custom OTTL Statements

Edge Delta has extended the OTTL with additional custom functionality.

EDXParseKeyValue

Consider a function that parses key value pairs into attributes. In circumstances where a log contains multiple key-value pairs with the same key, traditionally only the last instance is used. For example:

{
  "body": "key1=1.1|key1=2|key1=three"
}

would be parsed as

{
  "attributes": {
    "key1": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

The values 1.1 and 2 have been lost.

Edge Delta’s EDXParseKeyValue function exposes a number of options for dealing with this situation:

Syntax: EDXParseKeyValue(<target>, "<delimiter>", "<pair delimiter>", <string conversion>, "<merge strategy>")

  • target: the location of the field to be parsed into attributes
  • delimiter: The delimiter between key and value.
  • pair delimiter: The delimiter between key-value pairs.
  • string conversion: A Boolean value indicating whether to convert strings into float or int if a number is detected.
  • merge strategy: Select a merge strategy: first|last|append|concat|indexed

Note: identical key-value pairs are treated as a single key-value.

Examples:

Consider this input:

{
  "body": "key1=1.1|key1=2|key1=three"
}

Keep First

set(attributes, EDXParseKeyValue(body, "=", "|", false, "first"))

In this example the first value is kept and the rest are dropped.

{
  "attributes": {
    "key1": "1.1"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Keep Last

set(attributes, EDXParseKeyValue(body, "=", "|", false, "last"))

In this example the last value is kept and the rest are dropped.

{
  "attributes": {
    "key1": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Append Values

set(attributes, EDXParseKeyValue(body, "=", "|", false, "append"))

In this example the values are all included as an array.

{
  "attributes": {
    "key1": [
      "1.1",
      "2",
      "three"
    ]
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Concatenate Values

set(attributes, EDXParseKeyValue(body, "=", "|", false, "concat"))

In this example the values are all included as a single comma separated value.

{
  "attributes": {
    "key1": "1.1, 2, three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Index Values

set(attributes, EDXParseKeyValue(body, "=", "|", false, "indexed"))

In this example the values are all included by creating multiple indexed keys.

{
  "attributes": {
    "key1_0": "1.1",
    "key1_1": "2",
    "key1_2": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Convert strings to int or float

set(attributes, EDXParseKeyValue(body, "=", "|", true, "indexed"))

In this example some strings are detected as being an integer and a float and converted into the appropriate data types. The string remains unchanged.

{
  "attributes": {
    "key1_0": 1.1,
    "key1_1": 2,
    "key1_2": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ottl_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

statements

The statements parameter defines a collection of OTTL expressions that specify how each data item should be transformed. Each statement is written using OTTL syntax as a block scalar in YAML format, allowing multiple transformation instructions to be included within a single statements parameter. The pipe character | signifies that the subsequent content is a multi-line string.

nodes:
  - name: <name>
    type: ottl_transform
    statements: |
      <OTTL expression>
      <OTTL expression>
      <OTTL expression>

Note: When defining multiple transformation statements, ensure that each statement appropriately considers data dependencies and transformation order.