Edge Delta OTTL Transform Node

Transform any data type using OTTL statements.

3 minute read

Overview

OTTL is used to express transformations and filtering of data. You can use it for altering field values, upserting (update or insert) new fields, and applying conditional logic to modify the data set based on certain criteria.

You can specify transformations on all data types using OTTL in the OTTL Transform node. See Use OTTL in Edge Delta to learn more.

Note: advanced transformations may have an impact on agent performance.

incoming_data_types: archive, cluster_pattern_and_sample, custom, datadog_payload, diagnostic, health, heartbeat, log, metric, signal, source, source_samples, splunk_payload, trace
outgoing_data_types: archive, cluster_pattern_and_sample, custom, datadog_payload, diagnostic, health, heartbeat, log, metric, signal, source, source_samples, splunk_payload, trace

Each transformation node is designed for enriching different sections of the data item:

Node	Attribute	Resource	Body	Timestamp	Any Field	Restrictions
Output Transform	Y	Y	Y	Y	Y	Outputs a Custom type data item that cant be ingested by the Edge Delta Archive node. The whole payload is flattened and sent as the event, with all other fields empty.
Log Transform	Y	N	N	Y	N	Can only ingest logs, and it outputs only logs.
Resource Transform	N	Y	N	N	N	Can only ingest logs, and it outputs only logs.
Mask	N	N	Y	N	N	Can only ingest logs, and it outputs only logs. New value can only be a static string.
Generic Transform	Y	Y	N	Y	N	Can only transform non-body fields.
OTTL Transform	Y	Y	Y	Y	Y	Can transform any field on any data type.

The body field is protected from dynamic enrichment until the end of the pipeline (Output Transform) to prevent schema changes from disabling pipeline functionality.

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ottl_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

statements

The statements parameter defines a collection of OTTL expressions that specify how each data item should be transformed. Each statement is written using OTTL syntax as a block scalar in YAML format, allowing multiple transformation instructions to be included within a single statements parameter. The pipe character | signifies that the subsequent content is a multi-line string.

nodes:
  - name: <name>
    type: ottl_transform
    statements: |
      <OTTL expression>
      <OTTL expression>
      <OTTL expression>

Note: When defining multiple transformation statements, ensure that each statement appropriately considers data dependencies and transformation order.