Edge Delta Output Transform Node

Transform and output logs.

4 minute read

Overview

The Output Transform node is used to transform and then output logs. Output transform nodes are similar to log transform nodes except they have no restrictions on fields that can be modified or deleted. This node emits a custom type that can be sent to an HTTP destination node as well as Datadog, Splunk, Sumo, and Elastic.

incoming_data_types: custom, log, metric, cluster_pattern_and_sample
outgoing_data_types: custom

Output transform creates a custom item with potentially deleted fields. Therefore when this item is sent to Datadog or Splunk without supplying a mapping, the whole payload will be sent as the event and all other fields will be empty. This makes the custom item harder to search for in the destination. Elastic and Sumo attempt to index the fields of the whole payload as an object instead.

Each transformation node is designed for enriching different sections of the data item:

Node	Attribute	Resource	Body	Timestamp	Any Field	Restrictions
Output Transform	Y	Y	Y	Y	Y	Outputs a Custom type data item that cant be ingested by the Edge Delta Archive node. The whole payload is flattened and sent as the event, with all other fields empty.
Log Transform	Y	N	N	Y	N	Can only ingest logs, and it outputs only logs.
Resource Transform	N	Y	N	N	N	Can only ingest logs, and it outputs only logs.
Mask	N	N	Y	N	N	Can only ingest logs, and it outputs only logs. New value can only be a static string.
Generic Transform	Y	Y	N	Y	N	Can only transform non-body fields.
OTTL Transform	Y	Y	Y	Y	Y	Can transform any field on any data type.

The body field is protected from dynamic enrichment until the end of the pipeline (Output Transform) to prevent schema changes from disabling pipeline functionality.

Example Configuration

In the following example the transformations are applied in order. The node upserts the first two values with their new value and “new_field“ is added with an empty string because the field supplied for it’s value doesn’t exist. Finally the delete operations are applied.

nodes:
  - name: output_transform_example
    type: output_transform
    transformations:
      - field_path: message
        operation: upsert
        value: json(item["body"]).msg
      - field_path: tag
        operation: upsert
        value: item["resource"]["ed.tag"]
      - field_path: new_field
        operation: upsert
        value: item["resource"]["nonexistent_field"]
      - field_path: resource
        operation: delete
      - field_path: body
        operation: delete

Example Input

	{
		"_type": "log",
		"body": "{\"timestamp\": \"2023/07/11 09:40:21\", \"msg\": \"Failed to do something\"}",
		"resource": {
			"ed.conf.id": "12345678-1x234-4abc-def5-12345678910g",
			"ed.org.id": "1098765a-432b-1cde-2345-fg6789hij101",
			"ed.tag": "parallel_container",
			"host.ip": "10.0.0.1",
			"host.name": "ED_TEST",
			"src_type": "memory_input"
		},
		"timestamp": 1715690460477
	}

Example Output Log

[
	{
		"_type": "log",
		"message": "Failed to do something",
		"new_field": "",
		"tag": "parallel_container",
		"timestamp": 1715690460477
	}
]

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: output_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Transformations

The transformations parameter is used to specify the transformation operations. It consists of three child parameters:

operationis used to specify the transformation operation. Currently it can be delete or upsert, which updates the field if it exists or adds it if it doesn’t exist.
field_path is the dot separated path where the operation should be applied.
value is the CEL expression that determines the value to be applied and you can use CEL macros. The OTEL indexing method of CEL must be used, for example item["resource"]["host.name"].

nodes:
  - name: <node name>
    type: output_transform
    transformations:
    - operation: upsert|delete
      field_path: <dot separated path>
      value: <CEL expression>

Optional Parameters

ignore_if_empty

The ignore_if_empty parameter determines whether to perform an upsert if the value is null. It is configured as a Boolean and the default is false. If false, empty values will be added to the payload. If true, the value will be omitted when it is empty. The ignore_if_empty parameter requires the upsert operation and it is optional.

nodes:
  - name: <node name>
    type: output_transform
    transformations:
    - operation: upsert
      field_path: <dot separated path>
      value: <CEL expression>
      ignore_if_empty: true

See Also:

Knowledge Library