Edge Delta Output Transform Node

Transform and output logs.

  4 minute read  

Paid
Visual Pipelines Beta

Overview

The Output Transform node is used to transform and then output logs. Output transform nodes are similar to log transform nodes except they have no restrictions on fields that can be modified or deleted. This node emits a custom type that can be sent to Datadog, Splunk, Sumo, and Elastic.

Ouput transform creates a custom item with potentially deleted fields. Therefore when this item is sent to Datadog or Splunk without supplying a mapping, the whole payload will be sent as the event and all other fields will be empty. This makes the custom item harder to search for in the destination. Elastic and Sumo attempt to index the fields of the whole payload as an object instead.

Unlike Enrichment nodes which calculate the enrichment using only the first log per source, transform node transformations are calculated for each log processed. Transform nodes therefore have a slower throughput than enrichments.

Example Configuration 1

In the following example the transformations are applied in order. The node upserts the first two values with their new value and “new_field“ is added with an empty string because the field supplied for it’s value doesn’t exist. Finally the delete operation is applied.

nodes:
  - name: output_transform_example
    type: ​​output_transform
    transformations:
      - field_path: _raw
        operation: upsert
        value: json(item["body"]).msg
      - field_path: tag
        operation: upsert
        value: item["resource"]["tag"]
      - field_path: new_field
        operation: upsert
        value: item["resource"]["nonexistent_field"]
      - field_path: resource
        operation: delete
      - field_path: body
        operation: delete

Example Input

{
    "timestamp": "1581452773000000789",
    "body": "{\"timestamp\": \"2023/07/11 09:40:21\",\"msg\": \"Failed to do something\"}",
    "resource": {
      "host": "host-1",
      "tag": "app-dev",
      "__source": {
        "type": "K8s",
        "short_name": "short_source_name",
        "name": "source_name",
        "group_name": "group_name",
        "logical_name": "logical_source",
      },
      "k8s.namespace.name": "edgedelta",
      "k8s.pod.name": "api-deployment-d79fab72249c",
      "k8s.container.name": "echo:latest",
      "k8s.controller.kind":"api-deployment",
      "k8s.controller.logical_name":"Deployment",
      "k8s.labels.app":"my-api",
    },
    "type": "log",
    "attributes": {
        "pod_id":"api-deployment-d79fab72249c-vtq9x",
        "instance_id":"i-1234567890abcdef0",
        "instance_name":"test-name",
    }
}

Example Output Log

{
    "timestamp": "1581452773000000789",
    "_raw": "Failed to do something",
    "tag": "app-dev",
    "new_field": "",
    "type": "log",
    "attributes": {
        "pod_id":"api-deployment-d79fab72249c-vtq9x",
        "instance_id":"i-1234567890abcdef0",
        "instance_name":"test-name",
    }
}

Example Configuration 2

nodes:
  - name: output_transform_example
    type: ​​output_transform
    transformations:
      - field_path: resource.__source.type
        operation: upsert
        value: "LocalK8s"
      - field_path: tag
        operation: upsert
        value: item["resource"]["tag"]
      - field_path: nonexistent_field
        operation: delete

Example Input

{
    // ED fields 
    "_ed": {
		"type": "log",
		"timestamp": 1666045135471,  // milliseconds epoch
		"host": "host-1",
		"tag": "app-dev",
		"src_type": "K8s",
	    "src_name": "source_name",
	    "k8s_namespace": "edgedelta",
		"k8s_pod_name": "api-deployment-d79fab72249c",
		"k8s_container_image": "echo:latest",
		"k8s_controller_kind": "Deployment",
		"k8s_controller_logical_name": "api-deployment",
		"k8s_labels": {
			"app": "my-api",
		},
    },

    // raw message as a string
    "_raw": "{\"msg\":\"hello world\",\"severity\":1}",

    // parsed field if JSON parsing is applied
    "_parsed": {
		"msg": "hello world",
		"severity": 1,
    },

    // customer defined enrichment fields
    "role": "admin",
    "service": "db-service"

    // fields for internal comms.
    "__is_apm_data": true,
}

Example Output Log

{
    "timestamp": "1581452773000000789",
    "body": "{\"timestamp\": \"2023/07/11 09:40:21\",\"msg\": \"Failed to do something\"}",
    "resource": {
      "host": "host-1",
      "tag": "app-dev",
      "__source": {
        "type": "LocalK8s",
        "short_name": "short_source_name",
        "name": "source_name",
        "group_name": "group_name",
        "logical_name": "logical_source",
      },
      "k8s.namespace.name": "edgedelta",
      "k8s.pod.name": "api-deployment-d79fab72249c",
      "k8s.container.name": "echo:latest",
      "k8s.controller.kind":"api-deployment",
      "k8s.controller.logical_name":"Deployment",
      "k8s.labels.app":"my-api",
    },
    "tag": "app-dev",
    "type": "log",
    "attributes": {
        "pod_id":"api-deployment-d79fab72249c-vtq9x",
        "instance_id":"i-1234567890abcdef0",
        "instance_name":"test-name",
    }
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: output_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Transformations

The transformations parameter is used to specify the log transformation operations. It consists of three child parameters:

  • operationis used to specify the transformation operation. Currently it can be delete or upsert, which updates the field if it exists or adds it if it doesn’t exist.
  • field_path is the dot separated path where the operation should be applied.
  • value is the CEL expression that determines the value to be applied. There is a set of UDF’s for CEL that you can use. The OTEL indexing method of CEL must be used, for example item["resource"]["host.name"].
nodes:
  - name: <node name>
    type: ​output_transform
    transformations:
    - operation: upsert|delete
      field_path: <dot separated path>
      value: <CEL expression>