Edge Delta Log Transform Node
4 minute read
Overview
The log transform node is used to transform logs using upsert and delete operations. Core log fields such as _raw and _ed in the Edge Delta schema, and body and resources in the OTEL schema, are protected.
Unlike Enrichment nodes which calculate the enrichment using only the first log per source, transform node transformations are calculated for each log processed. Transform nodes therefore have a slower throughput than enrichments.
Example Configuration
In the following example there are matching logs and logs that do not match the log transform node configuration.
nodes:
- name: log_transform_example
type: log_transform
transformations:
- operation: upsert
field_path: attributes.parsed
value: json(item["body"])
- operation: upsert
field_path: attributes.original_timestamp
value: item["attributes"]["parsed"]["timestamp"]
- operation: upsert
field_path: body
value: item["attributes"]["parsed"]["msg"]
Example Logs
Matching Input
{
"timestamp": "1581452773000000789",
"body": "{\"timestamp\": \"2023/07/11 09:40:21\",\"msg\": \"Failed to do something\"}",
"resource": {
"host": "host-1",
"tag": "app-dev",
"__source": {
"type": "K8s",
"short_name": "short_source_name",
"name": "source_name",
"group_name": "group_name",
"logical_name": "logical_source",
},
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.controller.kind":"api-deployment",
"k8s.controller.logical_name":"Deployment",
"k8s.labels.app":"my-api",
},
"type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x",
"instance_id":"i-1234567890abcdef0",
"instance_name":"test-name",
}
}
Matching Output
{
"timestamp": "1581452773000000789",
"body": "Failed to do something",
"resource": {
"host": "host-1",
"tag": "app-dev",
"__source": {
"type": "K8s",
"short_name": "short_source_name",
"name": "source_name",
"group_name": "group_name",
"logical_name": "logical_source",
},
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.controller.kind":"api-deployment",
"k8s.controller.logical_name":"Deployment",
"k8s.labels.app":"my-api",
},
"type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x",
"instance_id":"i-1234567890abcdef0",
"instance_name":"test-name",
"parsed": {
"timestamp": "2023/07/11 09:40:21",
"msg": "Failed to do something",
},
"original_timestamp": "2023/07/11 09:40:21"
}
}
Not Matching Input
{
"timestamp": "1581452773000000789",
"body": "hello world",
"resource": {
"host": "host-1",
"tag": "app-dev",
"src_type": "K8s",
"__short_src_name": "short_source_name",
"__src_name": "source_name",
"__group_name": "group_name",
"__logical_source": "logical_source",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.controller.kind":"api-deployment",
"k8s.controller.logical_name":"Deployment",
"k8s.labels.app":"my-api",
},
"type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x",
"instance_id":"i-1234567890abcdef0",
"instance_name":"test-name",
}
}
Not Matching Output
{
"timestamp": "1581452773000000789",
"body": "",
"resource": {
"host": "host-1",
"tag": "app-dev",
"src_type": "K8s",
"__short_src_name": "short_source_name",
"__src_name": "source_name",
"__group_name": "group_name",
"__logical_source": "logical_source",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.controller.kind":"api-deployment",
"k8s.controller.logical_name":"Deployment",
"k8s.labels.app":"my-api",
},
"type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x",
"instance_id":"i-1234567890abcdef0",
"instance_name":"test-name",
"parsed": "",
"original_timestamp": "",
}
}
In the not matching example, the log is not in JSON format. Since the JSON parsing of the log message fails, an empty string is used to assign to the new fields. In addition, the body field that stores the log message is also replaced with the empty string, causing the original log message to be lost.
The only expressions the node will consider are
- string literals represented by a single quoted string (e.g. ‘foo’),
- Field References represented by an unquoted identifier prefixed by a dot (for example, ._ed.sourceType) and
- a small collection of Edge Delta specified special functions, including environment(key) to resolve an environment variable, and now() to provide the current time.
For deletion cases the default behavior is to do nothing if the field is not found.
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: log_transform
The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
Transformations
The transformations parameter is used to specify the log transformation operations. It consists of three child parameters:
operationis used to specify the transformation operation. Currently it can be delete or upsert, which updates the field if it exists or adds it if it doesn’t exist.field_pathis the dot separated path where the operation should be applied.valueis the CEL expression that determines the value to be applied. There is a set of UDF’s for CEL that you can use. With the OTEL schema the indexing method of CEL must be used, for exampleitem["resource"]["host.name"]because OTEL has some fields that already contain dots in them.
nodes:
- name: <node name>
type: log_transform
transformations:
- operation: upsert|delete
field_path: <dot separated path>
value: <CEL expression>