Edge Delta Resource Transform Node

Overview

The Resource Transform node changes or adds source information (resources section) of incoming log items. It supports both Common Expression Language (CEL) and Go templates for evaluating expressions.

• incoming_data_types: log
• outgoing_data_types: log

Each transformation node is designed for enriching different sections of the data item:

The body field is protected from dynamic enrichment until the end of the pipeline (Output Transform) to prevent schema changes from disabling pipeline functionality.

Example Configuration

The following configuration will add three fields to the resources data: k8s.container.name, k8s.namespace.name, and k8s.pod.name. The k8s.container.name and k8s.namespace.name values will be extracted from the JSON body of the log. The k8s.pod.name value will be a static value -.

nodes:
- name: resource_transform
  type: resource_transform
  transformations:
  - field_path: k8s.container.name
    value: json(item["body"]).kubernetes.container.name
  - field_path: k8s.namespace.name
    value: json(item["body"]).kubernetes.namespace
  - field_path: k8s.pod.name
    value: '"-"'

Consider the following input log:

{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}

Suppose it is ingested with the following resource data:

{
  "_type": "log",
  "body": "{\"kubernetes\": {\"container\": {\"name\": \"container-1\"}, \"namespace\": \"namespace-a\"}}",
  "resource": {
    "ed.conf.id": "12345678-1x234-4abc-def5-12345678910g",
    "ed.org.id": "1098765a-432b-1cde-2345-fg6789hij101",
    "ed.tag": "parallel_container",
    "host.ip": "10.0.0.1",
    "host.name": "ED_TEST",
    "src_type": "memory_input"
  },
  "timestamp": 1715691719745
}

After processing by the Resource Transform node, the data item contains additional k8s resource fields.

{
  "_type": "log",
  "body": "{\"kubernetes\": {\"container\": {\"name\": \"container-1\"}, \"namespace\": \"namespace-a\"}}",
  "resource": {
    "container.id": "",
    "container.image.name": "",
    "ed.conf.id": "12345678-1x234-4abc-def5-12345678910g",
    "ed.filepath": "",
    "ed.org.id": "1098765a-432b-1cde-2345-fg6789hij101",
    "ed.tag": "parallel_container",
    "host.ip": "10.0.0.1",
    "host.name": "ED_TEST",
    "k8s.container.name": "container-1",
    "k8s.namespace.name": "namespace-a",
    "k8s.node.name": "",
    "k8s.pod.name": "-",
    "k8s.pod.uid": "",
    "src_type": "K8s"
  },
  "timestamp": 1715691719745
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: resource_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

transformations

The transformations parameter defines the resource fields and the logic for determining their values. It is specified with two child parameters: field_path and value. A transformations is required.

The following field paths can be used:

docker

• container.name, container.image.name (Mandatory)
• container.image.tag (Optional but not defining it can cause loss of information)

ecs:

• aws.ecs.container.arn (Mandatory)
• aws.ecs.cluster.arn, aws.ecs.task.family, aws.ecs.task.revision (Optional but not defining them can cause loss of information)

file:

• ed.filepath (Mandatory)

k8s:

• k8s.namespace.name, k8s.pod.name, k8s.container.name (Mandatory)
• k8s.deployment.name, k8s.statefulset.name, k8s.daemonset.name, k8s.replicaset.name, k8s.job.name, k8s.job.name (Optional but not defining them can cause loss of information)

The expression defines the logic to use to determine the values using a CEL macro or Go template. Alternatively, instead of defining a path, you can use “-” to skip a field to decrease cardinality.

nodes:
- name: <node name>
 type: resource_transform
 transformations:
 - field_path: <field name>
   value: <expression to determine fields value>
 - field_path: <field name>
   value: <expression to determine fields value>