Edge Delta Resource Transform Node

Overview

The Resource Transform node changes or adds source information (resources section) of incoming log items. It supports both Common Expression Language (CEL) and Go templates for evaluating expressions.

Each transformation node is designed for enriching different sections of the data item:

Example Configuration

The following configuration will add three fields to the resources data: k8s.container.name, k8s.namespace.name, and k8s.pod.name. The k8s.container.name and k8s.namespace.name values will be extracted from the JSON body of the log. The k8s.pod.name value will be a static value -.

nodes:
- name: resource_transform
  type: resource_transform
  target_source_type: k8s
  source_field_overrides:
  - field: k8s.container.name
    expression: json(item["body"]).kubernetes.container.name
  - field: k8s.namespace.name
    expression: json(item["body"]).kubernetes.namespace
  - field: k8s.pod.name
    expression: '"-"'

Consider the following input log:

{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}

Suppose it is ingested with the following resource data:

{
  "_type": "log"
  "body": "{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}"
  "resource": {
    "ed.conf.id": "<redacted>"
    "ed.org.id": "<redacted>"
    "ed.tag": "testing pipeline"
    "host.ip": "10.0.0.1"
    "host.name": "ED_TEST"
    "src_type": ""
  }
  "timestamp": 1712670236661
}

After processing by the Resource Transform node, the data item contains additional k8s resource fields.

{
  "_type": "log"
  "body": "{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}"
  "resource": {
    "config_id": "<redacted>"
    "host": "ED_TEST"
    "ip": "10.0.0.1"
    "k8s_container_image": ""
    "k8s_container_name": "container-1"
    "k8s_controller_kind": ""
    "k8s_controller_logical_name": ""
    "k8s_controller_name": ""
    "k8s_docker_id": ""
    "k8s_file_path": ""
    "k8s_namespace": "namespace-a"
    "k8s_node_name": ""
    "k8s_pod_id": ""
    "k8s_pod_name": "-"
    "org_id": "<redacted>"
    "src_type": "K8s"
    "tag": "testing pipeline"
  }
  "timestamp": 1712670236661
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: resource_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

target_source_type

The target_source_type parameter defines the type of source that the logs should reflect in their resource section. It can be docker, k8s, ecs, file, or custom. This determines the required fields in the source_field_overrides section. For Custom, at least one source field override must be defined. A target_source_type is required.

nodes:
- name: <noode name>
  type: resource_transform
  target_source_type: docker | k8s | ecs | file | custom
  source_field_overrides:
  - field: <field name>
    expression: <expression to determine fields value>

source_field_overrides

The source_field_overrides parameter defines the resource fields and the logic for determining their values. It is specified with two child parameters: field and expression. A source_field_overrides is required.

The fields are determined by the target_source_type selected:

docker

• container.name, container.image.name (Mandatory)
• container.image.tag (Optional but not defining it can cause loss of information)

ecs:

• aws.ecs.container.arn (Mandatory)
• aws.ecs.cluster.arn, aws.ecs.task.family, aws.ecs.task.revision (Optional but not defining them can cause loss of information)

file:

• ed.filepath (Mandatory)

k8s:

• k8s.namespace.name, k8s.pod.name, k8s.container.name (Mandatory)
• k8s.deployment.name, k8s.statefulset.name, k8s.daemonset.name, k8s.replicaset.name, k8s.job.name, k8s.job.name (Optional but not defining them can cause loss of information)

The expression defines the logic to use to determine the values using a CEL macro or Go template. Alternatively, instead of defining a path, you can use “-” to skip a field to decrease cardinality.

nodes:
- name: <noode name>
 type: resource_transform
 target_source_type: docker | k8s | ecs | file | custom
 source_field_overrides:
 - field: <field name>
   expression: <expression to determine fields value>
 - field: <field name>
   expression: <expression to determine fields value>