Edge Delta Resource Transform Node
4 minute read
Overview
The Resource Transform node changes or adds source information (resources section) of incoming log items. It supports both Common Expression Language (CEL) and Go templates for evaluating expressions.
Each transformation node is designed for enriching different sections of the data item:
Node | Attribute | Resource | Body | Timestamp | Any Field | Restrictions |
---|---|---|---|---|---|---|
Output Transform | Y | Y | Y | Y | Y | Outputs a Custom type data item that cant be ingested by the Edge Delta Archive node. The whole payload is flattened and sent as the event, with all other fields empty. |
Log Transform | Y | N | N | Y | N | Can only ingest logs, and it outputs only logs. |
Resource Transform | N | Y | N | N | N | Can only ingest logs, and it outputs only logs. |
Mask | N | N | Y | N | N | Can only ingest logs, and it outputs only logs. New value can only be a static string. |
Example Configuration
The following configuration will add three fields to the resources data: k8s.container.name
, k8s.namespace.name
, and k8s.pod.name
. The k8s.container.name
and k8s.namespace.name
values will be extracted from the JSON body of the log. The k8s.pod.name
value will be a static value -
.
nodes:
- name: resource_transform
type: resource_transform
target_source_type: k8s
source_field_overrides:
- field: k8s.container.name
expression: json(item["body"]).kubernetes.container.name
- field: k8s.namespace.name
expression: json(item["body"]).kubernetes.namespace
- field: k8s.pod.name
expression: '"-"'
Consider the following input log:
{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}
Suppose it is ingested with the following resource data:
{
"_type": "log"
"body": "{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}"
"resource": {
"ed.conf.id": "<redacted>"
"ed.org.id": "<redacted>"
"ed.tag": "testing pipeline"
"host.ip": "10.0.0.1"
"host.name": "ED_TEST"
"src_type": ""
}
"timestamp": 1712670236661
}
After processing by the Resource Transform node, the data item contains additional k8s resource fields.
{
"_type": "log"
"body": "{"kubernetes": {"container": {"name": "container-1"}, "namespace": "namespace-a"}}"
"resource": {
"config_id": "<redacted>"
"host": "ED_TEST"
"ip": "10.0.0.1"
"k8s_container_image": ""
"k8s_container_name": "container-1"
"k8s_controller_kind": ""
"k8s_controller_logical_name": ""
"k8s_controller_name": ""
"k8s_docker_id": ""
"k8s_file_path": ""
"k8s_namespace": "namespace-a"
"k8s_node_name": ""
"k8s_pod_id": ""
"k8s_pod_name": "-"
"org_id": "<redacted>"
"src_type": "K8s"
"tag": "testing pipeline"
}
"timestamp": 1712670236661
}
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: resource_transform
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
target_source_type
The target_source_type
parameter defines the type of source that the logs should reflect in their resource section. It can be docker
, k8s
, ecs
, file
, or custom
. This determines the required fields in the source_field_overrides
section. For Custom, at least one source field override must be defined. A target_source_type
is required.
nodes:
- name: <noode name>
type: resource_transform
target_source_type: docker | k8s | ecs | file | custom
source_field_overrides:
- field: <field name>
expression: <expression to determine fields value>
source_field_overrides
The source_field_overrides
parameter defines the resource fields and the logic for determining their values. It is specified with two child parameters: field and expression. A source_field_overrides
is required.
The fields are determined by the target_source_type
selected:
docker
- container.name, container.image.name (Mandatory)
- container.image.tag (Optional but not defining it can cause loss of information)
ecs:
- aws.ecs.container.arn (Mandatory)
- aws.ecs.cluster.arn, aws.ecs.task.family, aws.ecs.task.revision (Optional but not defining them can cause loss of information)
file:
- ed.filepath (Mandatory)
k8s:
- k8s.namespace.name, k8s.pod.name, k8s.container.name (Mandatory)
- k8s.deployment.name, k8s.statefulset.name, k8s.daemonset.name, k8s.replicaset.name, k8s.job.name, k8s.job.name (Optional but not defining them can cause loss of information)
The expression defines the logic to use to determine the values using a CEL macro or Go template. Alternatively, instead of defining a path, you can use “-” to skip a field to decrease cardinality.
nodes:
- name: <noode name>
type: resource_transform
target_source_type: docker | k8s | ecs | file | custom
source_field_overrides:
- field: <field name>
expression: <expression to determine fields value>
- field: <field name>
expression: <expression to determine fields value>