Edge Delta Generic Transform Node

Transform logs using a generic transformation on any non-body field.

  4 minute read  

Beta

Overview

The generic transform node can upsert or delete all fields except “body” and top level fields starting with “_”. It can transform many different data types:

Note: Changes to some of these data items may result in the Edge Delta features not functioning as expected.

Each transformation node is designed for enriching different sections of the data item:

Node Attribute Resource Body Timestamp Any Field Restrictions
Output Transform Y Y Y Y Y Outputs a Custom type data item that cant be ingested by the Edge Delta Archive node. The whole payload is flattened and sent as the event, with all other fields empty.
Log Transform Y N N Y N Can only ingest logs, and it outputs only logs.
Resource Transform N Y N N N Can only ingest logs, and it outputs only logs.
Mask N N Y N N Can only ingest logs, and it outputs only logs. New value can only be a static string.
Generic Transform Y Y N Y N Can only transform non-body fields.

The body field is protected from dynamic enrichment until the end of the pipeline (Output Transform) to prevent schema changes from disabling pipeline functionality.

Example Configuration

nodes:
- name: Generic Transform
  type: generic_transform
  transformations:
  - field_path: attributes.host/.name
    operation: upsert
    value: 'regex_capture(item["body"], "\\\"host\\\": \\\"(?P<host>[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})\\\"").host'
  - field_path: attributes.metadata
    operation: upsert
    value: '"Project:Phoenix"'
  - field_path: attributes.datetime
    operation: upsert
    value: convert_timestamp(item["timestamp"], "Unix Milli", "2006-01-02T15:04:05.000Z")

This node performs three transformations:

  1. The attributes.host/.name field is upserted using the value derived from a CEL macro: The macro captures the host from the body, handling escape sequences for JSON string parsing. If the field exists and the capture is valid, the host information will be updated accordingly.
  2. The attributes.metadata field is upserted with a static value “Project:Phoenix”. It provides a label or category for the log, signifying its association with a specific project.
  3. The attributes.datetime field is upserted with a formatted version of the timestamp. It is calculated using a CEL macro that converts the Unix Millisecond timestamp into an ISO 8601 formatted date-time string, making it human-readable. This will create a new field with a clear and understandable timestamp.

Input

{
  "id": "669a5fb7-5fb1-4fbf-9dda-58cf369fae9a",
  "timestamp": 1730079992826,
  "severity_text": "",
  "body": {
    "host": "36.46.105.83",
    "user-identifier": "-",
    "time_local": "28/10/2024:01:46:32 +0000",
    "method": "DELETE",
    "request": "/portals",
    "protocol": "HTTP/1.0",
    "status": 400,
    "bytes_sent": 17826
  },
  "resource": {
    ...
  },
  "attributes": {
    ...
  }
}

Output

{
  "id": "669a5fb7-5fb1-4fbf-9dda-58cf369fae9a",
  "timestamp": 1730079992826,
  "severity_text": "",
  "body": {
    "host": "36.46.105.83",
    "user-identifier": "-",
    "time_local": "28/10/2024:01:46:32 +0000",
    "method": "DELETE",
    "request": "/portals",
    "protocol": "HTTP/1.0",
    "status": 400,
    "bytes_sent": 17826
  },
  "resource": {
    ...
  },
  "attributes": {
    "datetime": "1970-01-21T00:34:39.000Z",
    ...,
    "host.name": "36.46.105.83",
    "metadata": "Project:Phoenix"
  }
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: generic_transform

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Transformations

The transformations parameter is used to specify the transformation operations. It consists of three child parameters:

  • operationis used to specify the transformation operation. Currently it can be delete or upsert, which updates the field if it exists or adds it if it doesn’t exist.
  • field_path is the dot separated path where the operation should be applied.
  • value is the CEL expression that determines the value to be applied and you can use CEL macros. With the OTEL schema the indexing method of CEL must be used, for example item["resource"]["host.name"] because OTEL has some fields that already contain dots in them.
nodes:
  - name: <node name>
    type: ​​log_transform
    transformations:
    - operation: upsert|delete
      field_path: <dot separated path>
      value: <CEL expression>

Optional Parameters

ignore_if_empty

The ignore_if_empty parameter determines whether to perform an upsert if the value is null. It is configured as a Boolean and the default is false. If false, empty values will be added to the payload. If true, the value will be omitted when it is empty. The ignore_if_empty parameter requires the upsert operation and it is optional.

nodes:
  - name: <node name>
    type: ​​log_transform
    transformations:
    - operation: upsert
      field_path: <dot separated path>
      value: <CEL expression>
      ignore_if_empty: true