Edge Delta Transformation Nodes

Transformation nodes in v3 configurations.

2 minute read

Overview

Transformation nodes manipulate, enhance, and format log data in the pipeline before it is used for monitoring, analytics, or storage. Together, these nodes ensure that the data is optimized for actionable insights and a streamlined monitoring workflow.

Consider a basic log data item:


{
  "_type": "log"
  "attributes": {
    "newfield": "I added this new field value"
  }
  "body": "{"timestamp":"2023-04-23T12:34:56.789Z","logLevel":"ERROR","serviceName":"AuthService","nodeId":"node2","message":"Login failed","clientIP":"192.168.1.10","username":"user123","event":"login_attempt","outcome":"failure"}"
  "resource": {
    "config_id": "87654321-1321-69874-9456-s5123456h7"
    "ed.tag": "ed_parallel"
    "host.name": "ED_TEST"
    "ip": "10.0.0.1"
    "src_type": ""
  }
  "timestamp": 1703677287365
}

Each transformation node is designed for enriching different sections of the data item:

Node	Attribute	Resource	Body	Timestamp	Any Field	Restrictions
Output Transform	Y	Y	Y	Y	Y	Outputs a Custom type data item that cant be ingested by the Edge Delta Archive node. The whole payload is flattened and sent as the event, with all other fields empty.
Log Transform	Y	N	N	Y	N	Can only ingest logs, and it outputs only logs.
Resource Transform	N	Y	N	N	N	Can only ingest logs, and it outputs only logs.
Mask	N	N	Y	N	N	Can only ingest logs, and it outputs only logs. New value can only be a static string.
Generic Transform	Y	Y	N	Y	N	Can only transform non-body fields.
OTTL Transform	Y	Y	Y	Y	Y	Can transform any field on any data type.

The body field is protected from dynamic enrichment until the end of the pipeline (Output Transform) to prevent schema changes from disabling pipeline functionality.

Scenarios

To modify resource values that may have been changed after a log is generated (for example by downstream processing), consider using Resource Transform.
To add attributes to, for example, assist in log search or team ownership consider using Log Transform. It can upsert static values such as “owner”: “team12”, dynamic values using field paths such as item[“resources”][“host.ip”], or CEL macros.
To delete metrics attributes and resources to reduce cardinality consider using Output Transform.
To comply with data protection regulations such as CCPA and GDPR by masking sensitive data, consider using the Mask node.
To update the timestamp to match the body timestamp consider using Log Transform.

Edge Delta Enrichment Node

Apply one enrichment to all logs in a stream.

Edge Delta Generic Transform Node

Transform logs using a generic transformation on any non-body field.

Edge Delta Log Transform Node

Transform logs using upsert and delete operations.

Edge Delta Mask Node

Mask values in logs using the Edge Delta Mask Node.

Edge Delta OTTL Transform Node

Transform any data type using OTTL statements.

Edge Delta Output Transform Node

Transform and output logs.

Edge Delta Resource Transform Node

Transform resource fields.