Create an Edge Delta Compound Node

Create a Compound Node.

  3 minute read  

Beta

Overview

A compound node is an aggregation of pipeline nodes that you can add to a pipeline as a single object. This is useful if you have a common node pattern that needs to be inserted into many pipelines. For example, suppose many of your pipelines require a specific field to be masked and a particular enrichment applied. You can create a Compound Node containing the required Mask and Enrichment node configurations, and add that compound node in all the pipelines that require that logic. This reduces the visual complexity of busy pipelines and it reduces configuration overhead.

Use the following steps to create a new compound node from scratch. Alternatively, see Create a Compound Node from a Pipeline to create a compound node using an existing pipeline.

1. Create a Compound Node

To create a Compound Node:

  1. Click Pipelines - Compound Nodes.
  1. Click Add Compound Node.

The Compound Node builder opens. It looks similar to the Visual Pipelines builder.

  1. Click Edit in the Name field to specify a name for the Compound Node (optional).
  2. Click Edit in the Description field to describe the Compound Node function (optional).

2. Add Processors to the Compound Node

Add the required processors. For example, add a Mask node:

  1. Click Add Processor, expand Transformations and select Mask.
  2. Configure the mask processor by adding the Golang regex pattern to capture log content that needs to be masked, for example \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b to mask emails.
  3. Click OK.

See Mask and the Mask Emails in Logs how-to for more information on configuring this type of node.

Continuing the example, add a Log Transform node:

  1. Click Add Processor, expand Transformations and select Log Transform.
  2. Specify a name and click Add New
  3. Enter a log field path for a field that should be created or updated, for example: attributes.event.
  4. Select upsert from the Operation list.
  5. Enter a CEL expression that defines where to obtain the value that should be used for the new or updated field, for example: json(item["body"]).event
  6. Click OK.

See Log Transform for more information on configuring this type of node.

4. Add Additional Outputs (Optional)

If the Compound Node requires more than one output, for example to deal with a split in the pipeline, you can add additional outputs:

  1. Click Add Output.
  2. Specify a name for the output to describe that output path, for example Passthrough. This will be used to identify the output when viewing the Compound Node.
  3. Click OK.
  1. Next, rename the default output if required: double click compound_output
  2. Enter Processed as the Name.
  3. Click OK.

5. Connect the Nodes

Connect the processor nodes to the input, each other, and appropriate outputs. In this example the logs are duplicated along two paths with one path undergoing masking and log transformation before exiting via the processed output, while a copy of the original logs without masking or log transformation is passed directly from the input to the passthrough output:

  1. Delete the existing connection then connect compound_input to the mask node’s input path.
  2. Connect the mask node’s output path to the log_transform node’s input path.
  3. Connect the log_transform node’s output path to the Processed output node.
  4. Connect the compound_input directly to the Passthrough output node.

6. Save the new Compound Node.

  1. Click Review Changes.
  2. Click Save.

A Compound Node has been created. It has two paths so the logs will be duplicated to each path, one path processes logs by masking the email address and adding an event attribute, the other path does not modify the logs. Each path has its own output.

  1. Click Close.

Next Steps

Now you can add your Compound Node to one or more pipelines.