Create an Edge Delta Compound Node
3 minute read
Overview
A compound node is an aggregation of pipeline nodes that you can add to a pipeline as a single object. This is useful if you have a common node pattern that needs to be inserted into many pipelines. For example, suppose many of your pipelines require a specific field to be masked and a particular enrichment applied. You can create a Compound Node containing the required Mask and Enrichment node configurations, and add that compound node in all the pipelines that require that logic. This reduces the visual complexity of busy pipelines and it reduces configuration overhead.
Use the following steps to create a new compound node from scratch. Alternatively, see Create a Compound Node from a Pipeline to create a compound node using an existing pipeline.
1. Create a Compound Node
To create a Compound Node:
- Click Pipelines and select the Compound Nodes tab.
- Click Add Compound Node.
The Compound Node builder opens. It looks similar to the Visual Pipelines builder.
- Click Edit in the Name field to specify a name for the Compound Node (optional).
- Click Edit in the Description field to describe the Compound Node function (optional).
2. Add Processors to the Compound Node
Add the required processors. For example, add a Mask node:
- Click Add Processor, expand Transformations and select Mask.
- Configure the mask processor by adding the Golang regex pattern to capture log content that needs to be masked, for example
\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b
to mask emails. - Click OK.
See Mask and the Mask Emails in Logs how-to for more information on configuring this type of node.
Continuing the example, add a Log Transform node:
- Click Add Processor, expand Transformations and select Log Transform.
- Specify a name and click Add New
- Enter a log field path for a field that should be created or updated, for example:
attributes.event
. - Select upsert from the Operation list.
- Enter a CEL expression that defines where to obtain the value that should be used for the new or updated field, for example:
json(item["body"]).event
- Click OK.
See Log Transform for more information on configuring this type of node.
4. Add Additional Outputs (Optional)
If the Compound Node requires more than one output, for example to deal with a split in the pipeline, you can add additional outputs:
- Click Add Destination.
- Specify a name for the output to describe that output path, for example
Passthrough
. This will be used to identify the output when viewing the Compound Node. - Click OK.
- Next, rename the default output if required: double click compound_output
- Enter
Processed
as the Name. - Click OK.
5. Connect the Nodes
Connect the processor nodes to the source, each other, and appropriate outputs. In this example the logs are duplicated along two paths with one path undergoing masking and log transformation before exiting via the processed output, while a copy of the original logs without masking or log transformation is passed directly from the input to the passthrough output:
- Delete the existing connection then connect compound_input to the mask node’s input path.
- Connect the mask node’s output path to the log_transform node’s input path.
- Connect the log_transform node’s output path to the Processed destination node.
- Connect the compound_input directly to the Passthrough destination node.
6. Save the new Compound Node.
- Click Review Changes.
- Click Save.
A Compound Node has been created. It has two paths so the logs will be duplicated to each path, one path processes logs by masking the email address and adding an event attribute, the other path does not modify the logs. Each path has its own output.
- Click Close.
Next Steps
Now you can add your Compound Node to one or more pipelines.