Create an Edge Delta Pack

  4 minute read  

Overview

A pack is an aggregation of pipeline nodes that you can add to a pipeline as a single object. This is useful if you have a common node pattern that needs to be inserted into many pipelines. For example, suppose many of your pipelines require a specific field to be masked and a particular enrichment applied. You can create a pack containing the required Mask and Enrichment node configurations, and add that pack in all the pipelines that require that logic. This reduces the visual complexity of busy pipelines and it reduces configuration overhead.

Use the following steps to create a new pack from scratch. Alternatively, see Create a Pack from a Pipeline to create a pack using an existing pipeline. You can also add an out of the box pack from the packs library.

1. Create a Pack

To create a pack:

1. Click Pipelines and select the Packs tab.

2. Click Add Pack.

The pack builder opens. It looks similar to the Visual Pipelines builder.

3. Click Edit in the Name field to specify a name for the pack (optional).
4. Click Edit in the Description field to describe the pack function (optional).

2. Add Processors to the Pack

Add the required processors. For example, add a Mask node:

1. Click Add Processor, expand Transformations and select Mask.
2. Configure the mask processor by adding the Golang regex pattern to capture log content that needs to be masked, for example \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b to mask emails.
3. Click OK.

See Mask and the Mask Emails in Logs how-to for more information on configuring this type of node.

Continuing the example, add a Log Transform node:

4. Click Add Processor, expand Transformations and select Log Transform.
5. Specify a name and click Add New
6. Enter a log field path for a field that should be created or updated, for example: attributes.event.
7. Select upsert from the Operation list.
8. Enter a CEL expression that defines where to obtain the value that should be used for the new or updated field, for example: json(item["body"]).event
9. Click OK.

See Log Transform for more information on configuring this type of node.

4. Add Additional Outputs (Optional)

If the pack requires more than one output, for example to deal with a split in the pipeline, you can add additional outputs:

1. Click Add Destination.
2. Specify a name for the output to describe that output path, for example Passthrough. This will be used to identify the output when viewing the pack.
3. Click OK.
4. Next, rename the default output if required: double click compound_output
5. Enter Processed as the Name.
6. Click OK.

5. Connect the Nodes

Connect the processor nodes to the source, each other, and appropriate outputs. In this example the logs are duplicated along two paths with one path undergoing masking and log transformation before exiting via the processed output, while a copy of the original logs without masking or log transformation is passed directly from the input to the passthrough output:

1. Delete the existing connection then connect compound_input to the mask node’s input path.
2. Connect the mask node’s output path to the log_transform node’s input path.
3. Connect the log_transform node’s output path to the Processed destination node.
4. Connect the compound_input directly to the Passthrough destination node.

6. Save the new Pack.

1. Click Review Changes.
2. Click Save.

A pack has been created. It has two paths so the logs will be duplicated to each path, one path processes logs by masking the email address and adding an event attribute, the other path does not modify the logs. Each path has its own output.

3. Click Close.

7. (Optionally) Export or Import

You can export a pack as a YAML file, and import existing pack configurations:

• To export a pack, click the kebab (⋮) icon in the Actions column for the pack and select Export as YAML.
• To import a pack, click Import and select the YAML file to add to your organization’s pack table.

Next Steps

Now you can add your pack to one or more pipelines.