Add a Compound Node to a Pipeline

Add Compound Nodes to an Edge Delta Visual Pipeline.

  3 minute read  

Beta

Overview

Recap

You can create a segment of pipeline and save it as a single pipeline object called a Compound Node. This is useful for reusing common pipeline functionality and reducing pipeline complexity. For example, suppose logs across many pipelines require specific masking and a specific enrichment. You can configure the required mask and enrichment nodes in a Compound Node, and add that Compound Node object to the pipelines using Visual Pipelines. The values you used to create the mask and enrichment node are now applied in all pipelines where your Compound Node is used.

See Creating an Edge Delta Compound Node.

Use a Compound Node

Open a pipeline in which you want to add an existing Compound Node:

  1. Click Pipelines - Pipelines.
  2. Select the pipeline you want to add the Compound Node to.
  3. Click Edit Mode.
  4. Click Add Processor, expand Compound Nodes and select the Compound Node you want to add.
  5. Optionally, update the name.
  6. Optionally, click Edit Node to make changes to the contents of this Compound Node instance.

Adding a Compound Node adds an instance of it with its own name. If you make changes to the Compound Node instance in the pipeline, the original Compound Node you created earlier will not be updated. Similarly, if the original Compound Node is updated in the Compound Node editor, it will not change instances of that Compound Node that are saved in any pipelines. If a Compound Node is updated in the Compound Node editor and you want it in your pipeline, you need to delete the existing instance and re-add a new instance to the pipeline.

  1. Click OK.

Connect the Compound Node’s input and its outputs to the upstream and downstream nodes. These steps are taken in the following example:

  1. Delete an existing link where the Compound Node will reside.
  2. Connect the Input to the Compound Node.
  3. Connect the Processed output path of the Compound Node to the mask_ssn node.
  4. Connect the Passthrough output path of the Compound Node to the Raw_storage output node.

This configuration will add the Compound Node’s logic to the start of an existing pipeline. It will also output unprocessed logs to a separate output node.

  1. Click Review Changes.
  2. Click Deploy Changes.

The pipeline will be saved with the new Compound Node. If the pipeline has been used to install an agent, that agent’s configuration will be updated to include the new Compound Node.

Test a Compound Node

  1. Click Pipelines - Pipelines.
  2. Select the pipeline containing the Compound Node you want to test.
  3. Click Edit Mode.
  4. Select the Compound Node.
  5. Click Test Node.
  6. Select Processor.
  7. Paste a sample log in the Paste log data field.
  8. Select Test single processor node.
  9. Click Test Processor.

In this example the following log sample is tested against the Compound Node:

{"timestamp": "2024-02-21T18:32:49.619636Z", "node_id": "node11", "event":"exit", "logLevel": "TRACE", "request": {"method": "DELETE", "endpoint": "/api/v3/users/88", "headers": {"Host": "api1.myapp.com", "Authorization": "+psQPgKcS", "Accept": "application/json"}}, "response": {"status": 404, "time_ms": 271, "body": {"id": 59, "name": "user244", "email": "user934@example.com"}}, "userId": 5, "message": "Exiting from method processPaymentTransaction after 20ms"}

Note the event field in the JSON body and the presence of an email address. In the Outgoing Data Items section, in the Processed path, the event field has been added as an attribute and the email address has been obfuscated. Notice there is also a result in the Passthrough path, meaning the log has been duplicated and the duplicate has also been sent to an output but without processing.