Route Logs in a Branched Pipeline

Route logs in a branched pipeline in 5 Minutes.

  3 minute read  

Overview

The Route node in Edge Delta functions as a conditional traffic director within a log processing pipeline. This node evaluates specified conditions and directs logs to distinct paths based on whether each log meets the conditions. For example, logs indicative of errors such as failed transactions in a payment service should be sent to a path for alerting and immediate attention. On the other hand, logs related to inventory updates might be sent to a separate path for data warehousing and trend analysis. While logs related to user validation errors might need to be routed elsewhere for security.

Conceptually, a Route node with only one path and no links to its unmatched path functions as a Regex Filter node. The benefit of a Route node is the ability to process multiple matching criteria for distinct node outputs, as well as the ability to handle unmatched logs for further processing on another output path. A Route node can evaluate logs based on CEL macros, not just regex, and it can evaluate all logs fields while a Regex Filter only evaluates the log body field.

Prerequisites

To use route logs in a pipeline you need an Edge Delta account with an agent configuration already created. This is the configuration in which you will create the Route node.

Create the Route Node

  1. In the Edge Delta App, click Pipelines - Pipelines.
  2. Select the pipeline you want to edit.
  3. Click Edit Mode.
  4. Click Add Processor, expand Filters and select Route.
  1. Specify a Name for the file input node route_test.
  2. Click Add New in the Paths section.
  3. Enter a suitable name for the first path such as PaymentService_Path in the Path field.
  4. Specify a suitable CEL macro in the Condition field, such as the following one which identifies logs containing the word PaymentService:
regex_match(item["body"], "PaymentService")
  1. Click Add New in the Paths section.
  2. Enter a suitable name for the second path such as AuthService_Path in the Path field.
  3. Specify a suitable CEL macro in the Condition field, such as the following one which identifies logs containing the word AuthService:
regex_match(item["body"], "AuthService")
  1. Click OK.
  1. Connect the data source that emits both services’ logs to the input of the Route node.
  2. Connect the first output on the Route node, such as PaymentService_Path to the appropriate downstream node for processing of PaymentService logs.
  3. Connect the second output on the Route node, such as AuthService_Path to the appropriate downstream node for processing of AuthService logs.
  4. Click Review Changes.
  5. Click Deploy Changes.

Logs containing PaymentService will now be routed along the first pipeline path, those containing AuthService will be routed along the second path, while all others will be passed to the unmatched path and dropped.