Edge Delta Regex Filter

Overview

The Regex Filter node identifies and filters logs based on a regular expression (regex) pattern. The node isolates logs that contain specific patterns in the body field for processing, monitoring, or alerting purposes. For instance, if you are only interested in messages that contain a particular error code or keyword in the body, you can use a regex pattern designed to match that code or keyword. By default, the node will pass only those log entries that match the specified regex pattern. If the negate parameter is set to true, the filtering logic of the node is inverted: the node will block logs that match the specified pattern from being passed to the node’s output. This can be useful for excluding logs with known benign errors or irrelevant information, focusing the pipeline on more critical or interesting log data. The Regex Filter node searches the body log field for matches.

Conceptually, a Route node with only one path and no links to its unmatched path functions as a Regex Filter node. The benefit of a Route node is the ability to process multiple matching criteria for distinct node outputs, as well as the ability to handle unmatched logs for further processing on another output path. A Route node can evaluate logs based on CEL macros, not just regex, and it can evaluate all logs fields while a Regex Filter only evaluates the log body field.

• incoming_data_types: log
• outgoing_data_types: log

Example Configuration

nodes:
  - name: filter_with_regex
    type: regex_filter
    pattern: (?i)login failed

Suppose two logs are passed into the pipeline:

{
    "timestamp": "2023-04-23T12:34:56.789Z",
    "logLevel": "ERROR",
    "serviceName": "AuthService",
    "nodeId": "node2",
    "message": "Login failed",
    "clientIP": "192.168.1.10",
    "username": "user123",
    "event": "login_attempt",
    "outcome": "failure"
}

{
    "timestamp": "2023-04-23T12:35:20.543Z",
    "logLevel": "INFO",
    "serviceName": "AuthService",
    "nodeId": "node2",
    "message": "Login successful",
    "clientIP": "192.168.1.15",
    "username": "user456",
    "event": "login_attempt",
    "outcome": "success"
}

Only the following log is output by the pipeline.

{
  "timestamp":"2023-04-23T12:34:56.789Z",
  "logLevel":"ERROR",
  "serviceName":"AuthService",
  "nodeId":"node2",
  "message":"Login failed",
  "clientIP":"192.168.1.10",
  "username":"user123",
  "event":"login_attempt",
  "outcome":"failure"
}

Only the "message": "Login failed" log successfully passed the pipeline because the other log did not match the pattern.

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: regex_filter

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

pattern

The pattern parameter specifies a Golang regex pattern that the regex node will look for. It is a string that should be wrapped in quotes to handle escapes. A pattern is required for a regex filter node. See Regex Testing for details on writing effective regex patterns.

nodes:
  - name: <node name>
    type: regex_filter
    pattern: <regex pattern>

Optional Parameters

field_path

The field_path parameter specifies the dot-separated path to the field or nested field where the regex should be applied. This enables targeting specific fields within a log message for regex filtering. It is specified as a string.

nodes:
  - name: my_regex_filter
    type: regex_filter
    pattern: "Login failed"
    field_path: item["attributes"]["newfield"]

The field_path parameter is optional. If not specified the filter is applied to the default body field.

negate

The negate parameter inverses the filtering logic of the node. By default the node will identify and pass only those logs that match the specified pattern. With negate set to true, the node will block logs that match the specified pattern from being passed to the node output. It is a specified with a Boolean, the default value is false and it is optional.

nodes:
  - name: <node name>
    type: regex_filter
    pattern: <regex pattern>
    negate: true