Filter Out Debug Logs with Edge Delta

Filter Out Debug Logs with Edge Delta in 5 Minutes.

  3 minute read  

Overview

The regex filter node identifies and filters logs based on a regular expression (regex) pattern. The node isolates logs that contain specific patterns in the body field for processing, monitoring, or alerting purposes. By default, the node will pass only those log entries that match the specified regex pattern but this logic can be reversed - to pass only those log entries that don’t match the specified regex pattern . The Regex Filter node is useful for excluding debug logs from the pipeline, so you can focus on more critical or interesting log data. The Regex Filter node searches the body log field for matches.

Conceptually, a Route node with only one path and no links to its unmatched path functions as a Regex Filter node. The benefit of a Route node is the ability to process multiple matching criteria for distinct node outputs, as well as the ability to handle unmatched logs for further processing on another output path. A Route node can evaluate logs based on CEL macros, not just regex, and it can evaluate all logs fields while a Regex Filter only evaluates the log body field.

Prerequisites

To use a Regex Filter in a pipeline you need an Edge Delta account with a Pipeline configuration already created. This is the configuration in which you will create the Route node.

Scenario

Suppose your data source creates debug level logs that make your observability data noisy. Rather than turning off debug level logging at the source you can drop those logs in the Edge Delta pipeline.

An alternative solution is to use a Route node. This approach allows you to keep the debug level logs on one path for cold storage historical context, while removing them on another path that has a hot storage destination. See Route Logs in a Branched Pipeline.

Know Your Data

To effectively design and manage a pipeline you need to be familiar with your data. Consider these fictitious logs:

6144:M 20/Aug/2024:03:26:42 +0000 # DEBUG: Background saving started, forking child process
30866:M 20/Aug/2024:03:26:42 +0000 # VERBOSE: Command 'DEL' executed on 3 keys, freeing 2048 bytes of memory
1385:S 20/Aug/2024:03:26:42 +0000 # DEBUG: Detailed memory allocation report: 1024 bytes allocated for new connection handler
40698:M 20/Aug/2024:03:26:42 +0000 # DEBUG: Connection from IP 203.0.113.47 accepted on port 6379
40698:M 20/Aug/2024:03:26:42 +0000 # ERROR: Connection timeout on port 6379. Consider enabling detailed DEBUG logging for diagnostic purposes.

The logs describe a series of server activities. They structured with identifiers for the process and type, a detailed timestamp, specified logging levels (DEBUG, VERBOSE and ERROR), concise descriptions of actions performed, and mention of system resources and network interactions.

Of interest is the log level field. It starts with a # character, contains uppercase letters and ends with a : character.

Create the Regex Filter Node

  1. In the Edge Delta App, click Pipelines.
  2. Select the fleet you want to edit and click View/Edit Pipeline.
  3. Click Edit Mode.
  4. Click Add Processor, expand Filters and select Regex Filter.
  5. Specify a Name for the Regex Filter node drop-debug.
  6. Paste the sample data provided in the Know Your Data section into the Samples field.
  7. Specify a suitable regex pattern, in this instance:
# DEBUG:

Note: A simple debug|DEBUG pattern would over-match on the ERROR level log by matching on the word debug in the message field, causing it to also be dropped.

  1. Select Filter Out Matches True.
  2. Click Save Changes.
  1. Connect the data source that emits debug logs to the input of the Regex Filter node.
  2. Connect the output on the Regex Filter node to the appropriate downstream node for further processing, in this instance ed_log_output.
  3. Click Review Changes.
  4. Click Save Changes.

Logs containing DEBUG in the log level field of the body will now be dropped, and all other logs will be passed along the pipeline.