Filters Overview
  • Dark
    Light

Filters Overview

  • Dark
    Light

Overview

You can use a filter to identify or discard logs or to protect sensitive data. As a result, filters can help reduce the agent's resource load because of the reduced log ingestion. You can also use a filter to refine and transform collected logs before additional processing takes place. 


Note
Some input types offer additional filtering options that you can use to compliment the generic filters on this page. For example, when you create a Kubernetes input, you can add a filter in the input definition for a namespace or pod. 




Supported Filter Types

The Edge Delta App supports the following filter types:

Filter Type

Description

Base64

This filter type can be used to decode base64 encoding.

To learn more, see Base64 Filters.

Buffered Trace

This filter type handles trace logs.

  • Edge Delta defines trace log as a set of logs that can be tied together with an ID, such as a trace ID or request ID.

To learn more, see Buffered Trace Filters.

Combination 

This filter type allows you to combine already-existing filters into a single filter. 

To learn more, see Combination Filters.

Custom Attributes

This filter type can filter for custom attributes. 

To learn more, see Custom Attribute Filters.

EnrichmentThis filter type enriches data as it is being streamed through the Edge Delta agent. To learn more, see Enrichment. 
Enrichment JavaScriptThis filter type enriches logs using a JavaScript function as they pass through the Edge Delta agent. To learn more, see Enrichment JavaScript. 

JSON Field Drop

This filter type can filter and drop specified JSON fields from the incoming logs. 

To learn more, see JSON Field Drop Filters.

JSON Field Extractor

This filter type extracts a field's value and replaces the whole JSON content with the field's value.

To learn more, see JSON Field Extractor Filters.

Log Filter JavaScriptThis filter filters logs based on a condition that is defined using a JavaScript function. To learn more, see Log Filter JavaScript
Log Level DetectorThis filter identifies logs based on the log severity level. To learn more, see Log Level Detector. 
Log Transformation JavaScriptThis filter type uses Goja (a JavaScript type) to transform specified log messages. To learn more, see Log Transformer JavaScript. 

Mask

This filter type hides (or masks) specific data, based on the configured regex pattern.

To lean more, see Mask Filters.

Regex

This filter type passes all log lines that match the specified regular expression. All unmatched logs are discarded.

To learn more, see Regex Filters.

Split Lines

This filter type can be used to match, then split a single log into multiple logs.

To learn more, see Split Lines Filters.

Source DetectionThis filter type can be used to assign logs to sources using field mappings. For more information, see Source Detection

Create and Manage a Filter

To create and manage a filter, you must populate a YAML file.
To access the YAML file for a new configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
  2. Click Create Configuration.
  3. Click YAML.
  4. Enter your desired parameters, and then click Save.
    • To learn about these configurations, see Review Filter Types.
    • In a YAML file, filters are defined at the top level. Review the following example:
filters:
  - name: error
    type: regex
    pattern: "error"

To access the YAML file for an existing configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
  2. Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
  3. Review the YAML file, make your changes, and then click Save.
    • To learn about these configurations, see Review Filter Types.
    • In a YAML file, filters are defined at the top level. Review the following example:
filters:
  - name: error
    type: regex
    pattern: "error"

Understand the Workflow of a Filter

After you define a filter, filters can be referenced at different places in the YAML file:

  • Input filters apply right after the data ingestion from the input, but before running the workflows associated with the input.
  • Workflow filters apply before the processor runs within the workflow.
  • Processor filters apply before the processor runs, regardless of which workflow the processor is running within.

Inputs

The following example displays a file input with error and mask_card filters:

  inputs:
    files:
      - labels: "billing"
        path: "/var/log/billing/*.log"
        filters:
          - error
          - mask_card

To learn how inputs can be filtered, see Inputs Overview.


Workflows

The following example displays a workflow with the error filter:

workflows:
  application_workflow:
    input_labels:
      - system_stats
      - agent_stats
      - application_logs
    filters:
      - error
    processors:
      - error-check
      - fail-check
      - success-check
    destinations:
      - sumo-logic-devops-integration
      - slack-devops-integration

To learn how workflows can be filtered, see Workflows.


Processors

The following example displays the Dimension Counter Processor with the not_debug filter.

regexes:
  - name: "log"
    pattern: "level=(?P<level>\\w+) "
    dimensions: ["level"]
    trigger_thresholds:
      anomaly_probability_percentage: 90
    filters:
      - not_debug

To learn more, see Processors Overview.


Was this article helpful?

What's Next
Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.