Edge Delta Filter Processor
4 minute read
Overview
You can drop data items based on a specific condition in each data item. For example, you can drop activity performed by a test user.
See Pipeline Quickstart: Filter Data Items for a walkthrough.
Configuration
In this example, all logs with user email jane.smith@exampleemail.com
are dropped.

- name: Multi Processor
type: sequence
processors:
- type: ottl_filter
metadata: '{"id":"C583fhTKWFgbhdWCZOXEz","type":"filter","name":"Filter out test
user Jane"}'
condition: attributes["user"]["email"] == "jane.smith@exampleemail.com"
data_types:
- log
filter_mode: exclude
Options
Select a telemetry type
You can specify, log
, metric
, trace
or all
. It is specified using the interface, which generates a YAML list item for you under the data_types
parameter. This defines the data item types against which the processor must operate. If data_types is not specified, the default value is all
. It is optional.
It is defined in YAML as follows:
- name: multiprocessor
type: sequence
processors:
- type: <processor type>
data_types:
- log
Action
This option determines what the processor does with matching vs non-matching data items. Include means matching data items are passed and all others are dropped. Exclude means matching data items are dropped and all others are passed. The default is exclude. It is populated for you in YAML using the filter_mode
parameter.
It is defined in YAML as follows:
- name: Multi Processor
type: sequence
processors:
- type: ottl_filter
filter_mode: exclude
In this example, the action is set to Include so only logs containing the jane.smith@exampleemail.com
attribute are dropped.

condition
The condition
parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional.
Important: All conditions must be written on a single line in YAML. Multi-line conditions are not supported.
Comparison Operators
Operator | Name | Description | Example |
---|---|---|---|
== |
Equal to | Returns true if both values are exactly the same |
attributes["status"] == "OK" |
!= |
Not equal to | Returns true if the values are not the same |
attributes["level"] != "debug" |
> |
Greater than | Returns true if the left value is greater than the right |
attributes["duration_ms"] > 1000 |
>= |
Greater than or equal | Returns true if the left value is greater than or equal to the right |
attributes["score"] >= 90 |
< |
Less than | Returns true if the left value is less than the right |
attributes["load"] < 0.75 |
<= |
Less than or equal | Returns true if the left value is less than or equal to the right |
attributes["retries"] <= 3 |
matches |
Regex match | Returns true if the string matches a regular expression (generates IsMatch function) |
IsMatch(attributes["name"], ".*\\.log$") |
Logical Operators
Important: Use lowercase and
, or
, not
- uppercase operators will cause errors!
Operator | Description | Example |
---|---|---|
and |
Both conditions must be true | attributes["level"] == "ERROR" and attributes["status"] >= 500 |
or |
At least one condition must be true | attributes["log_type"] == "TRAFFIC" or attributes["log_type"] == "THREAT" |
not |
Negates the condition | not regex_match(attributes["path"], "^/health") |
Functions
Function | Description | Example |
---|---|---|
regex_match |
Returns true if string matches the pattern |
regex_match(attributes["message"], "ERROR\|FATAL") |
IsMatch |
Alternative regex function (UI generates this from “matches” operator) | IsMatch(attributes["name"], ".*\\.log$") |
Field Existence Checks
Check | Description | Example |
---|---|---|
!= nil |
Field exists (not null) | attributes["user_id"] != nil |
== nil |
Field doesn’t exist | attributes["optional_field"] == nil |
!= "" |
Field is not empty string | attributes["message"] != "" |
Common Examples
- name: _multiprocessor
type: sequence
processors:
- type: <processor type>
# Simple equality check
condition: attributes["request"]["path"] == "/json/view"
- type: <processor type>
# Multiple values with OR
condition: attributes["log_type"] == "TRAFFIC" or attributes["log_type"] == "THREAT"
- type: <processor type>
# Excluding multiple values (NOT equal to multiple values)
condition: attributes["log_type"] != "TRAFFIC" and attributes["log_type"] != "THREAT"
- type: <processor type>
# Complex condition with AND/OR/NOT
condition: (attributes["level"] == "ERROR" or attributes["level"] == "FATAL") and attributes["env"] != "test"
- type: <processor type>
# Field existence and value check
condition: attributes["user_id"] != nil and attributes["user_id"] != ""
- type: <processor type>
# Regex matching using regex_match
condition: regex_match(attributes["path"], "^/api/") and not regex_match(attributes["path"], "^/api/health")
- type: <processor type>
# Regex matching using IsMatch
condition: IsMatch(attributes["message"], "ERROR|WARNING") and attributes["env"] == "production"
Common Mistakes to Avoid
# WRONG - Cannot use OR/AND with values directly
condition: attributes["log_type"] != "TRAFFIC" OR "THREAT"
# CORRECT - Must repeat the full comparison
condition: attributes["log_type"] != "TRAFFIC" and attributes["log_type"] != "THREAT"
# WRONG - Uppercase operators
condition: attributes["status"] == "error" AND attributes["level"] == "critical"
# CORRECT - Lowercase operators
condition: attributes["status"] == "error" and attributes["level"] == "critical"
# WRONG - Multi-line conditions
condition: |
attributes["level"] == "ERROR" and
attributes["status"] >= 500
# CORRECT - Single line (even if long)
condition: attributes["level"] == "ERROR" and attributes["status"] >= 500
s
See Also
- For an overview and to understand processor sequence flow, see Processors Overview
- To learn how to configure a processor, see Configure a Processor.
- For optimization strategies, see Best Practices for Edge Delta Processors.
- If you’re new to pipelines, start with the Pipeline Quickstart Overview or learn how to Configure a Pipeline.
- Looking to understand how processors interact with sources and destinations? Visit the Pipeline Overview.