Edge Delta Mask Processor
5 minute read
Overview
The mask processor obfuscates sensitive data in logs by replacing them with a specified set of characters, such as a series of asterisks or a custom string. Masking is important for compliance with various data protection regulations and for privacy concerns. Sensitive data is identified using a regex pattern. There are several patterns available out of the box for common regex patterns such as email, different types of credit card numbers etc. You can also create multiple different masks.
Configuration
In this example, the email address attribute’s value has been replaced with the default word REDACTED
.

In this example, the IP addresses have also been redacted:

In this example, there is an api token in the body and this has been parsed into the attributes. So you use a custom regex to capture and mask both values:

This is the input:
{
"_type": "log",
"timestamp": 1745456397792,
"body": "{\"timestamp\": \"2025-04-24T00:59:56.527714Z\", \"level\": \"Emergency\", \"msg\": \"Critical error in processing\", \"user\": {\"email\": \"barbara.martinez@imaginarymail.com\", \"id\": \"8d25f352-dcde-4753-84a8-45960dc99f90\", \"name\": \"d6426c02-cd40-4278-90a4-c167d4e23370\"}, \"request\": {\"ip\": \"10.29.168.111\", \"method\": \"PUT\", \"path\": \"/json/submit\"}, \"status\": 503, \"response_time_ms\": 12521, \"api_token\": \"sk_live_51NWz4nEXAMPLExQbG7nB2t6h8EpF3Df7oMBez\"}",
"resource": {
...
},
"attributes": {
"api_token": "sk_live_51NWz4nEXAMPLExQbG7nB2t6h8EpF3Df7oMBez",
"level": "Emergency",
"msg": "Critical error in processing",
"request": {
"ip": "10.29.168.111",
"method": "PUT",
"path": "/json/submit"
},
"response_time_ms": 12521,
"status": 503,
"timestamp": "2025-04-24T00:59:56.527714Z",
"user": {
"email": "barbara.martinez@imaginarymail.com",
"id": "8d25f352-dcde-4753-84a8-45960dc99f90",
"name": "d6426c02-cd40-4278-90a4-c167d4e23370"
}
}
}
Note: the resource field has been omitted reduced for brevity.
This is the pattern:
sk_(live|test)_[A-Za-z0-9]{20,}
This is the output:
{
"_type": "log",
"timestamp": 1745456397792,
"body": "{\"timestamp\": \"2025-04-24T00:59:56.527714Z\", \"level\": \"Emergency\", \"msg\": \"Critical error in processing\", \"user\": {\"email\": \"barbara.martinez@imaginarymail.com\", \"id\": \"8d25f352-dcde-4753-84a8-45960dc99f90\", \"name\": \"d6426c02-cd40-4278-90a4-c167d4e23370\"}, \"request\": {\"ip\": \"10.29.168.111\", \"method\": \"PUT\", \"path\": \"/json/submit\"}, \"status\": 503, \"response_time_ms\": 12521, \"api_token\": \"REDACTED\"}",
"resource": {
...
},
"attributes": {
"api_token": "REDACTED",
"level": "Emergency",
"msg": "Critical error in processing",
"request": {
"ip": "10.29.168.111",
"method": "PUT",
"path": "/json/submit"
},
"response_time_ms": 12521,
"status": 503,
"timestamp": "2025-04-24T00:59:56.527714Z",
"user": {
"email": "barbara.martinez@imaginarymail.com",
"id": "8d25f352-dcde-4753-84a8-45960dc99f90",
"name": "d6426c02-cd40-4278-90a4-c167d4e23370"
}
}
}
The pattern is designed to match any string that looks like an API secret key, regardless of where it appears in the log. In the log, “body” is a string containing escaped quotes. The actual token value itself is not escaped, it appears as-is inside the larger string. Similarly, “attributes” is regular JSON: the token value is again unescaped. This regex will match any substring in the input that looks like a token, whether it’s inside the “body” JSON-encoded string or a regular attribute.
For values that might contain special characters that are escaped in JSON (e.g. strings with quotes or backslashes), test how those appear in logs in both the body (as a string) and in parsed attributes. You may need either a more flexible regex or two patterns.
Options
Select a telemetry type
You can specify, log
, metric
, trace
or all
. It is specified using the interface, which generates a YAML list item for you under the data_types
parameter. This defines the data item types against which the processor must operate. If data_types is not specified, the default value is all
. It is optional.
It is defined in YAML as follows:
- name: multiprocessor
type: sequence
processors:
- type: <processor type>
data_types:
- log
Condition
The condition
parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional. You can select one of the following operators:
Operator | Name | Description | Example |
---|---|---|---|
== |
Equal to | Returns true if both values are exactly the same |
attributes["status"] == "OK" |
!= |
Not equal to | Returns true if the values are not the same |
attributes["level"] != "debug" |
> |
Greater than | Returns true if the left value is greater than the right |
attributes["duration_ms"] > 1000 |
>= |
Greater than or equal | Returns true if the left value is greater than or equal to the right |
attributes["score"] >= 90 |
< |
Less than | Returns true if the left value is less than the right |
attributes["load"] < 0.75 |
<= |
Less than or equal | Returns true if the left value is less than or equal to the right |
attributes["retries"] <= 3 |
matches |
Regex match | Returns true if the string matches a regular expression |
isMatch(attributes["name"], ".*\\.name$" |
It is defined in YAML as follows:
- name: _multiprocessor
type: sequence
processors:
- type: <processor type>
condition: attributes["request"]["path"] == "/json/view"
Predefined Regex Patterns
You can toggle on or off a number of predefined regex patterns such as email addresses, IP addresses etc. If your sensitive data is not covered by these you create a custom mask and define a regex pattern.
Create a custom mask
Click Create a custom mask to add a mask pattern manually. There are a number of patterns you can quickly select from the regex patterns library. Alternatively, define your own pattern and test it in the live capture output pane. With a custom mask you can also define your own mask characters (REDACTED
by default).
Excluded Fields
If you have fields that you want to exclude from masking, include them in the excluded fields section. Even if the pattern matches these fields and their children, they will not be masked.
Final
Determines whether successfully processed data items should continue through the remaining processors in the same processor stack. If final
is set to true
, data items output by this processor are not passed to subsequent processors within the node—they are instead emitted to downstream nodes in the pipeline (e.g., a destination). Failed items are always passed to the next processor, regardless of this setting.
The UI provides a slider to configure this setting. The default is false. It is defined in YAML as follows:
- name: multiprocessor
type: sequence
processors:
- type: <processor type>
final: true
See Also
- For an overview and to understand processor sequence flow, see Processors Overview
- To learn how to configure a processor, see Configure a Processor.
- For optimization strategies, see Best Practices for Edge Delta Processors.
- If you’re new to pipelines, start with the Pipeline Quickstart Overview or learn how to Configure a Pipeline.
- Looking to understand how processors interact with sources and destinations? Visit the Pipeline Overview.