Edge Delta Delete Empty Field Processor

The Edge Delta Delete Empty Field Processor removes fields with no value to reduce traffic volume and improve data readability.

6 minute read

Overview

The delete empty field processor is used to clean up data items by removing extraneous fields. Typically empty fields are a result of upstream processing or the data source configuration. Removing these fields reduces traffic volume and makes the data items more readable.

Configuration

Consider the following log:

{
  "_type": "log",
  "timestamp": 1745463889804,
  "body": "{\"timestamp\": \"2025-04-24T00:59:56.527714Z\", \"level\": \"Emergency\", \"msg\": \"Critical error in processing\", \"user\": {\"email\": \"barbara.martinez@imaginarymail.com\", \"id\": \"8d25f352-dcde-4753-84a8-45960dc99f90\", \"name\": \"d6426c02-cd40-4278-90a4-c167d4e23370\"}, \"request\": {\"ip\": \"10.29.168.111\", \"method\": \"PUT\", \"path\": \"/json/submit\"}, \"status\": 503, \"response_time_ms\": 12521, \"api_token\": \"sk_live_51NWz4nEXAMPLExQbG7nB2t6h8EpF3Df7oMBez\", \"empty_string\": \"\", \"null_value\": null, \"empty_list\": [], \"empty_map\": {}}",
  "resource": {
...
  },
  "attributes": {
    "api_token": "sk_live_51NWz4nEXAMPLExQbG7nB2t6h8EpF3Df7oMBez",
    "empty_list": [],
    "empty_map": {},
    "empty_string": "",
    "level": "Emergency",
    "msg": "Critical error in processing",
    "null_value": null,
    "request": {
      "ip": "10.29.168.111",
      "method": "PUT",
      "path": "/json/submit"
    },
    "response_time_ms": 12521,
    "status": 503,
    "timestamp": "2025-04-24T00:59:56.527714Z",
    "user": {
      "email": "barbara.martinez@imaginarymail.com",
      "id": "8d25f352-dcde-4753-84a8-45960dc99f90",
      "name": "d6426c02-cd40-4278-90a4-c167d4e23370"
    }
  }
}

The body has been parsed which has resulted in an empty string field, a null value field, an empty list, and an empty map. Resources have been removed for brevity.

The following processor configuration is set to delete empty strings, nulls, empty lists, and empty maps:

This configuration generates the following YAML:

- name: Multi Processor_db04
  type: sequence
  processors:
  - type: delete_empty_values
    metadata: '{"id":"jrgX9yqsLSQJ7VqIO26xZ","type":"delete_empty_values","name":"Delete
      Empty Values"}'
    delete_empty_strings: true
    delete_empty_nulls: true
    delete_empty_lists: true
    delete_empty_maps: true

After being processed by the delete empty values processor:

{
  "_type": "log",
  "timestamp": 1745463889804,
  "body": "{\"timestamp\": \"2025-04-24T00:59:56.527714Z\", \"level\": \"Emergency\", \"msg\": \"Critical error in processing\", \"user\": {\"email\": \"barbara.martinez@imaginarymail.com\", \"id\": \"8d25f352-dcde-4753-84a8-45960dc99f90\", \"name\": \"d6426c02-cd40-4278-90a4-c167d4e23370\"}, \"request\": {\"ip\": \"10.29.168.111\", \"method\": \"PUT\", \"path\": \"/json/submit\"}, \"status\": 503, \"response_time_ms\": 12521, \"api_token\": \"sk_live_51NWz4nEXAMPLExQbG7nB2t6h8EpF3Df7oMBez\", \"empty_string\": \"\", \"null_value\": null, \"empty_list\": [], \"empty_map\": {}}",
  "resource": {
...
  },
  "attributes": {
    "api_token": "sk_live_51NWz4nEXAMPLExQbG7nB2t6h8EpF3Df7oMBez",
    "level": "Emergency",
    "msg": "Critical error in processing",
    "request": {
      "ip": "10.29.168.111",
      "method": "PUT",
      "path": "/json/submit"
    },
    "response_time_ms": 12521,
    "status": 503,
    "timestamp": "2025-04-24T00:59:56.527714Z",
    "user": {
      "email": "barbara.martinez@imaginarymail.com",
      "id": "8d25f352-dcde-4753-84a8-45960dc99f90",
      "name": "d6426c02-cd40-4278-90a4-c167d4e23370"
    }
  }
}

The empty_list, empty_map, empty_string and null_value fields have been deleted resulting in a 4% decrease in traffic volume.

Note: the original unparsed versions still exist in the body.

Options

Select a telemetry type

You can specify, log, metric, trace or all. It is specified using the interface, which generates a YAML list item for you under the data_types parameter. This defines the data item types against which the processor must operate. If data_types is not specified, the default value is all. It is optional.

It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    data_types:
    - log

Condition

The condition parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional.

Important: All conditions must be written on a single line in YAML. Multi-line conditions are not supported.

Comparison Operators

Operator	Name	Description	Example
`==`	Equal to	Returns `true` if both values are exactly the same	`attributes["status"] == "OK"`
`!=`	Not equal to	Returns `true` if the values are not the same	`attributes["level"] != "debug"`
`>`	Greater than	Returns `true` if the left value is greater than the right	`attributes["duration_ms"] > 1000`
`>=`	Greater than or equal	Returns `true` if the left value is greater than or equal to the right	`attributes["score"] >= 90`
`<`	Less than	Returns `true` if the left value is less than the right	`attributes["load"] < 0.75`
`<=`	Less than or equal	Returns `true` if the left value is less than or equal to the right	`attributes["retries"] <= 3`
`matches`	Regex match	Returns `true` if the string matches a regular expression (generates `IsMatch` function)	`IsMatch(attributes["name"], ".*\\.log$")`

Logical Operators

Important: Use lowercase and, or, not - uppercase operators will cause errors!

Operator	Description	Example
`and`	Both conditions must be true	`attributes["level"] == "ERROR" and attributes["status"] >= 500`
`or`	At least one condition must be true	`attributes["log_type"] == "TRAFFIC" or attributes["log_type"] == "THREAT"`
`not`	Negates the condition	`not regex_match(attributes["path"], "^/health")`

Functions

Function	Description	Example
`regex_match`	Returns `true` if string matches the pattern	`regex_match(attributes["message"], "ERROR\\|FATAL")`
`IsMatch`	Alternative regex function (UI generates this from “matches” operator)	`IsMatch(attributes["name"], ".*\\.log$")`

Field Existence Checks

Check	Description	Example
`!= nil`	Field exists (not null)	`attributes["user_id"] != nil`
`== nil`	Field doesn’t exist	`attributes["optional_field"] == nil`
`!= ""`	Field is not empty string	`attributes["message"] != ""`

Common Examples

- name: _multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    # Simple equality check
    condition: attributes["request"]["path"] == "/json/view"
    
  - type: <processor type>
    # Multiple values with OR
    condition: attributes["log_type"] == "TRAFFIC" or attributes["log_type"] == "THREAT"
    
  - type: <processor type>
    # Excluding multiple values (NOT equal to multiple values)
    condition: attributes["log_type"] != "TRAFFIC" and attributes["log_type"] != "THREAT"
    
  - type: <processor type>
    # Complex condition with AND/OR/NOT
    condition: (attributes["level"] == "ERROR" or attributes["level"] == "FATAL") and attributes["env"] != "test"
    
  - type: <processor type>
    # Field existence and value check
    condition: attributes["user_id"] != nil and attributes["user_id"] != ""
    
  - type: <processor type>
    # Regex matching using regex_match
    condition: regex_match(attributes["path"], "^/api/") and not regex_match(attributes["path"], "^/api/health")
    
  - type: <processor type>
    # Regex matching using IsMatch
    condition: IsMatch(attributes["message"], "ERROR|WARNING") and attributes["env"] == "production"

Common Mistakes to Avoid

# WRONG - Cannot use OR/AND with values directly
condition: attributes["log_type"] != "TRAFFIC" OR "THREAT"

# CORRECT - Must repeat the full comparison
condition: attributes["log_type"] != "TRAFFIC" and attributes["log_type"] != "THREAT"

# WRONG - Uppercase operators
condition: attributes["status"] == "error" AND attributes["level"] == "critical"

# CORRECT - Lowercase operators
condition: attributes["status"] == "error" and attributes["level"] == "critical"

# WRONG - Multi-line conditions
condition: |
  attributes["level"] == "ERROR" and 
  attributes["status"] >= 500  

# CORRECT - Single line (even if long)
condition: attributes["level"] == "ERROR" and attributes["status"] >= 500

Delete options

Toggle on or off the options to delete different types of empty values:

Delete empty strings
Delete nulls
Delete empty lists
Delete empty maps

Strings to delete

Use this option to delete fields that contain a particular static value. Such as an unwanted resource field.

Excluded Fields

Specify fields in with this option that should not be deleted even if they match the other conditions set in the processor, such as having an empty string.

Final

Determines whether successfully processed data items should continue through the remaining processors in the same processor stack. If final is set to true, data items output by this processor are not passed to subsequent processors within the node—they are instead emitted to downstream nodes in the pipeline (e.g., a destination). Failed items are always passed to the next processor, regardless of this setting.

The UI provides a slider to configure this setting. The default is false. It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
    - type: <processor type>
    final: true