Edge Delta Extract JSON Field Filter

See the latest version here.

Overview

The extract-json-field filter extracts a field’s value and replaces the whole JSON log content with only the field’s value.

When this filter is applied, the original JSON content is discarded. As a result, if the original JSON needs to be fed into another workflow or processor, then we recommend that you attach this filter to a processor and structure the workflow accordingly.

By default, non-JSON logs are dropped. However, you can choose to also ingest non-JSON logs and do trace logging on them by setting keep_log_if_failed to true. This option can be very useful when the agent has an input with mixed JSON and non-JSON logs. However, it will also allow other failure condition logs to be ingested.

Field Extraction Example

This example will extract the value of the message field and replace the JSON log with only the content of the message field. In addition, non-JSON logs will be ingested rather than dropped.

filters:
  - name: extract_message
    type: extract-json-field
    field_path: "message"
    keep_log_if_failed: true

Consider the following log:

{"timestamp":1623793757, "level": "info", "message": "Failed password for root from 22.44.55.55 port 28833 ssh2"}

After the filter is applied it will be reduced as follows:

Failed password for root from 22.44.55.55 port 28833 ssh2

Required Parameters

name

The name parameter specifies the name for the filter. You refer to this name in other places, for example to refer to a specific filter in a workflow or processor. Names must be unique within the filters: section. It is a YAML list element so it begins with a - and a space followed by the string. A name is a required parameter for a filter

filters:
  - name: <filter-name>

type: extract-json-field

The type parameter in the filter context specifies the type of filter to apply. A type is a required parameter for a filter.

filters:
  - name: <filter-name>
    type: <filter-type>

field_path

The field_path parameter is used to specify a value from a specific location in a JSON log. This value will be used to replace the entire log. It is written in JSONPath format as a dot separated path of the field. The field_path parameter can support indexing extraction and fetching all indexes with [*]. In that case all the items of the array will be ingested into the agent as separate logs. The field_path parameter is mandatory in an extract-json-field filter.

filters:
  - name: <filter-name>
    type: extract-json-field
    field_path: "<JSONPath>"

Optional Parameters

keep_log_if_failed

The keep_log_if_failed parameter defines how to handle an error when processing the filter. It is defined with a Boolean value true or false. The default value is false indicating that the filter will drop or ignore logs that caused an error. The keep_log_if_failed parameter is optional.

filters:
  - name: <filter-name>
    type: <filter-type>
    keep_log_if_failed: <true|false>