Edge Delta Lookup Processor

The Edge Delta lookup processor enriches a data item dynamically against a lookup table.

5 minute read

Overview

You can enrich data items dynamically using a lookup table. This is useful for enriching data based on multiple criteria. For example, you can enrich data items that contain codes with attributes that provide the code definitions based on a table of all possible codes and their definitions. You can host the lookup table in Edge Delta in your own location.

See how to use lookup tables.

Configuration

Suppose your logs contain FTD codes for errors. You can use a lookup table that provides an explanation of the code and the recommended action as attributes.

Consider this log:

<80>Apr 22 02:07:40 securegateway01 %FTD-1-104002: (Primary) Switching to STANDBY (cause: bad/incomplete config).

The following processor checks a lookup table for matching keys:

The following row is discovered in the table:

FTD Code, Explanation, Recommended Action
%FTD-1-104002, You have forced the failover pair to switch roles either by entering the failover active command on the standby unit or the no failover active command on the active unit, If the message occurs because of manual intervention no action is required. Otherwise use the cause reported by the secondary unit to verify the status of both units of the pair

The FTD Code row matches the %FTD-1-104002 code in the log body.

Logs containing this code are now populated with the additional attributes:

The processor YAML is as follows:

- name: Multi Processor
  type: sequence
  processors:
  - type: lookup
    metadata: '{"id":"Ayp5ZStWEwntpQjvJXA1-","type":"lookup","name":"Lookup"}'
    location_path: ed://ftd_code_explanation_action.csv
    reload_period: 10m0s
    match_mode: regex
    key_fields:
    - event_field: body
      lookup_field: FTD Code
    out_fields:
    - event_field: attributes["ftd_explanation"]
      lookup_field: Explanation
    - event_field: attributes["ftd_action"]
      lookup_field: Recommended Action

Options

Select a telemetry type

You can specify, log, metric, trace or all. It is specified using the interface, which generates a YAML list item for you under the data_types parameter. This defines the data item types against which the processor must operate. If data_types is not specified, the default value is all. It is optional.

It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    data_types:
    - log

condition

The condition parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional. You can select one of the following operators:

Operator	Name	Description	Example
`==`	Equal to	Returns `true` if both values are exactly the same	`attributes["status"] == "OK"`
`!=`	Not equal to	Returns `true` if the values are not the same	`attributes["level"] != "debug"`
`>`	Greater than	Returns `true` if the left value is greater than the right	`attributes["duration_ms"] > 1000`
`>=`	Greater than or equal	Returns `true` if the left value is greater than or equal to the right	`attributes["score"] >= 90`
`<`	Less than	Returns `true` if the left value is less than the right	`attributes["load"] < 0.75`
`<=`	Less than or equal	Returns `true` if the left value is less than or equal to the right	`attributes["retries"] <= 3`
`matches`	Regex match	Returns `true` if the string matches a regular expression	`isMatch(attributes["name"], ".*\\.name$"`

It is defined in YAML as follows:

- name: _multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    condition: attributes["request"]["path"] == "/json/view"

Location

You define the location of the lookup table. You can specify a lookup table hosted in Edge Delta, a file on the cluster, or Other for a URL. If you select an Edge Delta lookup table you can select it from a list. If you select File you enter the filename and path. Or you specify the URL for other.

The tool populates the location_path parameter in the YAML. This field is mandatory and the format is as follows depending on the location type:

"file://<path>"
"ed://<file name in ED stored lookup>"
"(http|https)://<URL to CSV>"

Reload Period

This option is used to specify how often the lookup table is reloaded. It is defined as a duration and defaults to 5 minutes if not specified. The tool populates the reload_period parameter in YAML.

Match mode

You can choose how to match the lookup key field. If the table contains regex patterns that will be used to match to events, select regex. If it will match on a static string, select exact. The tool populates the match_mode parameter with either exact or regex. The default is exact.

key_fields

The key_fields are pairs that map event fields to lookup fields to find matches.

For key_fields, the event_field specifies the key value in the log and binds it to the lookup_field. For each log, the node will extract the event_field value using the event field’s pattern and compare it to each value in lookup_field for a match.

See how to use lookup tables for information on how the key_fields bind a log field and a table field.

out_fields

The out_fields define mappings from lookup table to event attributes for enrichment upon successful matches. Supports default_value for no match, and append_mode if multiple rows are matched.

For out_fields, there are two binding pairs: For each, a new attribute will be created based on the event_field, and its value will be extracted from the lookup_field - for all rows matched by the key_field parameter.

See how to use lookup tables for information on how the out_fields bind a log field and a table field.

Final

Determines whether successfully processed data items should continue through the remaining processors in the same processor stack. If final is set to true, data items output by this processor are not passed to subsequent processors within the node—they are instead emitted to downstream nodes in the pipeline (e.g., a destination). Failed items are always passed to the next processor, regardless of this setting.

The UI provides a slider to configure this setting. The default is false. It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
    - type: <processor type>
    final: true