Edge Delta Lookup Node

Enrich logs based on fields in a lookup table.

  6 minute read  

Beta

Overview

You can enrich logs dynamically using a lookup table. This is useful for enriching logs based on multiple criteria. For example, you can enrich logs that contain codes with attributes that provide the code definitions based on a table of all possible codes and their definitions.

See how to use lookup tables.

Example Configuration

In this example, certain Cisco FTD logs that are alert level severity (FTD-1) are enriched with a description of the FTD code and the recommended steps to remedy the issue. FTD logs with codes that are not in the lookup table are passed without enrichment.

Suppose a lookup table like this extract is uploaded into the Lookups library in Edge Delta:

FTD Code,Explanation,Recommended Action
%FTD-1-104004,A previously failed unit reports that it is operating again,No action required
%FTD-1-101005,The failover cable is connected but the primary unit is unable to determine its status,Replace the cable

It is named ftd_code_explanation_action.csv:

In this example, a Lookup node is configured to match logs based on the first column FTD Code, and enrich matching logs with columns two (Explanation) and three (Recommended Action) for the row containing the matching FTD Code:

nodes:
- name: FTD Code
  type: lookup
  location_path: ed://ftd_code_explanation_action.csv
  reload_period: 5m0s
  match_mode: exact
  regex_option: first
  key_fields:
  - event_field: regex_capture(item["body"], "(?P<FTDCode>%FTD-\\d-\\d{6})")["FTDCode"]
    lookup_field: FTD Code
  out_fields:
  - event_field: item["attributes"]["Explanation"]
    lookup_field: Explanation
  - event_field: item["attributes"]["Recommended_Action"]
    lookup_field: Recommended Action

Explanation:

  • location_path: specifies where the lookup data resides, in this case, a CSV file named ftd_code_explanation_action.csv managed in the Edge Delta interface
  • reload_period: the lookup table is reloaded every 5 minutes to ensure that the pipeline uses updated data.
  • match_mode: specifies the matching mode to be used when looking up keys. exact indicates that the keys must match exactly as they appear in the lookup table.
  • regex_option: specifies how the regex should be applied. first configures the node to use the first match found by the regex in the given input.
  • key_fields: a list of mappings between keys in the incoming event and fields in the lookup table.
    • event_field: a CEL macro regex_capture to extract a value named FTDCode from the body of the event using a regex pattern (?P<FTDCode>%FTD-\\d-\\d{6}). The pattern captures strings like %FTD-1-104004, where %FTD- is followed by one digit, a hyphen, and exactly six digits.
      • lookup_field: specifies the field (column) in the lookup table that should be used as the key for matching against the FTDCode extracted from the event.
  • out_fields: defines the fields to be added to the event from the lookup table upon a successful match.
    • event_field: specifies the name and location of the field to be enriched - the Explanation attribute of the event.
      • lookup_field: specifies the column in the lookup table from which the Explanation value will be taken.
    • event_field: specifies the name and location of the field to be enriched - the Recommended_Action attribute of the event.
      • lookup_field: the column in the lookup table used to obtain the Recommended_Action value.

Note: When referring to column names, ensure you use the exact field name as specified between the commas in the CSV, and do not create a CSV with leading spaces such as FTD Code, Explanation, Recommended Action.

Input Logs

<80>Oct 17 07:03:33 firewall01 %FTD-1-104004: (Primary) Switching to OK.

This log’s FTD code %FTD-1-104004 matches the first row in the lookup table. So it will be enriched using the other columns in the table for that row.

Output Logs

{
  "id": "789e96ed-2ef9-4bfc-9f7b-69b14e03219c",
  "timestamp": 1729148615311,
  "severity_text": "",
  "body": "<80>Oct 17 07:03:33 firewall01 %FTD-1-104004: (Primary) Switching to OK.",
  "resource": {
    ...
  },
  "attributes": {
    "Explanation": "A previously failed unit reports that it is operating again",
    "Recommended_Action": "No action required",
    ...
  }
}

Two attributes have been added: Explanation and Recommended_Action and their values have been populated using the second and third column in the lookup table, within in the %FTD-1-104004 row.

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: lookup

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

location_path

The location_path parameter specifies where the lookup table is located. This field is mandatory and the format should be:

  • "file://<path>"
  • "ed://<file name in ED stored lookup>"
  • "(http|https)://<URL to CSV>"
nodes:
  - name: <node name>
    type: lookup
    location_path: "file://path/to/csv/example.csv"

key_fields

The key_fields are pairs that map event fields to lookup fields to find matches.

nodes:
  - name: <node name>
    type: lookup
    key_fields:
    - event_field: <event field path>
      lookup_field: <lookup field name>
    out_fields:
    - event_field: <path in log>
      lookup_field: <column in table>
      default_value: <value if no match>
      append_mode: <true or false>

For key_fields, the event_field specifies the key value in the log and binds it to the lookup_field. For each log, the node will extract the event_field value using the event field’s pattern and compare it to each value in lookup_field for a match.

See how to use lookup tables for information on how the key_fields bind a log field and a table field.

out_fields

The out_fields define mappings from lookup table to event attributes for enrichment upon successful matches. Supports default_value, and append_mode if multiple rows are matched.

nodes:
  - name: <node name>
    type: lookup
    key_fields:
    - event_field: <event field path>
      lookup_field: <lookup field name>    
    out_fields:
    - event_field: <path in log>
      lookup_field: <column in table>
      default_value: <value if no match>
      append_mode: <true or false>

For out_fields, there are two binding pairs: For each, a new attribute will be created based on the event_field, and its value will be extracted from the lookup_field - for all rows matched by the key_field parameter.

See how to use lookup tables for information on how the out_fields bind a log field and a table field.

Optional Parameters

reload_period

The reload_period parameter is used to specify how often the lookup table is reloaded. It is defined as a duration and defaults to 5 minutes if not specified.

nodes:
  - name: <node name>
    type: lookup
    reload_period: <duration>

match_mode

The match_mode parameter configures the match type, either exact or regex. The default is exact.

nodes:
  - name: <node name>
    type: lookup
    match_mode: <"exact" or "regex">

ignore_case

The ignore_case setting, when true, makes the match case-insensitive. This is applicable only in exact match mode. Defaults to false.

nodes:
  - name: <node name>
    type: lookup
    ignore_case: <true or false>

regex_option

The regex_option specifies regex behavior in regex match mode. It can be first (default) for the first match or all for all matches.

nodes:
  - name: <node name>
    type: lookup
    regex_option: <"first" or "all">