Use Lookup Tables in Edge Delta

Background

Suppose you want to add an attribute to a log if the body contains a certain string. This kind of simple if-then enrichment is easy to configure in an Edge Delta pipeline using, for example, a Route node followed by a Log Transform node. The Route node finds log instances that require transformation, and routes those logs to the Log Transform to perform the transformations.

But what if you have a large list of string conditions each requiring a different enrichment? The Route - Enrich approach quickly becomes unwieldy even with only 10 conditions.

In these instances you should use a Lookup node. In short, it matches logs that contain a string in any row of a given column of a lookup table, and enriches each matching log with values from other columns in the matching row. This will be discussed further:

Overview

Lookup Table

Each Lookup node references one lookup table. For example:

The Key

For a log to match and be processed by the node, a value in the log must match the value in a specified column in the lookup table.

{“host”: “R1C1”, “user-identifier”: “55c51e9b”, “time_local”: “2024-06-20T11:59:26.265285Z”, “action”: “create”}

In this instance, the host value matches the value in row 1, column 1 R1C1. This matching value is the key for the enrichment operations. It performs two functions:

1. Provide the value to match (if the log contains this string…)
2. It provides the row reference for the enrichment operations to follow (then enrich using values from other columns in the matching row)

You specify the key with a CEL macro for body fields or a normal field path, such as item["resource"]["host.name"] if the value is already parsed.

It captures a value from each log and it evaluates the value against values in a specified column in the lookup table.

The key fields therefore require two binding values: a reference to the key in the log, and the table column in which to look for matches.

When a match is found, the entire row becomes available to the node for enrichment:

Enrichments

Next the Lookup node enriches the log using values found in the matched row. For example R1C2 and R1C3 could become attributes.

As with the key fields, the enrichment fields also require two binding values: a reference to the field in the log that needs to be enriched, and the table column for the value to use to enrich the log.

Note: You can match on multiple rows, in which case the enrichment value is a comma separated list.

Example Configuration

This example illustrates the node configuration for the log example:

- name: FTD Code
  type: lookup
  location_path: ed://r1c1.csv
  reload_period: 5m0s
  match_mode: exact
  regex_option: first
  key_fields:
  - event_field: regex_capture(item["body"], "(?P<Value>R\d+\C1)")["Value"]
    lookup_field: Column 1
  out_fields:
  - event_field: item["attributes"]["new_attribute_1"]
    lookup_field: Column 2
  - event_field: item["attributes"]["new_attribute_2"]
    lookup_field: Column 3

Note the binding pairs:

For key_fields, the event_field specifies the key value in the log and binds it to the lookup_field. For each log, the node will extract the value using the event field’s pattern and compare it to each value in Column 1 for a match.

For out_fields, there are two binding pairs: For each, a new attribute will be created based on the event_field, and its value will be extracted from the lookup_field - for all rows matched by the key_field parameter.

Other Options

The Lookup node also takes other options:

Table Location

The table is formatted as a CSV. You make the table available to the node (on the edge) in one of three ways:

1. Upload it in Edge Delta on the Knowledge tab of the Pipelines page.
2. Save it on the host
3. Make it available on HTTP or HTTPS

You specify the location of the CSV file you uploaded using a different format depending on where you uploaded it.

Tables uploaded to Edge Delta: For Edge Delta tables specify ed:// followed by the filename you used when you uploaded it.
ed://lookuptablefilename.csv

Tables uploaded to a Windows OS: For Windows local files specify file:// followed by the location and filename. file://c:\location\lookuptablefilename.csv

Tables uploaded to a Linux OS or macOS: For Unix type local files specify file:// followed by the location and filename. file:///app/lookuptablefilename.csv

Tables uploaded to a website: For files available online specify the URL. It can be HTTP or HTTPS. https://docs.edgedelta.com/lookuptablefilename.csv

Reload Period

You can configure how often the node should check the lookup table location for update to it. For tables that do not change often this duration can be longer. For testing or tables that change regularly this should be a shorter duration. The default value is 5 minutes.

Match Mode

You can specify exact or regex to indicate the type of key matching. Exact will search the key column for an exact match against the value extracted from the log. The regex option will search for a matching pattern, in other words column 1 would consist of regex patterns and the node would attempt to find a matching pattern for the extracted log value.

Ignore Case

If you set the Match Mode to exact you can specify whether or not the match should be case-sensitive.

Regex Option

If you set the Match Mode to regex you can specify how many regex matches to match against. You specify first to stop searching after the first match, or all to find all matching rows.

If more than one matching row is found, the enrichments will be multiple comma separated values.