Lookup Examples Pack
6 minute read
Edge Delta Pipeline Pack on Lookup Examples
Overview
The Edge Delta Lookup Examples pack demonstrates various approaches to enhancing and transforming logs using local and remote lookups, schema mapping, and attribute addition. This pack showcases techniques for enriching logs through structured mappings and transformations using the lookup processor node.
Pack Description
1. Data Ingestion
The data flow initiates at the Source node, acting as the entry point for all logs to commence their processing sequence. All traffic is then split along 5 parallel paths. Each path illustrates use of a Lookup.
2.1. Add Host
On one path, the logs proceed to the Add host node, an OTTL Transform node.
- name: Add host
type: ottl_transform
statements: set(attributes["host"], "yjgdk-420.example.com")
This node sets the host attribute to a predefined value, yjgdk-420.example.com.
2.2. Apply Lookups by Host
The logs on this path flow from Add host to the Lookup by host node, a Lookup node. This node uses a lookup table stored locally to add attributes such as region, index, and timezone, with host as the key.
The table is structured as follows:
| Host | Region | Index | TimeZone |
|---|---|---|---|
| cwhvb-151.example.com | ap-northeast-1 | analytics | AWST |
| brxxw-255.example.com | ap-northeast-1 | logs | UTC |
- name: Lookup by host
type: lookup
location_path: ed://lookup_examples.csv
reload_period: 5m0s
match_mode: exact
regex_option: first
key_fields:
- event_field: item["attributes"]["host"]
lookup_field: Host
out_fields:
- event_field: item["attributes"]["region"]
lookup_field: Region
- event_field: item["attributes"]["index"]
lookup_field: Index
- event_field: item["attributes"]["timezone"]
lookup_field: TimeZone
default_value: UTC
The match_mode parameter specifies how the keys from the event will be matched against the keys in the lookup table. In this case, match_mode: exact indicates that the match must be an exact string match.
The regex_option parameter determines the behavior of the regex matching when the match mode is set to regex. In this configuration, however, the match_mode is set to exact, which means the regex_option does not apply to match operations.
This node performs exact matching on the host attribute against the Host column in the lookup table. Once a match is found, the node enriches the log by attaching corresponding region, index, and timezone attributes from the matching row in the lookup table.
From here traffic on this path flows to the Success output.
3.1 Add Region
On another path, logs go to the Add region node, another OTTL Transform node.
- name: Add region
type: ottl_transform
statements: set(attributes["region"], "us-west-2")
This operation assigns the region attribute a static value of "us-west-2". This transformation standardizes the region attribute across logs.
3.2. Apply Region-based Lookups
The next stage on this path involves the Lookup hosts in a region node, another Lookup node. Using the regex match mode, this node appends a list of hosts corresponding to the identified region.
The table is structured as follows:
| Host | Region | Index | TimeZone |
|---|---|---|---|
| cwhvb-151.example.com | ap-northeast-1 | analytics | AWST |
| brxxw-255.example.com | ap-northeast-1 | logs | UTC |
- name: Lookup hosts in a region
type: lookup
location_path: ed://lookup_examples.csv
reload_period: 5m0s
match_mode: regex
regex_option: all
key_fields:
- event_field: item["attributes"]["region"]
lookup_field: Region
out_fields:
- event_field: item["attributes"]["hosts"]
lookup_field: Host
append_mode: true
The match_mode parameter defines the method used to find matches between log events and the lookup table. In this configuration, match_mode: regex indicates that regular expressions are utilized for matching.
The regex_option specifies how extensively regex is applied during matching. With regex_option: all, the configuration expands to include all possible matches.
The append_mode parameter determines how multiple matched values are handled. When append_mode: true, the system concatenates values and separates them with commas.
In this configuration, the node uses regex to match the region attribute against values in the Region lookup field. It then appends all matching Host values to the hosts attribute of the event.
4.1. Add Region and Index
On another parallel path, logs flow to the Add region and index node, another OTTL Transform.
- name: Add region and index
type: ottl_transform
statements: |-
set(attributes["region"], "AP-NorthEast-1")
set(attributes["index"], "products")
This node assigns static values for both the region and index attributes.
4.2. Apply Index-based Lookups
The logs proceed to the Lookup by region and index node, a Lookup node,
The table is structured as follows:
| Host | Region | Index | TimeZone |
|---|---|---|---|
| cwhvb-151.example.com | ap-northeast-1 | analytics | AWST |
| brxxw-255.example.com | ap-northeast-1 | logs | UTC |
- name: Lookup by region and index
type: lookup
location_path: ed://lookup_examples.csv
reload_period: 5m0s
match_mode: exact
regex_option: first
key_fields:
- event_field: item["attributes"]["region"]
lookup_field: Region
- event_field: item["attributes"]["index"]
lookup_field: Index
out_fields:
- event_field: item["attributes"]["timezone"]
lookup_field: TimeZone
default_value: UTC
The match_mode: exact parameter specifies that the lookup should perform a direct string comparison between the event data and the lookup table values.
The regex_option is relevant only when match_mode is set to regex.
This node performs exact match-based lookups to enhance the logs with the timezone attribute, using both region and index as keys.
5. Lookup Log Level
On this path, traffic passes through the Lookup Log Level Lookup node.
The table is as follows:
| MessagePattern | Level |
|---|---|
| .\sERROR\s. | error |
| .\sINFO\s. | info |
| .\sDEBUG\s. | debug |
- name: Lookup Log Level
type: lookup
location_path: ed://lookup_regex_examples.csv
reload_period: 5m0s
match_mode: regex
regex_option: first
key_fields:
- event_field: item["body"]
lookup_field: MessagePattern
out_fields:
- event_field: item["attributes"]["log_level"]
lookup_field: Level
The match_mode: regex parameter specifies that regular expressions will be used for matching the event data against the lookup table values.
The regex_option: first parameter controls how many matches are found when using regex. Setting it to first means the process stops after finding the first occurrence.
It enriches logs with a log_level attribute derived from the first matching pattern found in the body of the log message.
6.1. Set Old Schema
On this path, traffic flows to Set Old Schema, an OTTL Transform node.
- name: Set Old Schema
type: ottl_transform
statements: |-
set(attributes["schema"], "ApplicationLogs")
set(attributes["app_id"], "payment-service-wxck")
set(attributes["timestamp"], "2021-08-02T10:00:00.000Z")
set(attributes["level"], "INFO")
set(attributes["msg"], "Payment service started")
This operation instantiates attributes corresponding to an older schema, such as schema, app_id, and timestamp. This task prepares logs for schema mapping.
6.2. Schema Mapping Lookup
Logs on this path proceed to the Lookup Schema Mapping Lookup node, executing a schema mapping lookup specifically tied to the older schema format.
The table is as follows:
| Schema | OldKey | NewKey |
|---|---|---|
| ApplicationLogs | app_id | application_id |
| ApplicationLogs | level | severity |
| ApplicationLogs | msg | message |
| SystemMetrics | cpu | cpu_usage |
| SystemMetrics | mem | memory_usage |
| SystemMetrics | disk | disk_usage |
| SystemMetrics | uptime | system_uptime |
| UserActivity | user_id | user_identifier |
| UserActivity | action | user_action |
| UserActivity | ts | action_time |
| UserActivity | location | geo_location |
- name: Lookup Schema Mapping
type: lookup
location_path: ed://lookup_schema_examples.csv
reload_period: 5m0s
match_mode: regex
regex_option: all
key_fields:
- event_field: item["attributes"]["schema"]
lookup_field: Schema
out_fields:
- event_field: item["attributes"]["old_keys"]
lookup_field: OldKey
- event_field: item["attributes"]["new_keys"]
lookup_field: NewKey
The match_mode: regex parameter indicates that this lookup will use regular expressions to match fields from the log event against entries in the lookup table.
The regex_option: all controls the match behavior to consider all possible matches that satisfy the regex conditions.
On a successful match, it extracts OldKey and NewKey from the lookup table and enriches the log with these under old_keys and new_keys.
6.3. Map Schema
The schema mapping then undergoes transformation at the Map Schema node, another OTTL Transform node. This remaps keys from old_keys to new_keys, and subsequently deletes the temporary mapping attributes.
- name: Map Schema
type: ottl_transform
statements: |-
edx_map_keys(attributes, attributes["old_keys"], attributes["new_keys"])
delete_key(attributes, "old_keys")
delete_key(attributes, "new_keys")
For each entry in the old_keys, this node will replace that key with the corresponding value in new_keys. This ensures that any data structure changes, such as field name updates, are consistently applied across logs. The delete_key functions remove the old_keys and new_keys from the log’s attributes once the mapping operation is complete.
7. Successful Completion
All complete paths are consolidated at the Success pack output node.
Sample Input
2025-01-10T15:52:14.172Z INFO middleware/authz.go:383 request flog_log_generator spec:{uri:/v1/orgs/b9df8fc0-084b-11ee-be56-0242ac120002/confs/- method:PUT password:U7FaXR} latency:628ms