Edge Delta Lookup Processor

The Edge Delta lookup processor enriches logs, metrics, and traces dynamically using a lookup table, supporting various match modes and field mappings.

16 minute read

Overview

You can enrich logs, metrics, and traces dynamically using a lookup table. This is useful for enriching data based on multiple criteria. For example, you can enrich data items that contain codes with attributes that provide the code definitions based on a table of all possible codes and their definitions. You can host the lookup table in Edge Delta or in your own location.

Use the configuration wizard below to generate a starter YAML configuration.

The following diagram illustrates how the lookup processor handles incoming data items:

flowchart TD A[Data Item Arrives] --> B{Extract key_fields} B --> C{Fields exist?} C -->|No| D[Pass through unchanged] C -->|Yes| E[Search Lookup Table] E --> F{Match found?} F -->|No| G{default_value set?} G -->|Yes| H[Apply default values] G -->|No| D F -->|Yes| I[Extract matched row values] I --> J{Multiple matches?} J -->|No| K[Apply to out_fields] J -->|Yes| L{append_mode true?} L -->|Yes| M[Concatenate values] L -->|No| K H --> N[Enriched Data Item] K --> N M --> N D --> O[Unchanged Data Item]

Diagram preview unavailable

View diagram source

flowchart TD
    A[Data Item Arrives] --> B{Extract key_fields}
    B --> C{Fields exist?}
    C -->|No| D[Pass through unchanged]
    C -->|Yes| E[Search Lookup Table]
    E --> F{Match found?}
    F -->|No| G{default_value set?}
    G -->|Yes| H[Apply default values]
    G -->|No| D
    F -->|Yes| I[Extract matched row values]
    I --> J{Multiple matches?}
    J -->|No| K[Apply to out_fields]
    J -->|Yes| L{append_mode true?}
    L -->|Yes| M[Concatenate values]
    L -->|No| K
    H --> N[Enriched Data Item]
    K --> N
    M --> N
    D --> O[Unchanged Data Item]

See Understanding Lookup Tables for concepts and background on how lookup tables work.

Prerequisites

JSON Log Parsing

When processing JSON-formatted logs, parse the JSON into attributes before lookup processors can access the fields. This ensures the lookup processor can find and match the fields specified in your configuration.

Add a Parse JSON processor before your lookup processor in the sequence:

- type: ottl_transform
  name: Parse JSON
  statements: |-
    set(cache["parsed-json"], ParseJSON(body))
    merge_maps(attributes, cache["parsed-json"], "upsert") where IsMap(attributes) and IsMap(cache["parsed-json"])

Field Availability

The lookup processor operates on data items (logs, metrics, or traces) that contain the fields specified in key_fields. Data items without these fields pass through unmodified to the next processor in the sequence. See Processors for more information about how processors handle data items in a multiprocessor stack.

Configuration

The lookup processor supports multiple match modes for different use cases. See the example pages for complete configurations with lookup tables, input data, and expected output:

Example	Match Mode	Use Case
FTD Codes	regex	Enrich firewall logs with code explanations
Exact Match	exact	Match error codes precisely
Prefix Match	prefix	Categorize by error code families
Suffix Match	suffix	Classify by file extensions
Contain Match	contain	Route by service keywords
Regex Match	regex	Match log level patterns
Multiple Matches	contain + all	Auto-tag with multiple keywords

Options

Select a telemetry type

You can specify, log, metric, trace or all. It is specified using the interface, which generates a YAML list item for you under the data_types parameter. This defines the data item types against which the processor must operate. If data_types is not specified, the default value is all. It is optional.

It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    data_types:
    - log

condition

The condition parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional.

Important: All conditions must be written on a single line in YAML. Multi-line conditions are not supported.

Comparison Operators

Operator	Name	Description	Example
`==`	Equal to	Returns `true` if both values are exactly the same	`attributes["status"] == "OK"`
`!=`	Not equal to	Returns `true` if the values are not the same	`attributes["level"] != "debug"`
`>`	Greater than	Returns `true` if the left value is greater than the right	`attributes["duration_ms"] > 1000`
`>=`	Greater than or equal	Returns `true` if the left value is greater than or equal to the right	`attributes["score"] >= 90`
`<`	Less than	Returns `true` if the left value is less than the right	`attributes["load"] < 0.75`
`<=`	Less than or equal	Returns `true` if the left value is less than or equal to the right	`attributes["retries"] <= 3`
`matches`	Regex match	Returns `true` if the string matches a regular expression (generates `IsMatch` function)	`IsMatch(attributes["name"], ".*\\.log$")`

Logical Operators

Important: Use lowercase and, or, not - uppercase operators will cause errors!

Operator	Description	Example
`and`	Both conditions must be true	`attributes["level"] == "ERROR" and attributes["status"] >= 500`
`or`	At least one condition must be true	`attributes["log_type"] == "TRAFFIC" or attributes["log_type"] == "THREAT"`
`not`	Negates the condition	`not regex_match(attributes["path"], "^/health")`

Functions

Function	Description	Example
`regex_match`	Returns `true` if string matches the pattern	`regex_match(attributes["message"], "ERROR\\|FATAL")`
`IsMatch`	Alternative regex function (UI generates this from “matches” operator)	`IsMatch(attributes["name"], ".*\\.log$")`

Field Existence Checks

Check	Description	Example
`!= nil`	Field exists (not null)	`attributes["user_id"] != nil`
`== nil`	Field doesn’t exist	`attributes["optional_field"] == nil`
`!= ""`	Field is not empty string	`attributes["message"] != ""`

Common Examples

- name: _multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    # Simple equality check
    condition: attributes["request"]["path"] == "/json/view"
    
  - type: <processor type>
    # Multiple values with OR
    condition: attributes["log_type"] == "TRAFFIC" or attributes["log_type"] == "THREAT"
    
  - type: <processor type>
    # Excluding multiple values (NOT equal to multiple values)
    condition: attributes["log_type"] != "TRAFFIC" and attributes["log_type"] != "THREAT"
    
  - type: <processor type>
    # Complex condition with AND/OR/NOT
    condition: (attributes["level"] == "ERROR" or attributes["level"] == "FATAL") and attributes["env"] != "test"
    
  - type: <processor type>
    # Field existence and value check
    condition: attributes["user_id"] != nil and attributes["user_id"] != ""
    
  - type: <processor type>
    # Regex matching using regex_match
    condition: regex_match(attributes["path"], "^/api/") and not regex_match(attributes["path"], "^/api/health")
    
  - type: <processor type>
    # Regex matching using IsMatch
    condition: IsMatch(attributes["message"], "ERROR|WARNING") and attributes["env"] == "production"

Common Mistakes to Avoid

# WRONG - Cannot use OR/AND with values directly
condition: attributes["log_type"] != "TRAFFIC" OR "THREAT"

# CORRECT - Must repeat the full comparison
condition: attributes["log_type"] != "TRAFFIC" and attributes["log_type"] != "THREAT"

# WRONG - Uppercase operators
condition: attributes["status"] == "error" AND attributes["level"] == "critical"

# CORRECT - Lowercase operators
condition: attributes["status"] == "error" and attributes["level"] == "critical"

# WRONG - Multi-line conditions
condition: |
  attributes["level"] == "ERROR" and 
  attributes["status"] >= 500  

# CORRECT - Single line (even if long)
condition: attributes["level"] == "ERROR" and attributes["status"] >= 500

Location

You define the location of the lookup table. You can specify a lookup table hosted in Edge Delta, a file on the cluster, or Other for a URL. If you select an Edge Delta lookup table you can select it from a list. If you select File you enter the filename and path. Or you specify the URL for other.

The tool populates the location_path parameter in the YAML. This field is mandatory and the format is as follows depending on the location type:

"file://<path>"
"ed://<file name in ED stored lookup>"
"(http|https)://<URL to CSV>"

The following diagram shows how the processor loads and caches lookup tables:

flowchart LR subgraph Source[Lookup Table Source] S1["ed:// EdgeDelta"] S2["file:// Local File"] S3["http:// Remote URL"] end subgraph Processor[Lookup Processor] CACHE[In-Memory Cache] TIMER[Reload Timer] end subgraph Processing[Data Processing] DATA[Incoming Data Items] ENRICH[Enrichment Logic] end Source -->|Initial Load| CACHE TIMER -->|Periodic Refresh| Source Source -->|Updated Data| CACHE DATA --> ENRICH CACHE -->|Fast Lookup| ENRICH ENRICH --> OUT[Enriched Output]

Diagram preview unavailable

View diagram source

flowchart LR
    subgraph Source[Lookup Table Source]
        S1["ed:// EdgeDelta"]
        S2["file:// Local File"]
        S3["http:// Remote URL"]
    end

    subgraph Processor[Lookup Processor]
        CACHE[In-Memory Cache]
        TIMER[Reload Timer]
    end

    subgraph Processing[Data Processing]
        DATA[Incoming Data Items]
        ENRICH[Enrichment Logic]
    end

    Source -->|Initial Load| CACHE
    TIMER -->|Periodic Refresh| Source
    Source -->|Updated Data| CACHE
    DATA --> ENRICH
    CACHE -->|Fast Lookup| ENRICH
    ENRICH --> OUT[Enriched Output]

Reload Period

This option is used to specify how often the lookup table is reloaded. It is defined as a duration and defaults to 5 minutes if not specified. The tool populates the reload_period parameter in YAML.

Match mode

You can choose how to match the lookup key field. The tool populates the match_mode parameter with one of the following options:

exact (default) - Matches when the event field value exactly equals the lookup field value
regex - Matches when the lookup field contains a regex pattern that matches the event field
contain - Matches when the event field value contains the lookup field value as a substring
prefix - Matches when the event field value starts with the lookup field value
suffix - Matches when the event field value ends with the lookup field value

Note: The lookup processor enriches data items (logs, metrics, or traces) where the specified event_field exists and matches the lookup criteria. Data items without matching fields pass through to the next processor unmodified.

regex_option

The regex_option parameter controls how many matches are returned when using regex match mode.

Value	Description
`first`	(Default) Stop after finding the first matching row
`all`	Return all matching rows. Use with `append_mode: true` on out_fields to collect multiple values

match_option

The match_option parameter controls how many matches are returned when using contain, prefix, or suffix match modes.

Value	Description
`first`	(Default) Stop after finding the first matching row
`all`	Return all matching rows. Use with `append_mode: true` on out_fields to collect multiple values

Note: For regex match mode, use regex_option instead. The match_option parameter only applies to contain, prefix, and suffix modes.

ignore_case

When enabled, the lookup matching becomes case-insensitive. This option is available when using exact, contain, prefix, or suffix match modes (not available for regex mode).

Value	Description
`false`	(Default) Case-sensitive matching
`true`	Case-insensitive matching

Match Mode Examples

Consider a lookup table with a key field containing “ERROR-500”:

Match Mode	Event Field Value	Matches?
exact	“ERROR-500”	Yes
exact	“ERROR-500 occurred”	No
contain	“ERROR-500 occurred in production”	Yes
prefix	“ERROR-500 internal server error”	Yes
prefix	“Server ERROR-500”	No
suffix	“Critical ERROR-500”	Yes
suffix	“ERROR-500 detected”	No

key_fields

The key_fields are pairs that map event fields to lookup fields to find matches.

For key_fields, the event_field specifies the key value in the data item and binds it to the lookup_field. For each data item, the processor extracts the event_field value and compares it to each value in lookup_field for a match.

See Understanding Lookup Tables for information on how the key_fields bind a data item field and a table field.

Multiple key_fields (compound matching)

You can specify multiple key_fields to create compound match conditions. When multiple keys are specified, all keys must match for a lookup row to be selected (AND logic).

flowchart TD subgraph DataItem[Data Item] D1["region = us-west-2"] D2["env = prod"] end subgraph LookupTable[Lookup Table] R1["Row 1: region=us-east-1, env=prod"] R2["Row 2: region=us-west-2, env=staging"] R3["Row 3: region=us-west-2, env=prod"] R4["Row 4: region=eu-west-1, env=prod"] end subgraph KeyFields[key_fields - AND Logic] K1["Key 1: region"] K2["Key 2: env"] end DataItem --> KeyFields KeyFields --> |Both must match| LookupTable R1 --> |region mismatch| X1[No Match] R2 --> |env mismatch| X2[No Match] R3 --> |Both match| MATCH[Match Found] R4 --> |region mismatch| X4[No Match]

Diagram preview unavailable

View diagram source

flowchart TD
    subgraph DataItem[Data Item]
        D1["region = us-west-2"]
        D2["env = prod"]
    end

    subgraph LookupTable[Lookup Table]
        R1["Row 1: region=us-east-1, env=prod"]
        R2["Row 2: region=us-west-2, env=staging"]
        R3["Row 3: region=us-west-2, env=prod"]
        R4["Row 4: region=eu-west-1, env=prod"]
    end

    subgraph KeyFields[key_fields - AND Logic]
        K1["Key 1: region"]
        K2["Key 2: env"]
    end

    DataItem --> KeyFields
    KeyFields --> |Both must match| LookupTable

    R1 --> |region mismatch| X1[No Match]
    R2 --> |env mismatch| X2[No Match]
    R3 --> |Both match| MATCH[Match Found]
    R4 --> |region mismatch| X4[No Match]

The following example matches rows where both region AND service fields match:

key_fields:
- event_field: attributes["region"]
  lookup_field: Region
- event_field: attributes["service"]
  lookup_field: Service

In this configuration, a lookup row is only matched if both the region AND service fields match their corresponding lookup table columns.

out_fields

The out_fields define mappings from lookup table to event attributes for enrichment upon successful matches.

For out_fields, there are two binding pairs: For each, a new attribute will be created based on the event_field, and its value will be extracted from the lookup_field - for all rows matched by the key_field parameter.

See Understanding Lookup Tables for information on how the out_fields bind a data item field and a table field.

default_value

The default_value parameter specifies a fallback value to use when no matching row is found in the lookup table. If omitted and no match occurs, the field is not added to the data item.

out_fields:
- event_field: attributes["timezone"]
  lookup_field: TimeZone
  default_value: UTC

append_mode

The append_mode parameter determines how multiple matched values are handled. When set to true, the processor concatenates all matching values with commas. Use this with regex_option: all or match_option: all to collect values from multiple matching rows.

Value	Description
`false`	(Default) Use the value from the first matching row
`true`	Concatenate values from all matching rows, separated by commas

out_fields:
- event_field: attributes["matched_hosts"]
  lookup_field: Host
  append_mode: true

Final

Determines whether successfully processed data items should continue through the remaining processors in the same processor stack. If final is set to true, data items output by this processor are not passed to subsequent processors within the node—they are instead emitted to downstream nodes in the pipeline (e.g., a destination). Failed items are always passed to the next processor, regardless of this setting.

The UI provides a slider to configure this setting. The default is false. It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
    - type: <processor type>
    final: true

Troubleshooting

Understanding Processor Behavior

The lookup processor follows standard multiprocessor logic: it processes data items that match its criteria and passes all data items (both processed and unprocessed) to the next processor in the sequence. If you set final: true, only unmatched data items continue to the next processor. See Processors for details on processor chaining behavior.

Common Scenarios

No enrichment occurring

Possible causes:

JSON not parsed: For JSON logs, ensure you have a Parse JSON processor before the lookup processor
Field not in attributes: Verify the field specified in event_field exists in your parsed attributes
No matching values: Check that your lookup table contains the values present in your logs
Case sensitivity: For exact matches, ensure case matches exactly (use ignore_case option if needed)

Lookup table issues

Possible causes:

Invalid CSV format: Ensure your lookup table is properly formatted CSV with headers
Location path: Verify the location_path format (e.g., ed://filename.csv for Edge Delta hosted tables)
Reload period: The table refreshes based on reload_period (default 5 minutes)

Testing Tips

Use Live Capture to verify fields are properly extracted and available in attributes
Test with simple exact matches first before trying complex match modes
Allow 1-2 minutes for log indexing when validating enrichment in search results
Send test logs with known lookup values to verify configuration
Check agent logs to confirm the lookup table is loaded successfully

Configuration Wizard

Use this interactive wizard to generate a starter configuration:

1 Source

2 Matching

3 Defaults

4 Condition

5 Config

Where is your lookup table hosted?

Choose the location of your lookup table.

Your answers stay in your browser

Edge Delta Lookup Processor

Overview

Prerequisites

JSON Log Parsing

Field Availability

Configuration

Options

Select a telemetry type

condition

Comparison Operators

Logical Operators

Functions

Field Existence Checks

Common Examples

Common Mistakes to Avoid

Location

Reload Period

Match mode

regex_option

match_option

ignore_case

Match Mode Examples

key_fields

Multiple key_fields (compound matching)

out_fields

default_value

append_mode

Final

Troubleshooting

Understanding Processor Behavior

Common Scenarios

No enrichment occurring

Lookup table issues

Testing Tips

Configuration Wizard

Where is your lookup table hosted?

Does your endpoint use HTTPS?

How should values be matched in the lookup table?

Should matching be case-sensitive?

How should multiple regex matches be handled?

How should multiple matches be handled?

Should matching be case-sensitive?

What should happen when no match is found?

How often should the lookup table be refreshed?

When should this processor run?

What comparison should be performed?

How should the body be matched?

How should the conditions be combined?

Your Lookup Processor Configuration

See Also

Edge Delta AI Assistant

Conversations

Hi! I'm your Edge Delta AI Assistant

Current Context