Edge Delta Enrichment Node

Overview

An enrichment node modifies data as it is being streamed through the node. The value used to enrich the logs is calculated only at the first log. This calculated value is then applied to all subsequent logs per source.

Enrichment does not re-calculate the enrichment value for each log - it uses the value determined by the first log.

Unlike Enrichment nodes which calculate the enrichment using only the first log per source, transform node transformations are calculated for each log processed. Transform nodes therefore have a slower throughput than enrichments.

Unlike transform nodes which calculate transformations for each log processed, enrichment nodes calculate the enrichment using only the first log per source. Transform nodes therefore have a slower throughput than enrichments.

You can configure enrichment to, for example add attributes or metadata to a new or existing field to make troubleshooting easier; or transform a field to match a particular data format.

Example Configuration

nodes:
  - name: my_enrichments
    type: enrich
    field_mappings:
    - field_name: pod_id
      value: regex_capture(item._ed.file_path, "/var/logs/(?P<id>(.+))/.*")["id"]
    - field_name: controller_kind
      value: from_k8s(item.pod_id, "k8s_controller_kind")
    - field_name: namespace
      value: from_k8s(item.pod_id, "k8s_namespace")
    - field_name: instance_id
      value: ec2_metadata("instance-id")
    - field_name: cluster
      value: first_non_empty([env("UNDEFINED_CLUSTER"), env("CLUSTER"), "default-cluster"])
    - field_name: instance_name
      value: gcp_metadata("instance.name")
    - field_name: app_host
      value: item._ed.host
    - field_name: tag
      value: |
        % gotemplate
          {{ .item._ed.tag }}        
      override_existing_value: true    
    - field_name: service_name
      value: json(item._raw).messages[0].service

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: enrich

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

field_mappings

The field_mappings parameter define the fields that will be enriched by the Enrichment node. All field_mappings have - field_name and a value child parameters. A field_mappings parameter is required for an enrichment node.

The field_name parameter defines the log field that will be enriched either by updating the value if the field already exists in the log, or by adding the field if it doesn’t already exist. It is specified as a string. The field_name parameter is required for an enrichment node.

The value parameter specifies the logic for creating a derived value for the field_name.

There are several predefined functions for the value parameter:

• first_none_empty(string ...params) string returns the first non empty string from the input parameters.
• regex_match(str, regex string) bool returns true if input string matches the regex pattern.
• regex_capture(str, regexWithCaptureGroups string) map[string]string grabs and returns one or more parts from the string using regex capture groups.
• env(envVarKey string) string returns the value from environment variables.
• from_k8s(podID, podAttributeName string) string returns the attribute value of a K8s pod from given pod id.
• json(str string) map[string]any un-marshalls and returns a map representation of the json object.
• ec2_metadata(key) string returns the value of given key from EC2 metadata service.
• gcp_metadata(key) string returns the value of given key from GCP metadata service.

nodes:
  - name: <node name>
    type: enrich
    field_mappings:
    - field_name: <name of field to enrich>
      value: <source of data to enrich the field>  
    - field_name: <name of field to enrich>
      value: <source of data to enrich the field>    

Optional Parameters

override_if_exists

The override_if_exists parameter configures whether to over-write the values derived from enrichment if they exist already. It is a child parameter of the field mappings and is specific to a particular field_name. It is specified as a Boolean and the default is false. It is optional.

nodes:
  - name: <node name>
    type: enrich
    field_mappings:
    - field_name: <name of field to enrich>
      value: <source of data to enrich the field>
      override_existing_value: true | false