Enrich Input Data with Edge Delta
4 minute read
Overview
You can add enrichment configurations to an input to extract custom attributes from the source. Afterwards, the enriched data will be sent to the destination configured in the workflow.
Enrichment Types
from_logs
from_logs is used to enrich data with fields extracted from specified logs.
The enriched data will be sent to the destination defined in the workflow.
Under from_logs, you must specify a field_name, as well as a pattern or json_path.
- Usefield_name to define the source field in the raw data.
- Usepatternto define how to extract the raw data.
- Any data after the colon
:
will be extracted whenpodname
is detected in the following example. - You must enter a capture pattern.
- You can only enter one capture group.
- Any data after the colon
- Usejson_path to enrich data with fields extracted from JSON logs.
from_logs is supported for the following input types:
from_logs:
field_mappings:
- field_name: podname
pattern: "podname: (\\w+)"
- field_name: component
json_path: fields.component
from_path
from_path is used to enrich data with fields extracted from a specified path.
The enriched data will be sent to the destination defined in the workflow.
Under from_path, you must specify a field_name and a pattern.
- Usefield_name to define the source field in the raw data.
- Usepatternto define how to extract the raw data.
- You must enter a capture pattern.
- You can only enter one capture group.
from_path is supported for the following input types:
enrichments:
from_path:
field_mappings:
- field_name: application
pattern: /var/logs/anyDir/(?:(.+)/)?users/.*
from_k8s
from_k8s is used to enrich streaming data with K8s attributes.
You can enter a pod, namespace, or node attributes. You can also enrich K8s controller information such as controllerName, controllerLogicalName, or controllerKind.
- The agent will fetch the controller name from the pod’s first owner reference name, e.g. flog-normal-57fd4d8b8d.
- The agent will examine the root controller map, find the corresponding controller logical name for the current pod_id, and make the root controller’s name the same as the controller logical name. This requires ED_ENABLE_PARENT_K8S_CONTROLLER_DISCOVERY to be enabled. If this env var is not enabled, there will be no root controller map and the agent will use the controller name as the controller logical name while trimming the last part (after the last dash, e.g. flog-normal).
- Controller kind uses the same logic as controller logical name, but it fetches the kind of controller instead of the name. The logic for disabled ED_ENABLE_PARENT_K8S_CONTROLLER_DISCOVERY is the same. It will use the pod’s first owner reference kind as the controller kind.
from_k8s is supported for the following input types:
from_k8s:
pod_identifier_pattern: /var/logs/anyDir/MyApp/users/(?:(.+)/)/.*
field_mappings:
- field_name: instance_id
pod_attribute: pod
transformers:
- source: "-"
target: "_"
type: "replace"
- source: "test*"
target: ""
type: "regex"
- field_name: namespace
pod_attribute: namespace
- field_name: service
pod_attribute: labels.service
from_k8s:
pod_identifier_pattern: pods\/([^\/+]+)\/
field_mappings:
- field_name: controllerName
pod_attribute: controllerName
- field_name: controllerLogicalName
pod_attribute: controllerLogicalName
- field_name: controllerKind
pod_attribute: controllerKind
- field_name: pod_id
pod_attribute: pod
- field_name: namespace
pod_attribute: namespace
dynamic
dynamic is used to enrich data from another source, field, or component to dynamically enrich the data. In other words, data will be pulled externally from the raw log message.
You must enter a field_name and a value.
For dynamic enrichment, consider the following statements:
- Dynamic fields can be derived from other enrichment fields, as well as static fields.
- Dependent fields should be ordered in a particular order.
- For example, if field2 is dependent on field1, then you must define field1 before you define field2.
dynamic is supported for the following input types:
The following example shows how to obtain data from a static field:
enrichments:
dynamic:
field_mappings:
- field_name: "app_name"
value: "transaction_api"
Obtain Data from Another Dynamic Enrichment Field
You can obtain data from another dynamic enrichment field:
enrichments:
dynamic:
field_mappings:
- field_name: "service_base_url"
value: "https://api.mycompany.com"
- field_name: "tansaction_api_url"
value: " {{ .service_base_url }}/transaction "
Obtain Data From Another Enrichment Source (Kubernetes)
Annotation keys are specific to the container in the pod. In value, enter the annotation key.
enrichments:
dynamic:
field_mappings:
- field_name: "source"
value: `.annotations.kubernetes.io/{{.container_name}}.logs`,
Obtain Data From an AWS EC2 Instance
To obtain data from an AWS EC2 instance, in the value parameter, you must enter aws-instance.
enrichments:
dynamic:
field_mappings:
- field_name: "instance_id"
value: '{{".aws-instance.instance-id"}}'
- field_name: "instance_type"
value: '{{".aws-instance.instance-type"}}'
For additional information, please review the Retrieve instance metadata document from Amazon.
Troubleshoot Mapping Failures
To troubleshoot potential mapping failures, you can configure the failure_behavior parameter.
failure_behavior option | Description |
---|---|
stop_enrichment | This option will terminate the enrichment. |
drop_source | This option will terminate the enrichment and also stop tailing the source. |
skip_failing_fields | This option will skip over the detected failed mapping in the enrichment. This option is the default value. |
Additionally, you can use the fallback_value parameter to troubleshoot. Specifically, if mapping fails based on the value or json_path parameter, then the configured value for fallback_value will be used until the agent confirms that the mapping has failed.
enrichments:
failure_behavior: stop_enrichment
dynamic:
field_mappings:
- field_name: "service"
value: '{{".labels.service"}}'
- field_name: "source"
value: '.annotations.kubernetes.io/{{.container_name}}.logs'
json_path: "[0].source"
fallback_value: '{{".short_container_image"}}'