Choose Data Item Metadata

Overview

Selecting metadata at the ingestion point is essential to ensure only the most relevant context is attached to your data before it flows through the pipeline. This reduces the processing and transmission of unnecessary metadata, saving both bandwidth and storage costs. It also helps maintain cleaner, more focused datasets that are easier to query and analyze. By controlling metadata early, you gain greater efficiency, clarity, and compliance across your observability strategy.

Background

The Edge Delta agent populates each data item it ingests with metadata that it detects. Which metadata is added depends on the environment type and the source node type. Some metadata is required for the Edge Delta agent and back end to work effectively. However, you can select which of the other data you want the agent to add.

Example

A Kubernetes Log source node can populate the following resources fields and attribute field:

resource:

• host.name
• host.ip
• service.name
• ed.source.name
• ed.source.type
• ed.domain
• k8s.node.name
• k8s.namespace.name
• k8s.pod.name
• k8s.container.name
• container.image.name
• k8s.pod.uid
• container.id
• ed.filepath
• k8s.statefulset.name
• k8s.daemonset.name
• k8s.replicaset.name
• k8s.job.name
• k8s.cronjob.name
• k8s.deployment.name
• event.domain
• event.name

attributes: ed.env.id

You might choose to remove the following if they are not populated or not used for filtering, routing, dashboards, alerting, or compliance:

• host.name: Remove if host-level visibility is not needed.
• host.ip: Remove if IP-based identification or filtering isn’t required.
• k8s.node.name: Remove if you don’t need node-level granularity.
• k8s.pod.name: Remove if you don’t need to correlate events to pod names (e.g., if k8s.pod.uid suffices).
• k8s.container.name: Remove if you’re not breaking down logs by container.
• container.image.name: Remove if container versioning or auditing is not needed.
• k8s.pod.uid: Remove if you’re not relying on the unique pod identifier.
• container.id: Remove unless you need deep container-specific tracing.
• ed.filepath: Remove if the file path inside the container isn’t meaningful for analysis.
• k8s.statefulset.name, k8s.daemonset.name, k8s.replicaset.name, k8s.job.name, k8s.cronjob.name, k8s.deployment.name: These are workload-level metadata. You can remove any of these if you are not running that type of workload or do not require its identification in telemetry.

If you need multi-tenant visibility or traceability, keep host.name, host.ip, k8s.namespace.name, and service.name. For security auditing, keep container.id, container.image.name, and ed.filepath. For troubleshooting specific workloads, keep the corresponding controller fields (e.g., k8s.job.name for batch jobs).

Selecting Metadata

1. In the visual builder for the pipeline, turn on Edit Mode.
2. Add a source node, or double click an existing node to edit it.
3. Expand Advanced Settings.
4. Select a Metadata Level.

• Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
• Default: This option includes the required resources and attributes plus those selected by Edge Delta
• High: This option includes the required resources and attributes along with a larger selection of common optional fields.
• Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

5. Optionally, if you selected Custom, choose which metadata to include by ensuring it has a checkmark.

6. Click Save.
7. Click Review Changes.
8. Click Save Changes.

YAML

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

This is a Custom example:

- name: kubernetes_input
  type: kubernetes_input
  user_description: loggen
  include:
  - k8s.namespace.name=loggen
  source_metadata:
    resource_attributes:
      container.id: true
      container.image.name: true
      ed.domain: true
      ed.filepath: false
      ed.source.name: true
      ed.source.type: true
      event.domain: true
      event.name: true
      host.ip: true
      host.name: false
      k8s.container.name: true
      k8s.cronjob.name: true
      k8s.daemonset.name: false
      k8s.deployment.name: true
      k8s.job.name: true
      k8s.namespace.name: true
      k8s.node.name: true
      k8s.pod.name: false
      k8s.pod.uid: false
      k8s.replicaset.name: false
      k8s.statefulset.name: true
      service.name: true
    attributes:
      ed.env.id: true

While this is a Required Only example:

- name: kubernetes_input
  type: kubernetes_input
  user_description: loggen
  include:
  - k8s.namespace.name=loggen
  source_metadata:
    resource_attributes:
      ed.domain: true
      ed.source.name: true
      ed.source.type: true
      event.domain: true
      event.name: true
      service.name: true
    attributes:
      ed.env.id: true