Edge Delta Kubernetes Source

Overview

You can use the Kubernetes source node to specify which Kubernetes pods and namespaces the agent should monitor.

AI Team: Configure this source using the Kubernetes Logs connector for streamlined setup in AI Team.

• outgoing_data_types: log

Example Configuration

nodes:
- name: my_k8s_input
  type: kubernetes_input
  include:
    - k8s.namespace.name=.*
  exclude:
    - k8s.namespace.name=kube-system
    - k8s.namespace.name=kube-public
    - k8s.namespace.name=kube-node-lease
    - k8s.pod.name=edgedelta
    - k8s.pod.name=prometheus
    - k8s.pod.name=promtail
    - k8s.pod.name=node-exporter
  resource_fields:
    pod_labels:
      - app.kubernetes.io/instance
      - app.kubernetes.io/name
      - helm.sh/chart
      - kubernetes.io/.*
    pod_annotations:
      - cluster-autoscaler.kubernetes.io/safe-to-evict
      - cluster-autoscaler.kubernetes.io/.*
    node_labels:
      - beta.kubernetes.io/arch
      - beta.kubernetes.io/instance-type
      - eks.amazonaws.com/.*
      - topology.kubernetes.io/zone
    namespace_labels:
      - kubernetes.io/metadata.name
      - name
  line_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
  preserve_original_timestamp: true
  enable_persisting_cursor: true

Controlling Ingestion Based on Object Tenancy

Each include/exclude line prefaced with a - is a complete object reference, and each object is treated as a complete set of selection criteria. For example, this configuration will exclude all telemetry data from namespace nginx and all pods that start with webserver anywhere in the cluster:

    exclude:
      - k8s.namespace.name=nginx
      - k8s.pod.name=webserver.*

However the following configuration will only exclude telemetry data from pods prefaced with webserver when they exist in the namespace nginx. webserver.* pods that exist in other namespaces will not be excluded in this example:

    exclude:
      - k8s.namespace.name=nginx,k8s.pod.name=webserver.*

Required Parameters

name

A descriptive name for the node. This is the name that will appear in pipeline builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: kubernetes_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Optional Parameters

auto_detect_line_pattern

The auto_detect_line_pattern parameter is a Boolean value that, when set to true, enables automatic detection of line patterns in logs. This automatically determine the structure of log lines, rather than relying on pre-defined patterns.

nodes:
  - name: my_input
    type: kubernetes_input
    auto_detect_line_pattern: true

The auto_detect_line_pattern parameter is optional and defaults to false if not specified.

boost_stacktrace_detection

The boost_stacktrace_detection parameter is used with auto_detect_line_pattern: true. It enables stack trace detection based on the Ragel FSM Based Lexical Recognition process. Found stack traces will be grouped together in the same log message. It is specified with a Boolean, the default is false and it is optional.

nodes:
  - name: my_input
    type: kubernetes_input
    auto_detect_line_pattern: true
    boost_stacktrace_detection: true

enable_persisting_cursor

The enable_persisting_cursor parameter is used to enable a persisting cursor in the event of an agent restart. It is specified as a Boolean, the default is false and it is optional.

nodes:
  - name: <node name>
    type: kubernetes_input
    enable_persisting_cursor: true

discovery

The discovery parameter over-rides Edge Delta’s default file discovery logic for looking for files on Kubernetes pods in mounted filesystems. This is useful for retaining Kubernetes metadata context. You specify two parameters:

• file_path: A glob path that defines a source for logs files.
• parsing_pattern: A Go regex pattern to parse captured literal file paths in order to observe the pod UID. This requires a capture group called pod_uid. You can optionally observe the container name, which requires the capture group container_name.

nodes:
  - name: <node name>
    type: kubernetes_input
    discovery:
      file_path: /var/logs/pods/*/kubernetes.io~emptyDir/**/*.log
      parsing_pattern: ^/var/logs/pods/(?P<pod_uid>[a-zA-Z0-9\-]+)/(?P<container_name>[a-zA-Z0-9\-_]+)/.*

exclude

The exclude parameter is used to excluded traffic from certain components from being ingested. It is specified in OTEL format with Golang regex or full text. If exclude is specified it drops logs that match its Golang regex rules. Comma separated values indicate AND rules, while separate rules indicate OR matching.

In this example logs that match rule-1 and rule-2 are excluded. Logs that match rule-3 are also excluded.

      exclude:
        - "rule-1,rule-2"
        - "rule-3"

Exclude is optional.

• If include is specified as well as exclude, then logs matching in include are passed unless they are dropped by exclude.

nodes:
  - name: <node name>
    type: kubernetes_input
    include:
    - k8s.namespace.name=.*
    exclude:
    - k8s.namespace.name=kube-system

Filtering Options

The exclude parameter supports multiple filtering dimensions to provide fine-grained control over which logs are collected:

File Path Filtering

You can filter logs based on file paths using the ed.filepath attribute. This is particularly useful for managing large Kubernetes deployments where generic file path patterns in discovery might match hundreds or thousands of sources:

nodes:
  - name: k8s_logs
    type: kubernetes_input
    discovery:
      file_path: /var/log/pods/**/*.log
      parsing_pattern: ^/var/log/pods/(?P<pod_uid>[^/]+)/(?P<container_name>[^/]+)/.*
    exclude:
      # Exclude specific log file patterns
      - ed.filepath=/var/log/pods/.*/kube-proxy.*
      - ed.filepath=.*system-component.*
      # Combine file path with namespace filtering
      - ed.filepath=.*/debug\.log,k8s.namespace.name=development

Recommended Configuration Patterns

For optimal performance and manageable configurations, consider these filtering strategies:

1. Namespace-based exclusion for system components:

exclude:
  - k8s.namespace.name=kube-system
  - k8s.namespace.name=kube-public
  - k8s.namespace.name=kube-node-lease

2. Combined filtering for specific workloads:

exclude:
  # Exclude debug logs from development namespaces
  - ed.filepath=.*/debug\.log,k8s.namespace.name=dev.*
  # Exclude verbose logging from specific deployments
  - k8s.deployment.name=load-generator,k8s.container.name=stress-test

3. Label-based exclusion for test environments:

exclude:
  - k8s.pod.labels.environment=test
  - k8s.pod.labels.log-level=debug

4. File pattern exclusion for high-volume, low-value logs:

exclude:
  # Exclude health check logs
  - ed.filepath=.*/health-check\.log
  # Exclude access logs from specific paths
  - ed.filepath=.*/nginx/access\.log,k8s.deployment.name=frontend

The following values can be excluded from a Kubernetes source:

• k8s.pod.name=
• k8s.namespace.name=
• k8s.cronjob.name=
• k8s.daemonset.name=
• k8s.deployment.name=
• k8s.job.name=
• k8s.replicaset.name=
• k8s.statefulset.name=
• k8s.pod.labels.=
• k8s.container.name=
• container.image.name=
• ed.filepath= (when using discovery mode)

For example:

• k8s.namespace.name=.*
• k8s.namespace.name=kube-system
• k8s.namespace.name=nginx,k8s.deployment.name=nginx-canary.*
• k8s.pod.labels.app=nginx

You can use k8s.{kind}.name=.+ rather than k8s.controller.kind=xxx (which is no longer supported), where the values of {kind} match the Kubernetes kind, such as k8s.daemonset.name, k8s.deployment.name, k8s.job.name, etc.

include

The include parameter is used to specify the scope of traffic sources to be ingested. It is specified in OTEL format with Golang regex or full text. If exclude is also specified, then logs matching in include are passed unless they are dropped by exclude. Include is optional.

nodes:
  - name: <node name>
    type: kubernetes_input
    include:
    - k8s.namespace.name=.*

You can also filter by file paths:

nodes:
  - name: k8s_logs
    type: kubernetes_input
    include:
      # Include only application logs
      - ed.filepath=.*/app\.log
      # Include specific namespace and file pattern
      - k8s.namespace.name=production,ed.filepath=.*/error\.log

The following values can be included in the scope for a Kubernetes source:

• k8s.pod.name=
• k8s.namespace.name=
• k8s.cronjob.name=
• k8s.daemonset.name=
• k8s.deployment.name=
• k8s.job.name=
• k8s.replicaset.name=
• k8s.statefulset.name=
• k8s.pod.labels.=
• k8s.container.name=
• container.image.name=
• ed.filepath= (when using discovery mode)

For example:

• k8s.namespace.name=.*
• k8s.namespace.name=kube-system
• k8s.namespace.name=nginx,k8s.deployment.name=nginx-canary.*
• k8s.pod.labels.app=nginx

You can use k8s.{kind}.name=.+ rather than k8s.controller.kind=xxx (which is no longer supported), where the values of {kind} match the Kubernetes kind, such as k8s.daemonset.name, k8s.deployment.name, k8s.job.name, etc.

line_pattern

The line_pattern parameter specifies a Golang regex pattern that will be used as a line break rather than using a new line. It is specified with a Golang regex pattern, including a ^ for the line start, and it is optional.

nodes:
  - name: <node name>
    type: <node type>
    line_pattern: ^\d{4}-\d{2}-\d{2}

log_parsing_mode

The log_parsing_mode parameter determines how the source node processes structured log data, particularly JSON-formatted logs. In the GUI, this setting appears as Log Parsing Mode.

Available options:

• basic: Treats all log data as plain text. JSON logs are preserved as escaped strings in the body field without any parsing or field extraction. This mode ensures the original log format is maintained exactly as received.

• full (or json): Intelligently parses structured logs. Valid JSON is parsed into structured objects with individual fields becoming queryable attributes. Arrays, nested objects, and data types are preserved. Invalid JSON or plain text logs gracefully fall back to string format. This mode provides the same functionality as adding a Parse JSON Processor to your pipeline, but applies the parsing directly at the source.

When to use each mode:

• Use basic mode when:

  • Downstream systems will handle JSON parsing
  • You need exact log preservation
  • Dealing with mixed or potentially malformed JSON
  • Performance is critical and parsing overhead must be minimized

• Use full mode when:

  • You need to search, filter, or alert on JSON field values
  • You want to transform or enrich based on JSON content
  • Your logs are consistently well-formed JSON
  • You need to extract specific fields for metrics or analysis

Example configuration:

nodes:
  - name: <source_name>
    type: <source_type>
    log_parsing_mode: full
    # Other source configuration parameters

preserve_original_timestamp

The preserve_original_timestamp parameter is used to configure the agent to preserve the timestamp that the incoming log already has. It is specified as a Boolean value and the default is false. It is optional.

nodes:
  - name: <node name>
    type: kubernetes_input
    preserve_original_timestamp: true

resource_fields

The resource_fields parameters define the labels and annotations to be collected as metadata fields and included. It is specified as a dictionary of regexes and is optional.

You can specify:

• pod_labels
• pod_annotations
• node_labels
• namespace_labels

nodes:
  - name: <node name>
    type: kubernetes_input
    resource_fields:
      pod_labels:
        - app.kubernetes.io/instance
        - app.kubernetes.io/name
        - helm.sh/chart
        - kubernetes.io/.*
      pod_annotations:
        - cluster-autoscaler.kubernetes.io/safe-to-evict
        - cluster-autoscaler.kubernetes.io/.*
      node_labels:
        - beta.kubernetes.io/arch
        - beta.kubernetes.io/instance-type
        - eks.amazonaws.com/.*
        - topology.kubernetes.io/zone
      namespace_labels:
        - kubernetes.io/metadata.name
        - name

sampling

The sampling parameter specifies the sampling rate for the input payloads. It is specified with a Boolean with false being 0.0 and true being 1.0, the default is false, and it is optional.

nodes:
  - name: my_input
    type: kubernetes_input
    sampling: true

source_metadata

This option is used to define which detected resources and attributes to add to each data item as it is ingested by Edge Delta. You can select:

• Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
• Default: This option includes the required resources and attributes plus those selected by Edge Delta
• High: This option includes the required resources and attributes along with a larger selection of common optional fields.
• Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.

See Also

Exclude 5 Namespaces