Source Detection with Edge Delta

Overview

Knowledge of your own environment and its telemetry data is crucial for developing effective data pipelines. In the past, a deployment pattern for Edge Delta was to deploy a catch all input, such as a Kubernetes source node with all namespaces included. Then you could optimize the pipeline by examining, and learning about the logs, metrics and events as they are processed.

Source Detection enables you to accelerate this process by identifying exiting sources after deploying agents but before configuring a pipeline for the fleet.

To use source detection, deploy a new fleet.

Defaults

If you choose to select the default sources, a generic source node is created with a wide net. For example in Kubernetes environments, a Kubernetes source node is automatically added to the new pipeline. It is configured to ingest all stdout messages from all namespaces, except messages from the edgedelta pods:

include:
- k8s.namespace.name=.*
exclude:
- k8s.pod.name=edgedelta

Specific Sources

If you choose to select a specific sources, the source node is created with a more focused net. For example in Kubernetes environments, a Kubernetes source node is automatically added to the new pipeline. It is configured to ingest all stdout messages from only the pods and namespaces of the sources you selected:

include:
- k8s.pod.name=broker-765b6b4794.*,k8s.namespace.name=broker

In this example, a workload named broker, in the broker namespace has messages consumed.

Source Detection Nodes

To enable source detection, two hidden nodes are added to all pipelines: A source detection input and a source detection output. These nodes have no configurable parameters.