Edge Delta Pipeline Installation

Recap

A pipeline is a set of agents with a single configuration. There are different types of pipelines that you can deploy depending on your architecture:

Node Pipeline

In a node pipeline, Edge Delta agents deploy directly within your computing infrastructure, such as Kubernetes clusters. This deployment strategy places it close to the data sources, facilitating immediate analysis and data optimization on the edge. The Edge Delta agents pre-process data, which includes extracting insights, generating alerts, creating summarized datasets, and performing additional tasks. Processed data is then transmitted to various endpoints in a streamlined manner, including Edge Delta’s processing engine (Edge Delta Back End), external monitoring tools, or data storage solutions.

In kubernetes, node pipeline agents include the Processing Agent, Compactor Agent,and Rollup Agent. The Processing Agent executes the pipelines. The Compactor Agent is designed to compress and encode data such as metrics and logs into efficient formats. The Rollup Agent aggregates metric data by optimizing data point frequency and cardinality, which notably reduces storage needs and can accelerate data retrievals.

Gateway Pipeline

Note: the gateway pipeline is currently for Kubernetes only.

The gateway pipeline serves as an aggregation and processing hub. Its primary role is to collect and process telemetry data from multiple sources. This centralized approach allows the gateway to perform tasks such as service-level metric aggregation and log deduplication.

In distributed tracing, a trace is composed of spans emitted from multiple services or nodes. Each node pipeline collects spans locally but has no visibility into the rest of the trace. This makes it impossible to make intelligent sampling decisions based on the trace’s complete picture at the agent level.

To address this, all node pipelines forward their spans to the gateway, which acts as a central point of aggregation. The gateway buffers these spans (in memory or on disk) and applies a Tail Sampling processor that can evaluate entire traces. This centralization allows the system to make informed sampling decisions—such as retaining error traces or dropping low-value ones—because it has visibility into all parts of the trace.

Tail Sampling helps balance observability and resource usage by retaining only the most meaningful traces. Since only the gateway has a complete view of the trace, you must deploy a gateway pipeline if you require tail sampling.

It can also ingest data from external sources, meaning information not captured by node agents can still be processed by the system.

Typically deployed as a scalable set of agents in Kubernetes, gateway pipelines allow for multiple instances within a cluster, each potentially handling different data types or processing tasks. The gateway pipeline is connected to node pipelines using a gateway Input node (in the gateway pipeline) and a matching gateway output node (in node pipelines).

Coordinator Pipeline

Note: the coordinator pipeline is for Kubernetes only.

The coordinator pipeline functions as a control plane agent, facilitating communication and coordination among node agents within a Kubernetes cluster. It is specifically designed to manage cluster-wide tasks and streamline agent management. One of its key functions is to act as an intermediary for control signals between the backend and node agents, effectively reducing communication overhead in larger clusters.

Deployed as a single-replica Kubernetes Deployment, there is only one coordinator per cluster, which prevents duplicate data collection and ensures effective cluster management. It also uses the Kubernetes API to discover other pipeline agents within the cluster, assisting in establishing groupings that provide a cohesive view.

The coordinator pipeline must be deployed in the same namespace as the node pipeline it is managing (usually edgedelta).

Kubernetes Details

See Kubernetes Manifests for details about the manifests for each pipeline.

See Kubernetes Deployment Examples.

Related Content