Edge Delta Architecture

  4 minute read  

Architectural Overview

At a high level, Edge Delta consists of three key components:

  1. Agents and Integrations – Lightweight agents run on hosts, containers, or sidecars in Kubernetes. These collect telemetry data from a wide range of sources. Cloud integrations allow collection from managed services without deploying agents everywhere.

  2. Edge Processing Pipelines – These pipelines run locally on the agents. They parse, enrich, filter, and transform data before it leaves the system. They can also extract metrics, perform anomaly detection, and detect patterns in real time.

  3. Destinations – In addition to the Edge Delta back end, processed or raw data is routed to one or more destinations. These might include log storage systems, metrics backends, tracing systems, SIEM tools, or object stores like Amazon S3.

Agent architecture

Node Pipeline

A pipeline is a set of agents with a single configuration. There are different types of pipelines that you can deploy depending on your architecture:

In a node pipeline, Edge Delta agents deploy directly within your computing infrastructure, such as Kubernetes clusters. This deployment strategy places it close to the data sources, facilitating immediate analysis and data optimization on the edge. The Edge Delta agents pre-process data, which includes extracting insights, generating alerts, creating summarized datasets, and performing additional tasks. Processed data is then transmitted to various endpoints in a streamlined manner, including Edge Delta’s processing engine (Edge Delta Back End), external monitoring tools, or data storage solutions.

Edge Delta’s agents include the Processing Agent, Compactor Agent,and Rollup Agent. The Processing Agent executes the pipelines. The Compactor Agent is designed to compress and encode data such as metrics and logs into efficient formats. The Rollup Agent aggregates metric data by optimizing data point frequency and cardinality, which notably reduces storage needs and can accelerate data retrievals.

Gateway Pipeline

The gateway pipeline serves as a central aggregation and processing hub within a Kubernetes environment. Its primary role is to collect and process telemetry data from multiple sources, including node pipelines operating at the node level and external inputs. This centralized approach allows the gateway to perform tasks such as service-level metric aggregation and log deduplication.

The gateway pipeline supports trace tail sampling, allowing decisions based on the full trace after it’s been collected. Since no single agent sees the whole trace, tail sampling must happen at the gateway, where all spans converge.

It can also ingest data from external sources via network protocols, meaning information not captured by node pipelines can still be processed by the system.

Typically deployed as a scalable set of agents in Kubernetes, gateway pipelines allow for multiple instances within a cluster, each potentially handling different data types or processing tasks. Connectivity is facilitated through a specialized gateway input to receive data from multiple agents and an Output in node pipelines specifically configured to send data to the gateway.

Coordinator Pipeline

The coordinator pipeline functions as a control plane agent, facilitating communication and coordination among node agents within a Kubernetes cluster. It is specifically designed to manage cluster-wide tasks and streamline agent management. One of its key functions is to act as an intermediary for control signals between the backend and node agents, effectively reducing communication overhead in larger clusters.

Deployed as a single-replica Kubernetes Deployment, there is only one coordinator per cluster, which prevents duplicate data collection and ensures effective cluster management. It also uses the Kubernetes API to discover other pipeline agents within the cluster, assisting in establishing groupings that provide a cohesive view. A coordinator is required to enable live capture in clusters with more than one node.

The gateway focuses on the aggregation and processing of telemetry data at the cluster level, operating as a central hub for both agent-collected and external data. Meanwhile, the coordinator is centered on cluster coordination and management, facilitating efficient operations and preparing for enhanced cluster-wide data capabilities in the future.

See Edge Delta Pipeline Installation for architecture options.