Integrate Edge Delta's Edge, Coordinator, and Gateway Pipelines

Ensure that your Edge Delta edge, gateway and coordinator pipelines are working together.

  7 minute read  

Overview

Edge Delta pipelines work together to process telemetry data efficiently in Kubernetes environments. You deploy the edge pipeline directly on your infrastructure, such as within Kubernetes clusters. These agents run close to the data source, allowing them to process logs and metrics immediately. They extract insights, generate alerts, compress and roll up data, and then send the output to the Edge Delta backend or other destinations.

Use the gateway pipeline to collect and process data from multiple edge agents or external sources. It aggregates metrics, removes duplicate logs, and supports service-level views. You can deploy multiple gateway agents to scale processing as needed. Connect edge and gateway pipelines using matching gateway input and output nodes.

Tail sampling enables you to decide whether to retain or drop a trace after seeing the entire set of spans that make it up. In a distributed environment, each edge pipeline collects only a subset of spans, making trace-level sampling impossible at the agent level. By routing all spans to a gateway pipeline, which buffers and processes them centrally, you can apply tail sampling logic to complete traces.

The coordinator pipeline manages communication between edge agents within the cluster. It also coordinates gateway pipelines within the same namespace. It handles control signals from the backend, reduces redundant data collection, and uses the Kubernetes API to discover and organize agents. Deploy one coordinator per cluster in the same namespace as the edge pipeline.

See:

Logical Grouping

The first aspect of pipeline integration is the Kubernetes Cluster Name you specify when creating the pipelines.

This typically refers to a common cluster that the pipelines are installed in, but it can be used to logically group pipelines that are in different clusters. You can view the pipelines grouped together in the pipelines table:

The Group by cluster option is enabled. In this instance there are two groups. One contains just an edge cluster prod005, the other contains an edge, coordinator and gateway cluster.

The cluster name becomes a resource field in your data items and you can search for data items with that resource:

This will add ed.cluster.name:"jpb-test-cluster" to the search string.

Integrating a Coordinator with an Edge Pipeline

The coordinator pipeline doesn’t require any manual integration or setup. Once installed, it automatically discovers Edge Delta agents in the same Kubernetes namespace using the Kubernetes API and begins coordinating them.

In large Kubernetes clusters, where an edge pipeline deploys processor agents as a DaemonSet (so all nodes share the same pipeline configuration), direct communication between each agent and the cloud backend can quickly become inefficient. As the number of nodes increases, features like live log tailing or cluster-wide data aggregation can become slow or unreliable due to the high volume of direct backend interactions.

By deploying a single coordinator pipeline, you shift coordination tasks to a dedicated control-plane agent. The coordinator handles backend communication on behalf of the edge agents—managing heartbeats, control messages, and live feature orchestration. It also performs tasks like leader election for the processor agents, ensuring only one node executes certain centralized logic. This setup reduces network overhead, speeds up agent–backend interactions, and improves the scalability and responsiveness of observability features across the cluster.

Integrating an Edge Pipeline with a Gateway Pipeline

How you integrate the edge and gateway pipelines depends on your deployment:

  • edge pipeline and gateway in the same cluster, or
  • gateway in a different cluster or host

Fleet and Gateway in the same cluster

In this setup, the edge and gateway pipelines are deployed in the same Kubernetes cluster.

The edge pipeline connects to the gateway via the gateway’s headless service, which is discovered automatically. This is the simplest and most common scenario in Kubernetes.

Configure the Gateway Pipeline

When you deploy a gateway pipeline, an ED Pipeline Source node is added to the configuration, and configured to listen on port 4319 for gRPC traffic by default. If you are connecting it to an edge pipeline in the same Kubernetes Cluster, this configuration does not need to change.

Configure the Edge Pipeline

  1. Open the edge pipeline visual builder.
  2. Turn on Edit mode.
  3. Add the Gateway Connection destination node.
  4. Select the gateway pipeline name with the appropriate gateway source node name (if it has more than one input) from the Gateway list.
  1. Click Save Changes.

Gateway in a different cluster or host

Here, the edge pipeline and gateway pipeline are deployed in different Kubernetes clusters or on different hosts.

You can use DNS or a static IP to connect to the gateway.

Configure the Edge Pipeline

DNS

The edge pipeline connects using a DNS hostname that resolves to the gateway’s address. Use this when the gateway is exposed via an Ingress, LoadBalancer, or public DNS name.

  1. Open the edge pipeline visual builder
  2. Turn on edit mode
  3. Add the Gateway Connection destination node.
  4. Select Custom in the Gateway field.
  5. Select DNS for the Endpoint Resolution Type.
  6. Specify one or more gateway Hostnames: Public or internal DNS name pointing to the gateway.
  7. Optionally, specify an interval: How often the agent re-resolves DNS, helpful if using a dynamic load balancer
  8. Specify the Pipeline you want to associate the node with in the UI.
  1. Click Save Changes.

Static IP

In this case, the edge pipeline connects to the gateway using a fixed IP address. This is useful in static environments or where DNS is not available or desired.

  1. Open the edge pipeline visual builder
  2. Turn on edit mode
  3. Add the Gateway Connection destination node.
  4. Select Custom in the Gateway field.
  5. Select Static for the Endpoint Resolution Type.
  6. Specify one or more Endpoints: IPs or URLs for gateway instances
  7. Specify the Pipeline you want to associate the node with in the UI.
  1. Click Save Changes.

Configure the Gateway Pipeline

When the edge pipeline and gateway pipeline are deployed in different clusters, the edge pipeline sends data over a public or private network using either DNS names or static IP addresses. In this scenario, the gateway must be externally reachable, typically through a Kubernetes Ingress or LoadBalancer service.

The gateway’s ED Pipeline Source node must be configured to accept external gRPC or HTTP traffic in a way that aligns with how it’s exposed:

  • The port field should match the internal port that the gateway container listens on.
  • The Kubernetes service (LoadBalancer, Ingress, or NodePort) should expose that internal port to the outside world.
  • TLS settings should reflect where TLS termination occurs — either in the gateway itself or at the ingress layer.

Balancing Traffic

When deploying multiple Edge Delta gateway instances, it’s important to understand how traffic is distributed to them especially in the context of Kubernetes networking.

Kubernetes Services (e.g., ClusterIP or LoadBalancer) provide effective load balancing for data pipelines. However, Kubernetes load balancing occurs at the TCP connection level. The edge pipeline implements application-layer routing logic: Consistent or Round Robin.

To set the Target Allocation Type, specify it in the Gateway Connection destination node of the edge pipeline.

Consistent

Consistent allocation ensures that related data, such as logs from the same service, is always sent to the same gateway pod. This is essential for deduplication, aggregation, and trace correlation.

This example uses consistent allocation. The otel-collector service’s traffic is sent only to gateway pod edgedelta-gateway-84d5dc785f-7vct6, while cart traffic is sent only to gateway pod edgedelta-gateway-84d5dc785f-6hnkl.

Note: If your gateway pipeline is behind a loadbalancer it may be subject to connection level load balancing. This means that even with consistent allocation specified at the edge, the service traffic will be distributed between the gateway pods.

Round Robin

Round Robin allocation distributes data across all available gateway pods in turn, ensuring an even distribution regardless of source.

This example uses round_robin allocation. The otel-collector service’s traffic (and all services’ traffic) is evenly distributed across both gateway pods edgedelta-gateway-84d5dc785f-7vct6 and edgedelta-gateway-84d5dc785f-6hnkl.