Edge Delta Kubernetes Manifests

Kubernetes Manifests

You can generate kubernetes manifests for the pipelines using Helm.

Node Pipeline

The node pipeline manifest contains the following sections:

• Namespace: The edgedelta namespace is reserved for all Edge Delta components, providing a separate environment to enhance organization and isolation from other services within the cluster.
• ServiceAccount: A service account named edgedelta is created specifically for authentication purposes. This account is vital for ensuring secure interactions of Edge Delta components with the Kubernetes API.
• ClusterRole and ClusterRoleBinding: These elements assign the necessary permissions to the edgedelta service account. The default ClusterRole rules are defined in clusterRoleRules in values.yaml, allowing you to modify or replace permissions directly. Use additionalClusterRoleRules to append custom rules for CRDs.
• DaemonSet: Deploys the Edge Delta Processor Agent across all nodes in the cluster via a DaemonSet, ensuring extensive data collection and processing coverage. The agent is provided with significant privileges to access and gather necessary metrics and logs through system interactions. It leverages host-level permissions like hostPID and hostNetwork, with volume mounts for persistent storage and access to critical system paths.
• Services: Two distinct services are configured: edgedelta-compactor-svc and edgedelta-rollup-svc. These services establish communication paths for backend processing units, using ports 9199 and 9200, respectively, to maintain connectivity with the Compactor and Rollup agents.
• Deployments:
  • edgedelta-rollup: This Deployment manages the Rollup component (2 replicas), facilitating metric data aggregation. It includes configurations for container resources and a rolling update strategy.
  • edgedelta-compactor: This Deployment manages the Compactor (1 replica), processing and compressing log data for downstream delivery. It includes container resource limits and a rolling update strategy.

Kubernetes Permissions

Role-Based Access Control (RBAC) is configured for Edge Delta out of the box. The ClusterRole defines a set of permissions for resources across the Kubernetes cluster. The default rules grant the following access:

The edgedelta service account and its ClusterRoleBinding grant the agent these permissions. You can customize the rules through clusterRoleRules in values.yaml, or append rules for custom resources using additionalClusterRoleRules.

The DaemonSet deploys the Edge Delta Processor Agent across all nodes, ensuring monitoring and log aggregation coverage across the cluster. The agent requires elevated privileges including hostPID and hostNetwork to access node-level data and system logs.

For some environments, additional securityContext configurations are required. For example see SELinux-enforced Kubernetes clusters.

Identity and Access Management (IAM) in Cluster

Within Kubernetes, IAM is primarily managed using service accounts in conjunction with RBAC (Role-Based Access Control). Each service account, like the edgedelta account in your deployment, is assigned specific roles and permissions, dictated by RBAC. Roles and role bindings ensure that each component of Edge Delta has access only to the required resources necessary for its operation. This integration of IAM ensures least privilege access, reducing the risk of unauthorized operations. IAM in Kubernetes is designed to separate roles and responsibilities, allowing developers, administrators, and operations teams to have access only to the tools and data they need. This approach limits the impact of potential security breaches or insider threats.

When running Kubernetes in cloud environments, integrating cloud IAM with Kubernetes RBAC becomes crucial. Cloud IAM roles and permissions can be extended to control access to cloud-native services and resources that Edge Delta might interact with, such as storage, networking, or specific APIs.

See AWS Authentication for Edge Delta and AWS Assumed Roles for Edge Delta.

Coordinator Pipeline

The manifest for the Edge Delta coordinator specifies configurations and resource specifications necessary for deploying and operating the coordinator pipeline within a Kubernetes cluster

Namespace and ServiceAccount:

• The manifest establishes the edgedelta namespace if not created already, this delineates a reserved space for Edge Delta components, ensuring their operations are segregated from other applications within the Kubernetes environment.
• A ServiceAccount named edgedelta-coordinator is created to enable secure API interactions.

Role-Based Access Control (RBAC):

The edgedelta-coordinator ClusterRole grants the following permissions:

A ClusterRoleBinding associates these permissions with the edgedelta-coordinator service account. You can customize the rules through clusterRoleRules in the coordinator chart’s values.yaml.

Service Configuration:

This configuration sets up a network path within the Kubernetes cluster for the Edge Delta coordinator pipeline to communicate efficiently with other components.

Deployment Strategy:

A Deployment is defined with a single replica, leveraging a rolling update strategy. This approach balances availability with reliability as updates are rolled out, minimizing service disruptions.

Storage:

A PersistentVolumeClaim (1Gi, ReadWriteOnce) is created and mounted at /edgedelta/data to persist coordinator state across restarts.

Environment and Resources:

The container environment is configured to include vital parameters such as ED_COORDINATOR_SERVER_ENABLED, ED_COORDINATOR_SERVER_PORT, ED_COORDINATOR_USE_BACKEND, and ED_DATA_PATH, ensuring the coordinator can efficiently manage operations and communications within Edge Delta. Resource allocations emphasize CPU and memory constraints, adhering to operational best practices for performance and efficiency.

Identity and Access Management (IAM):

Identity and Access Management (IAM) for the Edge Delta coordinator is structured using service accounts and Role-Based Access Control (RBAC). The edgedelta-coordinator service account is central to this setup, providing the coordinator with authenticated access to the Kubernetes API. RBAC enhances security by assigning specific roles and permissions through a ClusterRole tailored for the coordinator’s operations, encompassing permissions to manage resources like pods, events, and nodes. Binding this role to the edgedelta-coordinator service account ensures it operates within the scope of least privilege, mitigating the risk of unauthorized actions.

Gateway Pipeline

The gateway manifest, generated via Helm, deploys the core components required for the Edge Delta gateway pipeline type. It includes the following key sections:

• Namespace: All components are deployed into the edgedelta namespace to maintain logical separation and simplify management of Edge Delta workloads.
• ServiceAccount: A dedicated edgedelta-gateway ServiceAccount is created to securely authenticate components interacting with the Kubernetes API. This helps enforce strong identity and access boundaries for observability and processing workloads.
• ClusterRole and ClusterRoleBinding: The edgedelta-gateway ClusterRole defines permissions across core and extended Kubernetes APIs. A corresponding ClusterRoleBinding links the ClusterRole to the edgedelta-gateway ServiceAccount. You can customize the rules through clusterRoleRules in the gateway chart’s values.yaml.
• Deployments: The gateway consists of three distinct agent roles: Processor, Compactor, and Rollup, each deployed as separate Kubernetes Deployments:
  • edgedelta-gateway (Processor): A scalable deployment (2 replicas) that forms the primary data ingestion and processing tier. It collects metrics, logs, and traces from the cluster and coordinates further backend processing. It connects with the Compactor and Rollup services for downstream handling.
  • edgedelta-gateway-compactor: A single-replica deployment responsible for compressing and optimizing raw logs. The Compactor acts as a backend processor with an exposed port (9199) used internally by other agents like the Processor and Rollup.
  • edgedelta-gateway-rollup: A 2-replica deployment that handles metric aggregation and rollup tasks, enabling efficient storage and forwarding of metrics. This deployment exposes port 9200 and is referenced by the Processor for metric forwarding.
• Services: Internal cluster services are configured for inter-agent communication:
  • edgedelta-gateway-compactor-svc: Exposes the Compactor on port 9199 to other gateway agents.
  • edgedelta-gateway-rollup-svc: Provides access to the Rollup agent on port 9200.
  • gw-svc: A headless service (ClusterIP: None) that targets Processor pods for stateful or identity-aware operations.

Kubernetes Permissions

The gateway ClusterRole grants the following permissions:

These permissions allow the gateway to function autonomously and securely, providing observability, diagnostics, and telemetry across the Kubernetes environment.

IAM and Secure API Access

The use of a dedicated ServiceAccount (edgedelta-gateway) scoped via a ClusterRoleBinding ensures secure interactions with the Kubernetes API. This is a key aspect of Kubernetes-native Identity and Access Management (IAM), enabling each agent to operate with the principle of least privilege. Each component receives credentials automatically via the Kubernetes token projection mechanism, eliminating the need for embedded static secrets.

In cloud environments, this setup can be extended with cloud IAM integrations to support features like IRSA (IAM Roles for Service Accounts) in AWS, enabling access to cloud-native resources (e.g., object stores, secrets managers, telemetry endpoints) directly from Kubernetes workloads.