Edge Delta Kubernetes Manifests

Kubernetes Manifests

You can generate kubernetes manifests for the pipelines using Helm.

Edge Pipeline

The edge pipeline manifest contains the following sections:

• Namespace: The edgedelta namespace is reserved for all Edge Delta components, providing a separate environment to enhance organization and isolation from other services within the cluster.
• ServiceAccount: A service account named edgedelta is created specifically for authentication purposes. This account is vital for ensuring secure interactions of Edge Delta components with the Kubernetes API.
• ClusterRole and ClusterRoleBinding: These elements assign the necessary permissions to the edgedelta service account, enabling it to access and manage various Kubernetes resources critical for Edge Delta’s operations. Permissions include getting, listing, watching, creating, updating, patching, and deleting resources such as namespaces, pods, events, nodes, services, daemonsets, deployments, statefulsets, jobs, cronjobs, and leases.
• DaemonSet: Deploys the Edge Delta Processor Agent across all nodes in the cluster via a DaemonSet, ensuring extensive data collection and processing coverage. The agent is provided with significant privileges to access and gather necessary metrics and logs through system interactions. It leverages host-level permissions like hostPID and hostNetwork, with volume mounts for persistent storage and access to critical system paths.
• Services: Two distinct services are configured: ed-compactor-svc and ed-rollup-svc. These services establish communication paths for backend processing units, using ports 9199 and 9200, respectively, to maintain connectivity with the Compactor and Rollup agents.
• Deployments:
  • edgedelta-rollup: This Deployment manages the Rollup component, facilitating metric data aggregation. It includes configurations for container resources, persistent storage, and a rolling update strategy.
  • edgedelta-compactor: This Deployment manages the Compactor, processing log data with configurations that mirror those of the Rollup component, including container setup, resource allocation, and data persistency.

Kubernetes Permissions

Based Access Control (RBAC) is configured for Edge Delta out of the box. In the default Edge Delta installation, the ClusterRole defines a set of permissions (verbs) for certain resources across the entire Kubernetes cluster or within specific namespaces. For Edge Delta, the permissions are critical for its operation as it needs to monitor and log activity across multiple resources. These verbs include get, list, watch, create, update, patch, and delete. These operations allow Edge Delta to observe the cluster state and interact with resource definitions. The permissions give access to certain resources:

• Namespaces, Pods, Events, Nodes: For capturing overall system state and pod-level logs and metrics.
• Services, DaemonSets, Deployments, StatefulSets: To manage and observe the health and status of applications and their components.
• Jobs, CronJobs, Leases: For understanding and monitoring background and scheduled tasks.

A service account ensures Edge Delta components can securely interact with the Kubernetes API to perform operations such as log collection, monitoring, and data processing without manual or insecure handling of credentials. The edgedelta service account, as defined in the manifest, is essential for managing the necessary access to system-level resources in a controlled manner. ClusterRoleBinding associates the ClusterRole with the edgedelta service account. This binding is what actually grants the service account the permissions defined in the ClusterRole.

The DaemonSet is responsible for deploying the Edge Delta Processor Agent across all nodes, ensuring seamless monitoring and log aggregation from all parts of the cluster. Given the nature of its work, the agent requires elevated privileges to access node-level data and system logs. Host-level permissions like hostPID and hostNetwork allow the agent to operate with broader system visibility and network access, which are crucial for collecting comprehensive data.

For some environments, additional securityContext configurations are required. For example see SELinux-enforced Kubernetes clusters.

Identity and Access Management (IAM) in Cluster

Within Kubernetes, IAM is primarily managed using service accounts in conjunction with RBAC (Role-Based Access Control). Each service account, like the edgedelta account in your deployment, is assigned specific roles and permissions, dictated by RBAC. Roles and role bindings ensure that each component of Edge Delta has access only to the required resources necessary for its operation. This integration of IAM ensures least privilege access, reducing the risk of unauthorized operations. IAM in Kubernetes is designed to separate roles and responsibilities, allowing developers, administrators, and operations teams to have access only to the tools and data they need. This approach limits the impact of potential security breaches or insider threats.

When running Kubernetes in cloud environments, integrating cloud IAM with Kubernetes RBAC becomes crucial. Cloud IAM roles and permissions can be extended to control access to cloud-native services and resources that Edge Delta might interact with, such as storage, networking, or specific APIs.

See AWS Authentication for Edge Delta and AWS Assumed Roles for Edge Delta.

Coordinator Pipeline

The manifest for the Edge Delta coordinator specifies configurations and resource specifications necessary for deploying and operating the coordinator pipeline within a Kubernetes cluster

Namespace and ServiceAccount:

• The manifest establishes the edgedelta namespace if not created already, this delineates a reserved space for Edge Delta components, ensuring their operations are segregated from other applications within the Kubernetes environment.
• A ServiceAccount named edgedelta-coordinator is created to enable secure API interactions.

Role-Based Access Control (RBAC):

• The ClusterRole element specifies a comprehensive array of permissions for the coordinator. These permissions include capabilities such as get, list, watch, create, update, patch, and delete for resources like namespaces, pods, nodes, services, daemonsets, deployments, and leases.
• A ClusterRoleBinding associates these permissions with the edgedelta-coordinator service account. This configuration ensures that the coordinator can seamlessly manage Kubernetes resources as needed for its critical functionalities.

Service Configuration:

This configuration sets up a network path within the Kubernetes cluster for the Edge Delta coordinator pipeline to communicate efficiently with other components.

Deployment Strategy:

A Deployment is defined with a single replica, leveraging a rolling update strategy. This approach balances availability with reliability as updates are rolled out, minimizing service disruptions.

Environment and Resources:

The container environment is configured to include vital parameters such as ED_COORDINATOR_SERVER_ENABLED, ED_COORDINATOR_SERVER_PORT, and ED_API_ENDPOINT, ensuring the coordinator can efficiently manage operations and communications within Edge Delta. Resource allocations emphasize CPU and memory constraints, adhering to operational best practices for performance and efficiency.

Identity and Access Management (IAM):

Identity and Access Management (IAM) for the Edge Delta coordinator is structured using service accounts and Role-Based Access Control (RBAC). The edgedelta-coordinator service account is central to this setup, providing the coordinator with authenticated access to the Kubernetes API. RBAC enhances security by assigning specific roles and permissions through a ClusterRole tailored for the coordinator’s operations, encompassing permissions to manage resources like pods, events, and nodes. Binding this role to the edgedelta-coordinator service account ensures it operates within the scope of least privilege, mitigating the risk of unauthorized actions.

Gateway Pipeline

The gateway manifest, generated via Helm, deploys the core components required for the Edge Delta gateway pipeline type. It includes the following key sections:

• Namespace: All components are deployed into the edgedelta namespace to maintain logical separation and simplify management of Edge Delta workloads.
• ServiceAccount: A dedicated edgedelta-gateway ServiceAccount is created to securely authenticate components interacting with the Kubernetes API. This helps enforce strong identity and access boundaries for observability and processing workloads.
• ClusterRole and ClusterRoleBinding: The edgedelta-gateway ClusterRole defines a comprehensive set of permissions across core and extended Kubernetes APIs. It includes access to resources such as Pods, Events, Nodes, Services, and Deployments, along with metrics and coordination APIs. These permissions allow the gateway to observe, process, and act on cluster state and telemetry data. A corresponding ClusterRoleBinding links the ClusterRole to the edgedelta-gateway ServiceAccount, enabling elevated yet scoped access required by the gateway agents.
• Deployments: The gateway consists of three distinct agent roles — Processor, Compactor, and Rollup — each deployed as separate Kubernetes Deployments:
  • edgedelta-gateway (Processor): A scalable deployment (2 replicas) that forms the primary data ingestion and processing tier. It collects metrics, logs, and traces from the cluster and coordinates further backend processing. It connects with the Compactor and Rollup services for downstream handling.
  • edgedelta-gateway-compactor: A single-replica deployment responsible for compressing and optimizing raw logs. The Compactor acts as a backend processor with an exposed port (9199) used internally by other agents like the Processor and Rollup.
  • edgedelta-gateway-rollup: Handles metric aggregation and rollup tasks, enabling efficient storage and forwarding of metrics. This deployment exposes port 9200 and is referenced by the Processor for metric forwarding.
• Services: Internal cluster services are configured for inter-agent communication:
  • edgedelta-gateway-compactor-svc: Exposes the Compactor on port 9199 to other gateway agents.
  • edgedelta-gateway-rollup-svc: Provides access to the Rollup agent on port 9200.
  • gw-svc: A headless service (ClusterIP: None) that targets Processor pods for stateful or identity-aware operations.

Kubernetes Permissions

RBAC policies allow it to monitor and operate across the cluster securely and efficiently. Permissions include get, list, watch on core resources like Pods, Events, Nodes, Namespaces, and Logs for observability. It includes create permissions for Events to emit telemetry and alert data, as well as access to coordination (leases) APIs, enabling leader election and distributed coordination for agents. It includes read access to the metrics.k8s.io API for pods and nodes to gather resource usage statistics. Finally, it includes read access to workload resources under apps and batch API groups — including Deployments, DaemonSets, StatefulSets, Jobs, and CronJobs — to correlate metrics and logs with running workloads.

These permissions allow the gateway to function autonomously and securely without manual intervention, providing observability, diagnostics, and telemetry across the Kubernetes environment.

IAM and Secure API Access

The use of a dedicated ServiceAccount (edgedelta-gateway) scoped via a ClusterRoleBinding ensures secure interactions with the Kubernetes API. This is a key aspect of Kubernetes-native Identity and Access Management (IAM), enabling each agent to operate with the principle of least privilege. Each component receives credentials automatically via the Kubernetes token projection mechanism, eliminating the need for embedded static secrets.

In cloud environments, this setup can be extended with cloud IAM integrations to support features like IRSA (IAM Roles for Service Accounts) in AWS, enabling access to cloud-native resources (e.g., object stores, secrets managers, telemetry endpoints) directly from Kubernetes workloads.