Install the Edge Delta Fleet with Helm

Installing the Edge Delta Fleet using Helm.

  7 minute read  

Overview

An Edge Delta fleet can be installed in a Kubernetes environment using Helm. The processing agent is installed as a DaemonSet - a set of pods on every node. It analyses logs from each pod in each node as well as metrics from the cluster, and streams them to the configured destinations.

Install with Default Settings

You can install Edge Delta using Helm without changing any default settings.

Install an Edge Delta Fleet

Select the Helm template option while following these steps:

  1. Click Pipelines.
  2. Click New Fleet.
  3. Select Edge Fleet.
  4. Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
  5. Click Continue.
  6. Select the appropriate template and click Continue.
  7. Specify a name to identify the Fleet.
  8. Click Generate Config.
  9. Execute the installation commands, they include the unique ID for the Fleet.
  10. Expand the namespaces and select the input sources you want to monitor.
  11. Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
  12. Click Continue.
  13. Click View Dashboard.

Install with Custom Settings

As with other Helm charts, there are a few ways to pass in optional or customized configuration parameters. You can download the chart and edit the values.yaml file, you can create a YAML file and pass in the whole file as a child configuration, or you can pass in specific parameters.

Creating a copy of the chart and editing the values file will disconnect your chart from the Edge Delta repo so it won’t benefit from upgrades published there. If you update your local chart, you will have to re-configure your local values.yaml file before upgrading your customized release. Passing in a child values file or individual parameters is recommended for customizations.

Individual parameters passed in with –set take precedence over passed in values files, which in turn take precedence over the default values file. For example if all three are used and they all contain a parameter but with different values, the value passed in during installation with –set will be used.

Get the Pipeline ID

Each custom settings approach requires you to pass in the Pipeline ID generated by the Edge Delta web app for your Fleet.

  1. Click Pipelines.
  2. Select the Fleet.
  3. Copy its Pipeline ID.
  4. Use the Pipeline ID when installing the Helm chart by passing it in with --set, or by including it in your organization’s key store.

Secret keys should not be saved in plain text. A key store should be used in line with your organization’s secret management policy.

Pass in a Child Values File

You can pass in a child values.yaml file into the default chart during the installation. The values in this child manifest will override the values in the default chart (unless individual parameters are also passed in with –set, then they will be used). The parts of the default values file that aren’t contradicted by the passed in file will still apply - your custom values file doesn’t need to contain the entire default file, only the parts that have changed.

  1. Download and update the Edge Delta Helm repo
helm repo add edgedelta https://edgedelta.github.io/charts

helm repo update
  1. Create a YAML file and populate it with your customizations. See Values Variables for information on the changes that can be made.
  2. Navigate to the folder where you saved your YAML file.
  3. Install the chart, while passing in the override file with the --values flag. You should pass in the secret key copied in the previous step with --set. It is also best practice to pass in and create a namespace. The following example shows the command passing in a file named myvalues.yaml with an example key 123456789. It will install in the edgedelta namespace, which it will create:
helm install edgedelta --values myvalues.yaml edgedelta/edgedelta --set secretApiKey.value=123456789 -n edgedelta --create-namespace`

See the Tutorial in the Prometheus Integration page for an example of using a Helm values file.

Pass in Individual Parameters

If you only need to customize a few parameters, you can use a --set flag to pass in individual parameters during the installation. Bear in mind these parameters need to be passed in every time you install or upgrade the release.

  1. Download and update the Edge Delta Helm repo
helm repo add edgedelta https://edgedelta.github.io/charts

helm repo update
  1. Install the chart, while passing in the Pipeline ID copied in the previous step with --set. It is also best practice to pass in and create a namespace. The following example shows the command passing in an example key 123456789. In addition, it will install in the edgedelta namespace, which it will create.
helm install edgedelta edgedelta/edgedelta --set secretApiKey.value=123456789 -n edgedelta --create-namespace`

Edit the Chart Values File

You can download the default values file and make changes to your copy. Then you install the application using your local version.

  1. Download the default chart’s values.yaml file.
  2. Make changes to the values.yaml file. See Values Variables for information on the changes that can be made.
  3. Install the customized helm chart with the path to the un-archived files and pass in the Pipeline ID copied in the previous step. It is also best practice to pass in and create a namespace. The following example shows the command but with an example Pipeline ID 123456789:
helm install edgedelta path/to/values --set secretApiKey.value=123456789 -n edgedelta --create-namespace`

Installing Edge Delta as a Deployment

The recommended installation method for Edge Delta is as a DaemonSet for comprehensive Kubernetes support. However, you may prefer to deploy Edge Delta as a Deployment. The Deployment option is designed for central processing, receiving data from multiple upstream sources and performing all pipeline operations such transformation, aggregation and filtering.

We highly recommend using the DaemonSet architecture and pushing processing to the edge when possible since it is more efficient, easier to manage, and provides the full feature-set of the Edge Delta agent.

To deploy Edge Delta as a Deployment, set the deployment.kind variable to Deployment (add --set deployment.kind=Deployment to the installation command or set this in the values.yaml file). To specify the number of replicas, use the deployment.replicas variable (an integer value without quotes).

NOTE: It is not possible to use helm upgrade to change the deployment.kind from a DaemonSet to a Deployment or vice versa. If you are moving from one deployment.kind to another, you must first completely remove the existing Edge Delta agent installation using helm delete edgedelta -n edgedelta before running a clean Helm install with the new deployment type.

Install with a fixed replica count

Choose this option when you know exact amount of pods needed to handle the incoming data processing need.

helm install edgedelta edgedelta/edgedelta -n edgedelta --create-namespace \
    --set secretApiKey.value=123456789 \
    --set deployment.kind=Deployment \
    --set deployment.replicas=2

Install with autoscaling enabled

This is the most recommended option to enable auto scaling based on the workload. By default autoscaling happens via CPU utilization threshold.

helm install edgedelta edgedelta/edgedelta -n edgedelta --create-namespace \
    --set secretApiKey.value=123456789 \
    --set deployment.kind=Deployment \
    --set deployment.autoscaling.enabled=true \
    --set deployment.autoscaling.minReplica=2 \
    --set deployment.autoscaling.maxReplica=20

Install with a load balancer for the incoming data

When pipeline listens on a port, deployment agents should be placed behind a load balancer. If multiple ports are used in the pipeline, please add more ports to values.yaml via ports[1], ports[2], etc.

helm upgrade edgedelta ./edgedelta -n edgedelta --create-namespace \
    --set secretApiKey.value=123456789 \
    --set deployment.kind=Deployment \
    --set deployment.autoscaling.enabled=true \
    --set deployment.autoscaling.minReplica=2 \
    --set deployment.autoscaling.maxReplica=20
    --set ports[0].name=processor-port \
    --set ports[0].protocol=TCP \
    --set ports[0].port=4547 \
    --set ports[0].exposeInHost=false \
    --set pushService.type=LoadBalancer

Select a load balancer that you are confident operating in a highly available mode. We recommend using a managed load balancer such as AWS ALB, Azure LB, or Google Cloud Network Load Balancer. These managed load balancers are highly available and seamlessly integrate with your cloud environment.

For more information on Kubernetes load balancers, please refer to the Kubernetes Load Balancer Documentation.

Restrictions

Some of features for DaemonSets are not available for Deployments. These are:

  1. Collecting Kubernetes logs (indirect collection is possible via other collectors, including Edge Delta, which can push data to Deployment installations)
  2. Collecting Kubernetes metrics
  3. Collecting Kubernetes traffic metrics

These are not possible because Edge Delta needs one-to-one correspondence with each node in the cluster to have full-fledged data without any duplication. Topology spread constraints, for example, are hard to configure with a Deployment. Therefore, these features as not fully supported yet.

Uninstall Edge Delta using Helm

To uninstall the Edge Delta release from Kubernetes, use the helm delete command, specifying the release name and the namespace:

helm delete edgedelta -n edgedelta

See Also:

Upgrade

GitOps Principles Deployment of Edge Delta