Install the Edge Delta Agent with Helm
9 minute read
These instructions are specific to the Enterprise Plan. To install the Free Edition using Helm, see the instructions here.
The Edge Delta Agent can be installed in a Kubernetes environment using Helm. It is installed as a DaemonSet - a pod on every node. It analyses logs from each pod in each node as well as metrics from the cluster, and streams them to the configured destinations.
Install with Default Settings
You can install Edge Delta using Helm without changing any default settings. You first create an Edge Delta agent configuration for Helm if one doesn’t exist. Then you deploy the agent by downloading the Edge Delta Helm chart and installing it, passing in a secret key to your Edge Delta web app configuration.
Create an Edge Delta Agent Configuration
Select the Helm template option while following these steps:
- In the Edge Delta App, click Data Pipeline, and then click Agent Configs.
- Click Create Configuration.
- Select the appropriate template.
- Specify a tag to identify the agent and environment.
- Click Create Configuration.
- A new default agent is created. Review the agent configuration in Visual Pipelines.
When you return to the Agent Configs page, the agent table is updated with a new agent configuration with the agent tag you specified.
Deploy an Edge Delta Agent
- In the table, locate the newly created configuration, click the corresponding vertical green ellipses, and then click Deploy Instructions.
- Select Helm.
- In the window that appears, follow the on-screen instructions.
These onscreen instructions step you through getting an updated Edge Delta Helm chart and installing the chart with two extra parameters:
- to create an edgedelta namespace, and
- to specify an API key that the web app generates and automatically adds to the instructions.
The following helm command option shows an example key 123456789.
--set secretApiKey.value=123456789
Install with Custom Settings
As with other Helm charts, there are a few ways to pass in optional or customized configuration parameters. You can download the chart and edit the values.yaml file, you can create a yaml file and pass in the whole file as a child configuration, or you can pass in specific parameters.
Creating a copy of the chart and editing the values file will disconnect your chart from the Edge Delta repo so it won’t benefit from upgrades published there. If you update your local chart, you will have to re-configure your local values.yaml file before upgrading your customized release. Passing in a child values file or individual parameters is recommended for customizations.
Individual parameters passed in with –set take precedence over passed in values files, which in turn take precedence over the default values file. For example if all three are used and they all contain a parameter but with different values, the value passed in during installation with –set will be used.
Get the Agent Configuration Key
Each custom settings approach requires you to pass in the key generated by the Edge Delta web app for your agent.
- In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Configs.
- Click Copy in the Key column of an existing configuration, or create a new configuration and copy that key.
- Use the key when installing the Helm chart by passing it in with –set, or by including it in your organization’s key store.
Secret keys should not be saved in plain text. A key store should be used in line with your organization’s secret management policy.
Pass in a Child Values File
You can pass in a child values.yaml file into the default chart during the installation. The values in this child manifest will override the values in the default chart (unless individual parameters are also passed in with –set, then they will be used). The parts of the default values file that aren’t contradicted by the passed in file will still apply - your custom values file doesn’t need to contain the entire default file, only the parts that have changed.
- Download and update the Edge Delta Helm repo
helm repo add edgedelta https://edgedelta.github.io/charts
helm repo update
- Create a yaml file and populate it with your customizations. See Values Variables for information on the changes that can be made.
- Get the Edge Delta Agent configuration’s secret key.
- Navigate to the folder where you saved your yaml file.
- Install the chart, while passing in the override file with the –values flag. You should pass in the secret key copied in the previous step with –set. It is also best practice to pass in and create a namespace. The following example shows the command passing in a file named myvalues.yaml with an example key 123456789. It will install in the edgedelta namespace, which it will create:
helm install edgedelta --values myvalues.yaml edgedelta/edgedelta --set secretApiKey.value=123456789 -n edgedelta --create-namespace`
See the Tutorial in the Prometheus Integration page for an example of using a Helm values file.
Pass in Individual Parameters
If you only need to customize a few parameters, you can use a –set flag to pass in individual parameters during the installation. Bear in mind these parameters need to be passed in every time you install or upgrade the release.
- Download and update the Edge Delta Helm repo
helm repo add edgedelta https://edgedelta.github.io/charts
helm repo update
- Get the Edge Delta Agent configuration’s secret key.
- Install the chart, while passing in the secret key copied in the previous step with –set. It is also best practice to pass in and create a namespace. The following example shows the command passing in an example key 123456789. In addition, it will install in the edgedelta namespace, which it will create.
helm install edgedelta edgedelta/edgedelta --set secretApiKey.value=123456789 -n edgedelta --create-namespace`
Edit the Chart Values File
You can download the default values file and make changes to your copy. Then you install the application using your local version.
- Download the default chart’s values.yaml file.
- Make changes to the values.yaml file. See Values Variables for information on the changes that can be made.
- Get the Edge Delta Agent configuration’s secret key.
- Install the customized helm chart with the path to the unarchived files and pass in the secret key copied in the previous step. It is also best practice to pass in and create a namespace. The following example shows the command but with an example key 123456789:
helm install edgedelta path/to/values --set secretApiKey.value=123456789 -n edgedelta --create-namespace`
Values Variables
The following table lists the values variables that you can customize:
Variable | Description | Examples |
---|---|---|
aggregatorProps.enabled | Configures whether the aggregator agent is added to the deployment. Agents are deployed as daemonset. If enabled, an extra aggregator agent is deployed to collect and aggregate metrics. | false |
aggregatorProps.port | The port that the agents will use to communicate with the aggregator. | 9191 |
aggregatorProps.serviceDNSSuffix | Used to construct endpoint for aggregator agent service so that processor agents can communicate with aggregator. Default value is “svc.cluster.local”. Final endpoint looks like this: http://ed-aggregator-svc.my-namespace.svc.cluster.local:9191 | svc.cluster.local |
aggregatorProps.storageClassName | The aggregator agent needs a small amount of disk space to persist some internal source related information to be able to work smoothly after restarts and upgrades. If your k8s cluster has specific storage classes you can set the storageClassName accordingly. | slow |
apiKey | A plaintext key used to access the Edge Delta Agent configuration (not recommended for production). Use either apiKey or secretApiKey.value, not both, to provide an API Key to Agent. | 1a2b3c4d5e6f7g8h9i |
archiverProps.DiskSize | Allocate disk space for the archiver agent to persist data. | 10Gi |
archiverProps.enabled | Enable the archiver agent. | false |
archiverProps.port | Set the archiver agent port. | 9292 |
archiverProps.storageClassName | If your k8s cluster has specific storage classes you can set the storageClassName accordingly. | slow |
dockerContainerProps | Location of the remote Docker host to use. | “/var/lib/docker/containers” |
edCustomTags | Custom tags are pipe (|) delimited key:val pairs. These are attached to all outgoing data from agent to configured destinations. | cluster:prod_us_west_2_cluster|provider:aws|region:us_west_2 |
edSkipTlsVerify | Ignore SSL/TLS certificate errors when providing a client certificate and key directly. | |
edSuppressionMode | Configure whether to check for similar issue notifications by the agent or other agents and suppress new ones if they have already been reported. | |
edWorkflowPrefixes | A colon-separated list of workflow prefixes to enable all matching workflows. By default, all other workflow names are enabled when edWorkflows and edWorkflowPrefixes are not configured. | “billing:error” |
edWorkflows | A colon-separated list of workflow names to enable matching workflows. By default, all other workflow names are enabled when edWorkflows and edWorkflowPrefixes are not configured. | “billing-workflow:error-workflow” |
httpProxy | Address to route the agent’s outbound traffic through an HTTP internal proxy. For more information on httpproxy, review this document from Golang. | “http://127.0.0.1:3128” |
httpRecorderProps.enabled | Enable httpRecorder, a frontend layer which can consume logs with both http and tcp protocols. It is deployed as a sidecar for each ED agent. It dumps the incoming logs to filesystem (persisted via PVC) and agents grab from there. | false |
httpRecorderProps.image | Specify the httpRecorder image and version tag. | gcr.io/edgedelta/httprecorder:latest |
httpRecorderProps.ingress | Configure ingress for httpRecorder if the k8s cluster already has nginx + cert-manager installed. Without ingress enabled you can directly send logs within cluster using http://ed-httprecorder-svc.{namespace}.svc.cluster.local:8080 | class: nginx |
httpRecorderProps.port | Specify the httpRecorder port. | 8080 |
httpsProxy | Address to route the agent’s outbound traffic through an HTTPS internal proxy. For more information on httpsproxy, review this document from Golang. | “https://127.0.0.1:3128” |
image | The agent’s Docker image. Optionally override the image tag, which defaults to the chart appVersion. | edgedelta/agent |
noProxy | Disables the proxy for requests that hit a specific destination. For more information on httpproxy, review this document from Golang. | “https://your-endpoint.com” |
persistingCursorProps.containerMountPath | The container mount path to keep the persisting cursor state. | /var/lib/edgedelta |
persistingCursorProps.enabled | Enables or disables the persistent cursor feature. | false |
persistingCursorProps.hostMountPath | The host mount path to keep the persisting cursor state. | /var/lib/edgedelta |
profilerPort | Specify the port to use if you install Edge Delta with a profiler to monitor CPU and memory statistics. Alternatively you can use Prometheus with its dedicated endpoint. | 6060 |
promPort | Specify the metrics endpoint port number that Prometheus can use to scrape metrics. | 8087 |
resources.limits.cpu | The maximum CPU usage limit for the agent pod. | 1000m |
resources.limits.memory | The maximum memory usage limit for the agent pod. | 512Mi |
resources.requests.cpu | The minimum requested CPU for the agent pod. | 200m |
resources.requests.memory | The minimum requested memory for the agent pod. | 256Mi |
secretApiKey.key | The reference to use to create the key part of a key/value pair stored in a Kubernetes secret when the secretApiKey.value is passed in. | ed-api-key, username, password |
secretApiKey.name | The name to use for the Kubernetes secret object when the secretApiKey.value is passed in. | ed-api-key |
secretApiKey.value | The value part of a key/value pair that is saved in a Kubernetes secret. Passing in this parameter saves it in in the secret rather than the values file, and it uses the name and key specified by secretApiName and secretApiKey. Use either apiKey or secretApiKey.value, not both, to provide an API Key to Agent. | 1a2b3c4d5e6f7g8h9i |
serviceMonitor | Enable service monitor to scrape Prometheus metrics from agents. | |
storePort | A port number to expose agent metrics storage. | 6062 |
volumes | Specify the volumes to make available to a pod. This includes the volume type such as configmap or emptyDir. | .spec.volumes: - name: cache-volume emptyDir: sizeLimit: 500Mi |
volumeMounts | Specify where to mount the volumes listed in the volumes parameter into containers. | .spec.containers[*].volumeMounts: - mountPath: /cache name: cache-volume |
Uninstall Edge Delta using Helm
To uninstall the Edge Delta release from Kubernetes, use the helm delete command, specifying the release name and the namespace:
helm delete edgedelta -n edgedelta