ArgoCD

Installing Edge Delta with ArgoCD.

  12 minute read  

You can install and configure the Edge Delta Kubernetes agent using ConfigMaps in a GitOps workflow and a tool such as ArgoCD. ArgoCD is one of several applications that can monitor a target state expressed declaratively in a Git repo and pull changes into the deployed environment when the repo is updated. This approach leverages common code repository management workflows that are likely already in place in your organization. Using a common workflow aligns management of the Edge Delta agent with existing identity and access management, secret management, and version control. It also enables update strategies such as Blue-Green and Canary updates.

High Level Architecture

Kubernetes application definitions, application configurations and environment configurations are stored in Git repositories that are protected with appropriate information security controls. ArgoCD polls the repo regularly for changes. In addition, a web-hook can publish a change event to will trigger ArgoCD to poll the repo. ArgoCD performs automatic deployments based on the contents of the repository, which can be tracked by branch, tag, or version to match your organization’s deployment practices. For each component that needs to be deployed, ArgoCD is configured with an ArgoCD application definition. This definition in turn specifies the location of the component’s manifest.

Process Overview

Prerequisites

The following conditions are required to install and manage the Edge Delta Kubernetes agent using ArgoCD:

  1. A private Git repository with appropriate access control measures in place as per your organization’s policies. For example, GitHub or GitLab with SSH keys configured for access control.
  2. A secrets management solution. For example, Bitnami’s kubeseal or Hashicorp Vault.
  3. An existing Kubernetes cluster.
  4. ArgoCD installed on the cluster and configured with access to the private repo, for example, by providing it with an appropriate private SSH key as per your organization’s information security policies.

Process Steps

The high-level process to install and manage the Edge Delta Kubernetes agent using ArgoCD is as follows. This process uses Bitnami kubeseal for secrets management. The specific details depend on your architecture.

  1. Copy the Edge Delta API key and organization ID.
  2. Create API key and organization ID encrypted secrets.
  3. Define an Edge Delta agent configuration.
  4. Convert the agent configuration into a ConfigMap.
  5. Define the Edge Delta agent Kubernetes manifest.
  6. Define ArgoCD application definitions.
  7. Merge the manifests and ArgoCD application definitions to the Git repository.
  8. Apply all the application definitions.

Example Implementation

Scenario Description

The following example illustrates how to deploy the Edge Delta agent and manage its configuration using GitHub and ArgoCD. The agent is defined with a Kubernetes application definition and the configuration is deployed as a ConfigMap. The API key is also stored in the GitHub repository, but it is encrypted using Bitnami kubeseal along with the Bitnami Sealed-Secret controller. When it is deployed, the controller will decrypt it as a secret. Finally, there is a generic log generating workload called flog. ArgoCD is configured with a private SSH key to enable access to the GitHub repo. The app of apps pattern is used to deploy all the ArgoCD application definitions stored in the ArgoCD application definition folder.

The GitHub repo will be structured as follows. Indented bullets indicate the subfolder structure:

The GitHub repo contains an argo_apps folder of ArgoCD application definitions for each component:

  • argo_apps
    • edgedelta_app.yml
    • edgedelta_agent_config_app.yml
    • api_secret_app.yml
    • flog_app.yml

In addition, the GitHub repo contains a manifests folder with subfolder folders for each application. Within each of those folders there is a manifest.

  • manifests
    • edge_delta_app
      • edgedelta.yml
    • edge_delta_config
      • config.yml
    • api
      • apisecret.yml
    • workload
      • flog.yml

1. Copy the Edge Delta API key and Organization ID

Create and name an Edge Delta agent configuration in the Edge Delta web app. This will create an agent with an agent ID, an API key, and an agent configuration. The agent configuration in the Edge Delta web app will be ignored because the configuration will be supplied locally as a ConfigMap to the cluster. However, you can use the interface to test your configuration. Use the Helm template option while following these steps:

  1. In the Edge Delta App, click Data Pipeline, and then click Agent Configs.
  2. Click Create Configuration.
  3. Select the appropriate template.
  4. Specify a tag to identify the agent and environment.
  5. Click Create Configuration.
  6. Optionally, modify and test the agent configuration then click Save Edits or Done.

The agent table is updated with a new agent configuration with the agent tag you specified.

Note the key in the key column of the new agent. To see the Organization ID, click Management - My Organization

2. Create API key and organization ID encrypted secrets

Create sealed secret objects using Bitnami’s kubeseal and the sealed secrets controller, which must be already installed on the cluster. The sealed secret will be decrypted by the controller when it is applied.

The same instance of the secrets controller that was used to create the sealed secrets must be used to decrypt them because a new encryption key is generated and shared with kubeseal on the local machine whenever the controller is installed, or when the pod restarts. The pod containing the sealed secret therefore should be managed using rolling updates or a similar non-destructive approach. You can also save the encryption key used to encrypt your secrets in case you need to re-deploy the controller. See the kubeseal readme for installation and configuration instructions. For brevity, this scenario does not include the secrets controller in the application definition to be controlled by ArgoCD but in a production environment it would be.

The following command creates a sealed secret for the API key 123456789. It pipes the output of a dry run kubectl create secret command to a sealed secret file. You specify the namespace, the secret name, in this case ed-api-key, and the output filename.

The namespace configured here for the secret must be the same as the namespace used to apply the sealed secret in the ArgoCD application definition in step 6 - in both instances in this scenario it is edgedelta. In addition, the name of the secret should match the Edge Delta agent manifest defined in step 5, by default this is ed-api-key.

kubectl --namespace edgedelta \
    create secret \
    generic ed-api-key \
    --dry-run=client \
    --from-literal ed-api-key="123456789" \
    --output yaml \
    | kubeseal \
    | tee apisecret.yaml

See here for an example of the file that is generated.

The following command creates the organization ID secret, which must be named ed-org-id. Replace 987654321 with your ID.

kubectl --namespace edgedelta \
    create secret \
    generic ed-org-id \
    --dry-run=client \
    --from-literal ed-org-id="987654321" \
    --output yaml \
    | kubeseal \
    | tee orgid.yaml

3. Define an Edge Delta Agent Configuration

You can copy the default agent configuration from the Edge Delta web app and use it as a basis for creating your custom local configuration. You can also use the Edge Delta web app to test your configuration using specific log files. Specify an agent tag to identify your cluster in the Edge Delta web app. You may want to add the ArgoCD pod to included inputs list:

inputs:
  kubernetes:
    - labels: kubernetes_logs
      include:
      - pod=argocd*

See Agent Configuration for more details about configuring the agent.

4. Convert the agent configuration into a ConfigMap

After creating a configuration, you convert it into a ConfigMap, specify a name and namespace, and specify the ConfigMap filename that will be mounted. Note the output file is named config.yml to align with the expected filename specified in the Edge Delta agent environment variable commands configured in step 5.

apiVersion: v1
kind: ConfigMap
metadata:
  name: edgedelta-agent-config
  namespace: edgedelta
data:
  config.yml: |
    workflows:

If the ConfigMap data field does not parse correctly, you can pass your raw configuration yaml file into kubectl to automatically create a well formatted ConfigMap file. You specify the ConfigMap name, the namespace, and the source file that contains the raw configuration, and the destination file:

kubectl create configmap edgedelta-agent-config --from-file=agent-config.yml -n edgedelta --dry-run -o yaml  > config.yml

See here for an example of a ConfigMap containing an agent configuration.

5. Define the Edge Delta Agent Kubernetes Manifest

You can download the latest manifest from the Edge Delta repository and use it as a basis for your customized installation. The location is also listed in the Kubernetes agent installation instructions on the Edge Delta web app, in the kubeclt apply -f step:

These changes must be made to the file to ensure it will work with the ConfigMap.

  1. Configure the agent to not download a configuration from the Edge Delta back end by adding the ED_SKIP_CONF_DOWNLOAD environment variable value of 1.
  2. Add environment variable commands to specify the location of the mounted configuration:
  command:
    - /edgedelta/edgedelta
    - -c
    - /config/config.yml

The configuration file name is config.yml to align the ConfigMap configured in step 4.

  1. Add a volume with a ConfigMap called edgedelta-agent-config, which is the name of the ConfigMap, then add a volumeMount with the mountPath /config/. This adds the ConfigMap data to the directory specified in mountPath. This is required to ensure that updates made via the git repo to the ConfigMap will be automatically applied without the need to restart the pod.
  2. Create the ED_ORG_ID environmental variable with the secretKeyRef if it is missing.

Ensure that the secretKeyRef entries match the secret names defined in step 2: ed-api-key and ed-org-id.

See here for an example of a customized Edge Delta manifest.

6. Define ArgoCD application definitions

The ArgoCD application definitions are components that configure ArgoCD for each application that needs to be installed. The metadata names the ArgoCD application and namespace, which must be argocd. The spec section points to the repo and the folder name where the manifest is stored, defines the cluster location and target application’s namespace, and configures some optional ArgoCD synchronization policies. Target manifests - those refered to with repoURL and path - can be YAML, JSON, or Jsonnet manifests, Kustomize applications, Helm charts, or other custom config management plugin files. Each manifest must be stored in a unique folder because ArgoCD locates the manifest by its folder using the path variable. Use of public repositories is not recommended for production.

The Edge Delta agent ArgoCD application definition is as follows:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: edagent
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'git@github.com:exampleuser/examplerepo.git'
    path: manifests/edge_delta_app
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: edgedelta
  syncPolicy:
    syncOptions:
      - CreateNamespace=true:
    automated:
      prune: true
      selfHeal: true

This example directs ArgoCD to install the manifest located in the manifests/edge_delta_app. ArgoCD installs it in the edgedelta namespace, which will be created if it doesn’t exist. The syncPolicy section indicates that when the application is running, ArgoCD will synchronize automatically with the repository to check whether the target state has changed. The default sync interval is 3 minutes. If the repo has changed, the prune parameter requires ArgoCD to replace the changed resource rather than applying the changes to the existing resource. The selfHeal parameter means that if changes are made directly on the cluster to the agent, for example using kubectl apply, the target state repository will take precedence and the target state will be re-applied during the next sync, undoing the kubectl change. The finalizer ensures that the application will be deleted if the argocd application definition is uninstalled from the cluster.

If the application is a helm chart, the source section is configured as follows. Note the repoURL refers to the helm chart library and the chart specifies which chart to install, rather than a path containing the manifest. In addition, the version number is required:

  source:
    repoURL: 'https://bitnami-labs.github.io/sealed-secrets'
    targetRevision: 2.7.3
    helm:
      parameters:
        - name: fullnameOverride
          value: sealed-secrets-controller
    chart: sealed-secrets
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: kube-system
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: false
      selfHeal: true

The Quick Start section provides details on using ArgoCD to install the Edge Delta Helm chart from a private location without any agent configuration.

Note that prune is set to false. This preserves the controller’s encryption key from being reset by a pod restart.

In this example scenario, the following ArgoCD application definitions are created.

  • edgedelta_app.yml
  • edgedelta_agent_config_app.yml
  • api_secret_app.yml
  • flog_app.yml

The namespace configured in the ArgoCD application definition for the secret must be the same as the namespace configured for the sealed secret in step 2 - in both instances in this scenario it is edgedelta.

See here for examples of these ArgoCD definitions.

7. Merge the Manifests and ArgoCD Application Definitions to the git Repository

Use the normal Git workflow in your organization to push the files you created to the repository. Once the app of apps is deployed in Step 8 and ArgoCD is running, any further changes to the repo will result in changes to the environment managed by ArgoCD.

In this example scenario, the file structure is as follows:

  • argo_apps
    • edgedelta_app.yml
    • edgedelta_agent_config_app.yml
    • api_secret_app.yml
    • flog_app.yml
  • manifests
    • edge_delta_app
      • edgedelta.yml
    • edge_delta_config
      • config.yml
    • api
      • apisecret.yml
    • workload
      • flog.yml Note the structure reflects the path parameters defined in the ArgoCD application definitions.

8. Apply an app of apps ArgoCD application definition

Use the command line on the host to create one more ArgoCD application definition for the app of apps. This definition is applied in the argocd namespace. It applies all the definitions saved in the argo_apps folder.

argocd app create appofapps --repo git@github.com:exampleuser/examplerepo.git --path argo_apps --dest-server https://kubernetes.default.svc --dest-namespace argocd

To initiate the first synchronisation with the target manifest, synchronise the app of apps:

argocd app sync appofapps

Optional Steps

Secret Controller Application Definition

Add the secret controller definition to the app of apps. Ensure that it does not re-install the component as it will reset the encryption key used to create the sealed secret.

A best practice for a production environment would be to host the helm chart and application image in a private repository.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'https://bitnami-labs.github.io/sealed-secrets'
    targetRevision: 2.7.3
    helm:
      parameters:
        - name: fullnameOverride
          value: sealed-secrets-controller
    chart: sealed-secrets
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: kube-system

Webhook to Trigger Synchonization

Create a webhook to automatically synchronize ArgoCD with the target state repository whenever a change is merged to the repo. See the ArgoCD docs for more information.

Updating the Agent Configuration

To update the agent configuration, make changes to the ConfigMap and merge the changes into the git repository. ArgoCD will synchronise at the next poll - by default 3 minutes. Alternatively you can either manually synchronise the app of apps application, or you can configure a webhook in GitHub to trigger synchronization with ArgoCD.

Agent ConfigMap

Example Agent ConfigMap for ArgoCD.

Agent Manifest

Example Agent Mannifest for ArgoCD.

Agent Secret

Example Agent Secret for ArgoCD.

Application Definitions

Example application definitions for ArgoCD.