Example Agent ConfigMap for ArgoCD.
12 minute read
You can install and configure the Edge Delta Kubernetes agent using ConfigMaps in a GitOps workflow and a tool such as ArgoCD. ArgoCD is one of several applications that can monitor a target state expressed declaratively in a Git repo and pull changes into the deployed environment when the repo is updated. This approach leverages common code repository management workflows that are likely already in place in your organization. Using a common workflow aligns management of the Edge Delta agent with existing identity and access management, secret management, and version control. It also enables update strategies such as Blue-Green and Canary updates.
Kubernetes application definitions, application configurations and environment configurations are stored in Git repositories that are protected with appropriate information security controls. ArgoCD polls the repo regularly for changes. In addition, a web-hook can publish a change event to will trigger ArgoCD to poll the repo. ArgoCD performs automatic deployments based on the contents of the repository, which can be tracked by branch, tag, or version to match your organization’s deployment practices. For each component that needs to be deployed, ArgoCD is configured with an ArgoCD application definition. This definition in turn specifies the location of the component’s manifest.
The following conditions are required to install and manage the Edge Delta Kubernetes agent using ArgoCD:
The high-level process to install and manage the Edge Delta Kubernetes agent using ArgoCD is as follows. This process uses Bitnami kubeseal for secrets management. The specific details depend on your architecture.
The following example illustrates how to deploy the Edge Delta agent and manage its configuration using GitHub and ArgoCD. The agent is defined with a Kubernetes application definition and the configuration is deployed as a ConfigMap. The API key is also stored in the GitHub repository, but it is encrypted using Bitnami kubeseal along with the Bitnami Sealed-Secret controller. When it is deployed, the controller will decrypt it as a secret. Finally, there is a generic log generating workload called flog. ArgoCD is configured with a private SSH key to enable access to the GitHub repo. The app of apps pattern is used to deploy all the ArgoCD application definitions stored in the ArgoCD application definition folder.
The GitHub repo will be structured as follows. Indented bullets indicate the subfolder structure:
The GitHub repo contains an argo_apps folder of ArgoCD application definitions for each component:
In addition, the GitHub repo contains a manifests folder with subfolder folders for each application. Within each of those folders there is a manifest.
Create and name an Edge Delta agent configuration in the Edge Delta web app. This will create an agent with an agent ID, an API key, and an agent configuration. The agent configuration in the Edge Delta web app will be ignored because the configuration will be supplied locally as a ConfigMap to the cluster. However, you can use the interface to test your configuration. Use the Helm template option while following these steps:
The agent table is updated with a new agent configuration with the agent tag you specified.
Note the key in the key column of the new agent. To see the Organization ID, click Management - My Organization
Create sealed secret objects using Bitnami’s kubeseal and the sealed secrets controller, which must be already installed on the cluster. The sealed secret will be decrypted by the controller when it is applied.
The same instance of the secrets controller that was used to create the sealed secrets must be used to decrypt them because a new encryption key is generated and shared with kubeseal on the local machine whenever the controller is installed, or when the pod restarts. The pod containing the sealed secret therefore should be managed using rolling updates or a similar non-destructive approach. You can also save the encryption key used to encrypt your secrets in case you need to re-deploy the controller. See the kubeseal readme for installation and configuration instructions. For brevity, this scenario does not include the secrets controller in the application definition to be controlled by ArgoCD but in a production environment it would be.
The following command creates a sealed secret for the API key 123456789. It pipes the output of a dry run kubectl create secret command to a sealed secret file. You specify the namespace, the secret name, in this case ed-api-key, and the output filename.
The namespace configured here for the secret must be the same as the namespace used to apply the sealed secret in the ArgoCD application definition in step 6 - in both instances in this scenario it is
edgedelta. In addition, the name of the secret should match the Edge Delta agent manifest defined in step 5, by default this is ed-api-key.
kubectl --namespace edgedelta \ create secret \ generic ed-api-key \ --dry-run=client \ --from-literal ed-api-key="123456789" \ --output yaml \ | kubeseal \ | tee apisecret.yaml
See here for an example of the file that is generated.
The following command creates the organization ID secret, which must be named ed-org-id. Replace 987654321 with your ID.
kubectl --namespace edgedelta \ create secret \ generic ed-org-id \ --dry-run=client \ --from-literal ed-org-id="987654321" \ --output yaml \ | kubeseal \ | tee orgid.yaml
You can copy the default agent configuration from the Edge Delta web app and use it as a basis for creating your custom local configuration. You can also use the Edge Delta web app to test your configuration using specific log files. Specify an agent tag to identify your cluster in the Edge Delta web app. You may want to add the ArgoCD pod to included inputs list:
inputs: kubernetes: - labels: kubernetes_logs include: - pod=argocd*
See Agent Configuration for more details about configuring the agent.
After creating a configuration, you convert it into a ConfigMap, specify a name and namespace, and specify the ConfigMap filename that will be mounted. Note the output file is named config.yml to align with the expected filename specified in the Edge Delta agent environment variable commands configured in step 5.
apiVersion: v1 kind: ConfigMap metadata: name: edgedelta-agent-config namespace: edgedelta data: config.yml: | workflows:
If the ConfigMap data field does not parse correctly, you can pass your raw configuration yaml file into kubectl to automatically create a well formatted ConfigMap file. You specify the ConfigMap name, the namespace, and the source file that contains the raw configuration, and the destination file:
kubectl create configmap edgedelta-agent-config --from-file=agent-config.yml -n edgedelta --dry-run -o yaml > config.yml
See here for an example of a ConfigMap containing an agent configuration.
You can download the latest manifest from the Edge Delta repository and use it as a basis for your customized installation. The location is also listed in the Kubernetes agent installation instructions on the Edge Delta web app, in the
kubeclt apply -f step:
These changes must be made to the file to ensure it will work with the ConfigMap.
command: - /edgedelta/edgedelta - -c - /config/config.yml
The configuration file name is config.yml to align the ConfigMap configured in step 4.
Ensure that the secretKeyRef entries match the secret names defined in step 2:
See here for an example of a customized Edge Delta manifest.
The ArgoCD application definitions are components that configure ArgoCD for each application that needs to be installed. The metadata names the ArgoCD application and namespace, which must be argocd. The spec section points to the repo and the folder name where the manifest is stored, defines the cluster location and target application’s namespace, and configures some optional ArgoCD synchronization policies. Target manifests - those refered to with repoURL and path - can be YAML, JSON, or Jsonnet manifests, Kustomize applications, Helm charts, or other custom config management plugin files. Each manifest must be stored in a unique folder because ArgoCD locates the manifest by its folder using the path variable. Use of public repositories is not recommended for production.
The Edge Delta agent ArgoCD application definition is as follows:
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: edagent namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: 'firstname.lastname@example.org:exampleuser/examplerepo.git' path: manifests/edge_delta_app destination: server: 'https://kubernetes.default.svc' namespace: edgedelta syncPolicy: syncOptions: - CreateNamespace=true: automated: prune: true selfHeal: true
This example directs ArgoCD to install the manifest located in the manifests/edge_delta_app. ArgoCD installs it in the
edgedelta namespace, which will be created if it doesn’t exist. The syncPolicy section indicates that when the application is running, ArgoCD will synchronize automatically with the repository to check whether the target state has changed. The default sync interval is 3 minutes. If the repo has changed, the prune parameter requires ArgoCD to replace the changed resource rather than applying the changes to the existing resource. The selfHeal parameter means that if changes are made directly on the cluster to the agent, for example using kubectl apply, the target state repository will take precedence and the target state will be re-applied during the next sync, undoing the kubectl change. The finalizer ensures that the application will be deleted if the argocd application definition is uninstalled from the cluster.
If the application is a helm chart, the source section is configured as follows. Note the repoURL refers to the helm chart library and the chart specifies which chart to install, rather than a path containing the manifest. In addition, the version number is required:
source: repoURL: 'https://bitnami-labs.github.io/sealed-secrets' targetRevision: 2.7.3 helm: parameters: - name: fullnameOverride value: sealed-secrets-controller chart: sealed-secrets destination: server: 'https://kubernetes.default.svc' namespace: kube-system syncPolicy: syncOptions: - CreateNamespace=true automated: prune: false selfHeal: true
The Quick Start section provides details on using ArgoCD to install the Edge Delta Helm chart from a private location without any agent configuration.
Note that prune is set to false. This preserves the controller’s encryption key from being reset by a pod restart.
In this example scenario, the following ArgoCD application definitions are created.
The namespace configured in the ArgoCD application definition for the secret must be the same as the namespace configured for the sealed secret in step 2 - in both instances in this scenario it is
See here for examples of these ArgoCD definitions.
Use the normal Git workflow in your organization to push the files you created to the repository. Once the app of apps is deployed in Step 8 and ArgoCD is running, any further changes to the repo will result in changes to the environment managed by ArgoCD.
In this example scenario, the file structure is as follows:
Use the command line on the host to create one more ArgoCD application definition for the app of apps. This definition is applied in the argocd namespace. It applies all the definitions saved in the argo_apps folder.
argocd app create appofapps --repo email@example.com:exampleuser/examplerepo.git --path argo_apps --dest-server https://kubernetes.default.svc --dest-namespace argocd
To initiate the first synchronisation with the target manifest, synchronise the app of apps:
argocd app sync appofapps
Add the secret controller definition to the app of apps. Ensure that it does not re-install the component as it will reset the encryption key used to create the sealed secret.
A best practice for a production environment would be to host the helm chart and application image in a private repository.
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: sealed-secrets namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: 'https://bitnami-labs.github.io/sealed-secrets' targetRevision: 2.7.3 helm: parameters: - name: fullnameOverride value: sealed-secrets-controller chart: sealed-secrets destination: server: 'https://kubernetes.default.svc' namespace: kube-system
Create a webhook to automatically synchronize ArgoCD with the target state repository whenever a change is merged to the repo. See the ArgoCD docs for more information.
To update the agent configuration, make changes to the ConfigMap and merge the changes into the git repository. ArgoCD will synchronise at the next poll - by default 3 minutes. Alternatively you can either manually synchronise the app of apps application, or you can configure a webhook in GitHub to trigger synchronization with ArgoCD.
Example Agent ConfigMap for ArgoCD.
Example Agent Mannifest for ArgoCD.
Example Agent Secret for ArgoCD.
Example application definitions for ArgoCD.