Continuous Deployment of Edge Delta

Install Edge Delta declaratively using continuous deployment.

You can install the Edge Delta Free Edition agent on a Kubernetes cluster using Continuous Deployment (CD) with GitOps principles. To do this, you configure your CD tooling such as Argo CD, Flux or Jenkins X to synchronize your cluster with the manifests on your configuration repository.

When you make updates to the configuration repository, they are automatically applied by your CD tooling to the cluster. It does this by polling the configuration regularly. If there is a mismatch between the configuration (desired state) and the cluster (current state), the configuration is pulled and applied to the cluster. This enables cluster self healing.

In addition, you can save any dependent images in your own image repository. Keeping your images and configurations isolated gives you control over changes. With this workflow, change control on the configuration repository and image repository enables you to have change control on your cluster. You can also monitor changes to images and configurations using code diffs before applying them to your image repo, configuration repos, and in turn, cluster. This also makes maintaining custom settings easier as you can isolate all your customizations from vendor updates to the default manifests.

Alternatively, for a lightweight implementation you can configure your CD tool to use the public Edge Delta chart repository and image store and simply pass in custom variables.

Install Edge Delta Declaratively

Complete the following steps. You can use the tutorial that follows as a guide.

  1. Create the API key secret
  2. Create the Organization ID secret
  3. Create the Edge Delta configMap
  4. Create the Edge Delta agent manifest
  5. Create application definitions
  6. Push changes to the configuration repo
  7. Create and execute the app-of-apps definition

Tutorial

The following tutorial will use ArgoCD to install the Edge Delta agent using public images and the Bitnami secrets manager called Sealed Secrets. See the free edition CD quickstart for an example that uses privately stored images.

Argo CD is one of several applications that can monitor a target state expressed declaratively in a Git repo and pull changes into the deployed environment when the repo is updated. This approach leverages common code repository management workflows that are likely already in place in your organization. Using a common workflow aligns management of the Edge Delta agent with existing identity and access management, secret management, and version control. It also enables update strategies such as Blue-Green and Canary updates.

High Level Architecture

Kubernetes application definitions, application configurations and environment configurations are stored in Git repositories that are protected with appropriate information security controls. While Argo CD polls the repo regularly for changes, a web-hook can publish a change event to trigger polling. Argo CD performs automatic deployments based on the contents of the repository, which can be tracked by branch, tag, or version to match your organization’s deployment practices. For each component that needs to be deployed, ArgoCD is configured with an ArgoCD application definition. This definition in turn specifies the location of the component’s manifest.

The following example illustrates how to deploy the Edge Delta agent and manage its configuration using GitHub and ArgoCD. The agent is defined with a Kubernetes application definition and the configuration is deployed as a ConfigMap. The API key is also stored in the GitHub repository, but it is encrypted using Bitnami Kubeseal along with the Bitnami Sealed-Secret controller. When it is deployed, the controller will decrypt it as a secret. Argo CD is configured with a private SSH key to enable access to the GitHub repo. The app of apps pattern is used to deploy all the ArgoCD application definitions stored in the Argo CD application definition folder.

The GitHub repo will be structured as follows. Indented bullets indicate the subfolder structure:

An argocd_apps folder contains ArgoCD application definitions for each component.

  • argocd_apps
    • api_secret_app.yml
    • edgedelta_agent_config_app.yml
    • edgedelta_app.yml
    • orgidapp.yml
    • sealed_secrets.yml

An application_manifests folder contains subfolders for each application. Within each of those folders there is a manifest. Sealed Secrets is hosted in its own public repo so there is no manifest for it in the repo.

  • application_manifests
    • apikey
      • apisecret.yml
    • appconfig
      • config.yml
    • edgedelta
      • edmanifest.yml
    • orgid
      • orgid.yml

Prerequisites:

To follow along with this tutorial there are a few prerequisites:

  • A private GitHub repo cloned on the local machine.
  • An SSH key for the repo added to the ssh-agent (in this tutorial the private key is in id_ed25519 and it does not have a passphrase).
  • An Edge Delta Account.
  • Homebrew package manager installed.
  • Docker installed.
  • Kind installed.
  • Helm installed.
  • kubectl installed.

Create a Cluster

Use kind to create a cluster.

kind create cluster 

Install kubeseal and the Secrets Controller

You need Sealed Secrets to generate the secrets. It consists of kubeseal on the local host and the secrets controller in the cluster.

brew install kubeseal
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets 

Create an Edge Delta Agent

Select the Helm template option while following these steps:

  1. In the Edge Delta App, click Data Pipeline, and then click Agent Configs.
  2. Click Create Configuration.
  3. Select the appropriate template.
  4. Specify a tag to identify the agent and environment.
  5. Click Create Configuration.
  6. A new default agent is created. Review the agent configuration in Visual Pipelines.

When you return to the Agent Configs page, the agent table is updated with a new agent configuration with the agent tag you specified.

Create a Demo Input Node

Create a demo input node and connect it to the mask_ssn node.

  • name: demo_input
  • speed: 1s
  • error_interval: 1ms
  • error_count: 20

Copy the Agent API Key, Config YAML, and Organization ID

You need to create a sealed secret using the API key. So copy the key for the agent you configured in the previous step. Also copy the agent configuration YAML and your organization ID for use in later steps.

  1. In the Edge Delta App, click Data Pipeline - Agent Configs.
  2. Open the Actions menu for the agent and select Edit to open the YAML editor.
  3. Copy the agent API key and paste it in a temporary file.
  4. Copy the configuration YAML and paste it in the temporary file for use in a later step.
  5. Click Admin - My Organization and copy your organization ID, paste it in the temporary file for use in a later step..

Create the API key Secret

  1. Open to your cloned repository, for example using VS Code.
  2. Create and navigate to the application_manifests/apikey folder.
  3. Use Kubeseal to create the sealed secret for the Edge Delta API in the application_manifests/apikey folder, replacing 123456789 with the Edge Delta agent API key you copied earlier:
kubectl --namespace edgedelta \
    create secret \
    generic ed-api-key \
    --dry-run=client \
    --from-literal ed-api-key="123456789" \
    --output yaml \
    | kubeseal \
    | tee apisecret.yml

This command creates the API secrets key using kubeseal. It saves the sealed secret as application_manifests/apikey/apisecret.yml. This path will be referenced by Argo CD configuration file that you create later.

Create the organization ID sealed secret

  1. Create and navigate to the application_manifests/orgid folder.
  2. Use Kubeseal to create the organization ID sealed secret for your Edge Delta organization ID in the orgid folder, replacing 123456789 with your ID:
kubectl --namespace edgedelta \
    create secret \
    generic ed-org-id \
    --dry-run=client \
    --from-literal ed-org-id="123456789" \
    --output yaml \
    | kubeseal \
    | tee orgid.yml

This command creates the Organization ID sealed secrets using kubeseal. It saves the sealed secret in application_manifests/orgid/orgid.yml. This path will be referenced by the Argo CD configuration file that you create later.

Create the Edge Delta ConfigMap definition

  1. Create a temporary file named config.yml in the application_manifests folder.
  2. Paste the configuration YAML contents you copied earlier into it.
  3. Create a folder in application_manifests called appconfig (but dont navigate your terminal to that folder).
  4. Navigate the terminal to the temporary config.yml file location and use this command to convert it into a configMap. The new configMap will also called config.yml but it will be saved in the appconfig folder. Call the configMap edgedelta-agent-config and specify the edgedelta namespace:
kubectl create configmap edgedelta-agent-config --from-file=config.yml -n edgedelta --dry-run=client -o yaml  > appconfig/config.yml

At the time of writing the configMap is as follows:

apiVersion: v1
data:
  config.yml: |+
    version: v3

    settings:
      tag: deployed_by_argocd
      log:
        level: debug

    links:
    - from: kubernetes_logs
      to: failure_monitoring
    - from: kubernetes_logs
      to: negative_sentiment_monitoring
    - from: kubernetes_logs
      to: pattern_analytics
    - from: kubernetes_logs
      to: error_monitoring
    - from: kubernetes_logs
      to: exception_monitoring
    - from: kubernetes_logs
      to: ed_archive
    - from: error_monitoring
      to: ed_metrics
    - from: exception_monitoring
      to: ed_metrics
    - from: failure_monitoring
      to: ed_metrics
    - from: negative_sentiment_monitoring
      to: ed_metrics
    - from: pattern_analytics
      to: ed_patterns
    - from: demo_input
      to: ed_archive
    - from: demo_input
      to: pattern_analytics
    - from: demo_input
      to: negative_sentiment_monitoring
    - from: demo_input
      to: failure_monitoring
    - from: demo_input
      to: exception_monitoring
    - from: demo_input
      to: error_monitoring

    nodes:
    - name: kubernetes_logs
      type: kubernetes_input
      include:
      - k8s_namespace=.*
      exclude:
      - k8s_namespace=kube-system
      - k8s_namespace=kube-public
      - k8s_namespace=kube-node-lease
      - k8s_pod_name=edgedelta
      - k8s_pod_name=prometheus
      - k8s_pod_name=promtail
      - k8s_pod_name=node-exporter
    - name: error_monitoring
      type: log_to_metric
      pattern: (?i)error
    - name: exception_monitoring
      type: log_to_metric
      pattern: (?i)exception
    - name: failure_monitoring
      type: log_to_metric
      pattern: (?i)fail
    - name: negative_sentiment_monitoring
      type: log_to_metric
      pattern: (?i)(exception|fail|timeout|broken|caught|denied|abort|insufficient|killed|killing|malformed|unsuccessful|outofmemory|panic|undefined)
    - name: ed_archive
      type: ed_archive_output
    - name: ed_metrics
      type: ed_metrics_output
    - name: pattern_analytics
      type: log_to_pattern
    - name: ed_patterns
      type: ed_patterns_output
    - name: demo_input
      type: demo_input
      speed: 1s
      error_interval: 1ms
      error_count: 20    

kind: ConfigMap
metadata:
  creationTimestamp: null
  name: edgedelta-agent-config
  namespace: edgedelta
  1. Delete the temporary configuration file application_manifests/config.yml.

Do not delete application_manifests/appconfig/config.yml.

Back Up the Encryption Key

Back up the Sealed-Secrets main encryption key that was used to encrypt the API and ORG ID secrets. It should not be stored with the sealed secrets on GitHub. When you create the “Production” cluster, you will apply the main key before installing (or allowing Argo CD to install) Sealed-Secrets.

  1. Navigate out of your cloned repo to a local private folder.
  2. Copy the encryption key and save it as main.key file.
`kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key`.

Create the Edge Delta Agent Manifest

  1. Open this yaml in a browser: https://edgedelta.github.io/k8s/edgedelta-agent.yml
  2. Copy the current Edge Delta Agent manifest.

The current location is also listed in the Kubernetes agent installation instructions in the kubectl apply -f step:

  1. Create a folder in application_manifests called edgedelta.
  2. Paste the agent manifest in a file called edmanifest.yml in the application_manifests/edgedelta folder.

Customize the Edge Delta Agent Manifest

Open edmanifest.yml and make the following changes to the file to ensure it will work with the configMap:

  1. Configure the agent to not download a configuration from the Edge Delta back end by adding the ED_SKIP_CONF_DOWNLOAD environment variable value of "1".
spec:
  template: 
    spec:
      containers: 
      - name: edgedelta-agent
        env:
          - name: ED_SKIP_CONF_DOWNLOAD
            value: "1"
  1. Create the ED_ORG_ID environmental variable with the secretKeyRef of ed-org-id if it is missing.
spec:
  template: 
    spec:
      containers: 
      - name: edgedelta-agent
        env:
          - name: ED_ORG_ID
            valueFrom:
              secretKeyRef:
                key: ed-org-id
                name: ed-org-id
  1. Ensure that the secretKeyRef entry for ED_API_KEY matches the secret name: ed-api-key.
spec:
  template: 
    spec:
      containers: 
      - name: edgedelta-agent
        env:
          - name: ED_API_KEY
            valueFrom:
              secretKeyRef:
                key: ed-api-key
                name: ed-api-key
  1. Add commands to specify the location of the mounted configuration:
spec:
  template: 
    spec:
      containers: 
      - name: edgedelta-agent
        command:
          - /edgedelta/edgedelta
          - -c
          - /config/config.yml

The configuration file name is config.yml to align with the ConfigMap configured in the previous step.

  1. Add a volumeMount called edconfig with the mountPath of /config/. This adds the ConfigMap data to the directory specified in mountPath. This is required to ensure that updates made to the ConfigMap will be automatically applied without the need to restart the pod.
spec:
  template: 
    spec:
      containers: 
      - name: edgedelta-agent
        volumeMounts:
          - name: edconfig
            mountPath: /config/
  1. Add a volume called edconfig with a configMap called edgedelta-agent-config, which is the name of the ConfigMap.
spec:
  template: 
    spec:
      volumes:
        - name: edconfig
          configMap:
            name: edgedelta-agent-config

The final result as of time of writing:

apiVersion: v1
kind: Namespace
metadata:
  name: edgedelta
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: edgedelta
  namespace: edgedelta
  annotations:
    prometheus.io/scrape: "true"
  labels:
    k8s-app: edgedelta-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: edgedelta-logging
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: edgedelta-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: edgedelta
      containers:
      - name: edgedelta-agent
        image: gcr.io/edgedelta/agent:v0.1.71
        env:
          - name: ED_SKIP_CONF_DOWNLOAD
            value: "1"
          - name: ED_API_KEY
            valueFrom:
              secretKeyRef:
                key: ed-api-key
                name: ed-api-key
          - name: ED_ORG_ID
            valueFrom:
              secretKeyRef:
                key: ed-org-id
                name: ed-org-id
          - name: ED_HOST_OVERRIDE
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: ED_LEADER_ELECTION_ENABLED
            value: "1"
        command:
          - /edgedelta/edgedelta
          - -c
          - /config/config.yml
        resources:
          limits:
            memory: 2048Mi
          requests:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
          - name: edconfig
            mountPath: /config/
          - name: varlog
            mountPath: /var/log
            readOnly: true
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
            readOnly: true
          - name: persisting-cursor-storage
            mountPath: /var/lib/edgedelta
      terminationGracePeriodSeconds: 10
      volumes:
        - name: edconfig
          configMap:
            name: edgedelta-agent-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: persisting-cursor-storage
          hostPath:
            path: /var/lib/edgedelta
            type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: edgedelta
subjects:
- kind: ServiceAccount
  name: edgedelta
  namespace: edgedelta
roleRef:
  kind: ClusterRole
  name: edgedelta
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: edgedelta
  labels:
    k8s-app: edgedelta-logging
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  - events
  - nodes
  verbs:
  - get
  - watch
  - list
- apiGroups: [""] 
  resources:
  - events
  verbs:
  - create
- apiGroups: ["apps"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - watch
  - list
- apiGroups: ["batch"]
  resources:
  - jobs
  - cronjobs
  verbs:
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups: ["metrics.k8s.io"]
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: edgedelta
  namespace: edgedelta
  labels:
    k8s-app: edgedelta-logging

Create Argo CD application definitions

Next you create the Argo CD application definitions, which point to the manifests and configure the Argo CD settings. They set Argo CD to prune and self-heal resources. This removes resources that no longer have manifests in the configuration repo (prune), and it re-applies configuration repo manifests if conflicting changes are made directly to the cluster (selfHeal).

Note how the resource paths align with the folder structure configured so far.

  1. Create a folder in the repo root called argocd_apps
  2. Populate the following files in the argocd_apps folder.
  • api_secret_app.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: apisecret
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'git@github.com:user/repo.git'
    path: application_manifests/apikey
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: edgedelta
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
  • edgedelta_agent_config_app.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: edconfig
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'git@github.com:user/repo.git'
    path: application_manifests/appconfig
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: edgedelta
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
  • edgedelta_app.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: edagent
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'git@github.com:user/repo.git'
    path: application_manifests/edgedelta
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: edgedelta
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
  • orgidapp.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orgsecret
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'git@github.com:user/repo.git'
    path: application_manifests/orgid
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: edgedelta
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

Configure the Repo Location

So far, the Argo CD application definitions contain references to a placeholder repo:

repoURL: 'git@github.com:user/repo.git'

Replace all instances of this reference with your actual repo’s SSH URL.

Create the Sealed-Secrets Controller Application Definition

Create an application definition for the Sealed-Secrets Controller helm chart from Bitnami. At the time of writing the version is 2.10.0. This definition includes the helm parameter override required in the Sealed-Secrets installation instructions at the time of writing.

  • sealed_secrets.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: 'https://bitnami-labs.github.io/sealed-secrets'
    targetRevision: 2.10.0
    helm:
      parameters:
        - name: fullnameOverride
          value: sealed-secrets-controller
    chart: sealed-secrets
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: kube-system
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      selfHeal: true     

Push changes to the private repo

Add, commit and push the changes to your repo.

Create a new Cluster

  1. Delete the cluster that was used to create the sealed secrets. A new cluster will be used to demonstrate how a production cluster would be set up with Argo CD and Sealed Secrets.
kind delete cluster
  1. Create a new cluster
kind create cluster

Deploy the Main Encryption Key

Deploy the main encryption key that was backed up earlier. This will be used by Sealed Secrets to decrypt the secrets.

  1. Navigate to the main.key file’s location
  2. Deploy the main.key manifest:
kubectl apply -f main.key

Install Argo CD

Next, install Argo CD in the cluster.

kubectl create namespace argocd 
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Port Forward the Service to the localhost

When the pods are running, open a new terminal and port forward the service to your localhost.

kubectl get pods -n argocd

In a new terminal:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Log into Argo CD

In a terminal that isn’t being use for port forwarding, get the default Argo CD password.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Copy the password and use it to log into Argo CD using the admin account:

argocd login localhost:8080

Configure SSH keys

Provide Argo CD with your private key to access your repo. Replace the GitHub location with the SSH address of your own repo and specify the file containing the ssh key for your repo if it isn’t id_ed25519.

argocd repo add git@github.com:user/repo.git --ssh-private-key-path ~/.ssh/id_ed25519

Create the app-of-apps Definition

  1. Create a file named app-of-apps.yaml and paste the following contents. It should not be in the argocd_apps folder. Replace the repoURL with the SSH address of your own repo.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
spec:
  destination:
    name: ''
    namespace: argocd
    server: 'https://kubernetes.default.svc'
  source:
    path: argocd_apps
    repoURL: 'git@github.com:user/repo.git'
    targetRevision: HEAD
  sources: []
  project: default
  syncPolicy:
    syncOptions: []
  1. Deploy the app-of-apps application in the argocd namespace:
kubectl apply -n argocd -f app-of-apps.yml

Sync the app-of-apps application

Use the argocd app sync command to synchronize the app-of-apps application with its configured repo.

argocd app sync app-of-apps 

This will cause it to launch each application mentioned in the argocd_apps folder. In turn they deploy all the manifests including the sealed secrets controller, the secrets, the agent configuration and the agent itself.

You can log into the Argo CD GUI to see the deployed applications.

After a few minutes, you can log into Edge Delta to view the output of the demo node starting to populate the interface.

Update the Demo Node Configuration

To update a configuration, for example to change the demo parameters in the configMap, simply make a change to the config.yaml file and push the change to the repo.

  1. Open config.yml
  2. Update the demo_input node
  3. Push changes to the repo.
  4. The edconfig ArgoCD application will show out of sync. You can wait for it to synchronize or click SYNC and synchronize it manually.
  5. You can also view the live manifest in the Argo CD GUI to ensure it has been updated.