Edge Delta in Multi-Region Kubernetes Environments

Setting Up Edge Delta for Multi-Cluster and Multi-Region Kubernetes Environments.

  9 minute read  

Recap

Generally, a fleet is composed of a pipeline configuration and a number of deployed agents. It can have no agents currently deployed, but it always has a pipeline configuration. Similarly a pipeline is always associated with a fleet. Usually, a fleet is representative of a single cluster. However, this architecture can be changed depending on your use case.

Introduction

This tutorial will guide you through the process of configuring Edge Delta to monitor a complex Kubernetes environment. It will provide a basic understanding of Edge Delta concepts such as fleets, pipelines, and effective use of labels, compound nodes, or the ED_TAG_OVERRIDE environment variable (edTagOverride Helm variable). The aim is to ensure high availability, consistent configuration, and efficient resource management.

Prerequisites

  • Edge Delta account.
  • Kubernetes clusters spread across the required regions.

Scenario Environment

The environment consists of multiple clusters spread across different regions and stages. In this use case, there are three stages:

  • Development (dev)
  • Testing (uat)
  • Production (prod)

Suppose a disaster recovery policy dictates that each stage should be deployed in two geographically distributed cloud regions, each on at least two clusters per region:

  • us-east1
  • us-west2

Regardless of the number of clusters deployed, all clusters of a given type are always 100% identical and runbooked. Therefore the following clusters at a minimum should exist:

Solution 1 - Compound Nodes

The recommended solution is to deploy unique fleets per cluster and use Compound Nodes to configure the common pipeline elements across all the pipelines.

Benefits

  • You can make a change to a single compound node and have it propagated across multiple pipelines.
  • The fleet pipelines are not dependant on each other so you can delete one fleet without affecting others.
  • You can have customized pipelines on a per cluster or region basis, for example by having a different transformation type node depending on the region, while common components are configured with compound nodes.
  • The dashboards in Edge Delta are natively segmented by fleet or cluster (agent tag).
  • Aggregated data will be for each fleet or cluster on the Pipelines Dashboard

Drawbacks

  • There may be some process planning required to manage compound node changes.

Deployment:

Step 1: Configure Compound Nodes

  1. Create a compound node for processing data specifically from a dev environment regardless of its location or cluster within that location. Name it DEV or something similar.
  2. Similarly, create a compound node specifically for uat environments regardless of their location or cluster within that location. Name it UAT or something similar.
  3. Create a compound node specifically for prod environments regardless of their location or cluster within that location. Name it PROD or something similar.

Step 2: Deploy the dev_us-east1_cluster1 Fleet

Deploy a fleet for dev in us-east1 named dev_us-east1_cluster1. Configure its pipeline with the DEV compound node and add any other nodes specifically required for us-east1 or cluster1.

Step 3: Deploy the dev_us-east1_cluster2 Fleet

Deploy a fleet for dev in us-east1 named dev_us-east1_cluster2. You can copy dev_us-east1_cluster1 as a base pipeline and modify it with any other nodes specifically required for us-east1 or cluster2.

Step 4: Deploy other dev Fleets

Repeat step 3 for the us-west2 clusters.

Step 5: Deploy uat and prod Fleets

Repeat steps 2, 3 and 4 for uat and prod fleets using the UAT and PROD compound nodes respectively.

Step 6: Verify Deployment

Thoroughly test the deployment.

  1. Access Edge Delta: Log in to your Edge Delta account.
  2. Navigate to Fleets: Check that the newly created fleets correspond correctly to your environment.
  3. Monitor Agents: Ensure that agents are correctly reporting back from all clusters.
  4. Monitor Telemetry: Ensure that logs, metrics and events are being produced by all clusters and that they contain the ed.tag resource indicating which cluster they come from:
{
  "_type": "log"
  ...
  "resource": {
    "ed.tag": "dev_us-east1_cluster2"
    ...
  }
}

Solution 2 - Metadata Tags

An alternative approach may be to deploy the same fleet across all regions and clusters for each stage without a tag override, and use metadata such to identify the region and cluster names.

To uniquely identify each environment, leverage one or a combination of:

  • Edge Delta Custom Tags for tagging logs and resources.
  • Labels and Annotations on nodes and Kubernetes objects.
  • ConfigMaps or Secrets to store non-ephemeral cluster metadata.
  • Environment Variables set in your pod specifications.
  • Cloud Provider Metadata if using managed services.
  • Service Mesh Configuration for automatic metadata propagation.

Benefits

  • You can make a change to a single pipeline and have it propagated across multiple clusters.
  • The fleet pipelines are not dependant on each other so you can delete one fleet without affecting others.

Drawbacks

  • You might want to further customize pipelines on a per cluster or region basis, for example by having a different transformation type node depending on the region.
  • The dashboards in Edge Delta is natively segmented by fleet (agent tag). To differentiate between regions and clusters you will need to configure the cluster appropriately and configure custom facets in Log Search, Metric Explorer, etc.
  • Aggregated data will be for each fleet on the Pipelines Dashboard, rather than per region/cluster.

Deployment:

This deployment example uses the edCustomTag helm variable to add attributes to data items that can be used to identify their source. The steps will illustrate using a helm variable, but you can alternatively use an environment variable.

Step 1: Deploy dev_us-east1_cluster1

Helm Example

Use Helm to deploy a fleet named dev_us in cluster1 in us-east1. However, when deploying it, add the following Edge Delta Custom Tags by adding this flag to the upgrade command:

--set edCustomTags="stage:dev|region:us-east1|cluster:cluster1"

kubectl Example:

Use kubectl to deploy a fleet named dev_us in cluster1 in us-east1. However, when deploying it, use a modified version of the manifest that includes the following environment variable for the Processor DaemonSet, Rollup StatefulSet, and Compactor StatefulSet:

env:
  - name: ED_CUSTOM_TAGS
    value: "stage:dev|region:us-east1|cluster:cluster1"

Configure the pipeline appropriately for all dev clusters in all regions. If you need to have different processors per region or cluster you can use a route node that splits traffic on different paths depending on the relevant attributes.

Step 2: Deploy dev_us-east1_cluster2

Helm Example

  1. Click dev_us in the Pipelines - Dashboard page and click Add Agents.
  2. Select Helm.
  3. Use Helm to deploy agents for the dev_us fleet in cluster 2, us-east1. However, when deploying, add the following Edge Delta Custom Tags by adding this flag to the upgrade command:
--set edCustomTags="stage:dev|region:us-east1|cluster:cluster2"

Kubectl Example:

  1. Click dev_us in the Pipelines - Dashboard page and click Add Agents.
  2. Select Kubernetes.
  3. Open the file mentioned in the kubectl apply command and create a local copy of the manifest:
https://raw.githubusercontent.com/edgedelta/k8s/master/edgedelta-agent-k8s-from-helm.yaml

Add the following environment variable to the Processor DaemonSet, Rollup StatefulSet, and Compactor StatefulSet:

env:
  - name: ED_CUSTOM_TAGS
    value: "stage:dev|region:us-east1|cluster:cluster2"

Step 3: Deploy the remaining dev clusters

Repeat step 2 for the remaining dev clusters in us-west2. Each time, use the dev_us fleet installation instructions, but update the flag appropriately for cluster1:

Helm Example

--set edCustomTags="stage:dev|region:us-west2|cluster:cluster1"

Kubectl Example:

env:
  - name: ED_CUSTOM_TAGS
    value: "stage:dev|region:us-west2|cluster:cluster1"

For cluster2:

Helm Example

--set edCustomTags="stage:dev|region:us-west2|cluster:cluster2"

Kubectl Example:

env:
  - name: ED_CUSTOM_TAGS
    value: "stage:dev|region:us-west2|cluster:cluster2"

Step 4: Deploy UAT

Repeat steps 1, 2, and 3 for uat:

Fleet Name: uat_us

Tags:

"stage:uat|region:us-east1|cluster:cluster1"

"stage:uat|region:us-east1|cluster:cluster2"

"stage:uat|region:us-west2|cluster:cluster1"

"stage:uat|region:us-west2|cluster:cluster2"

Step 5: Deploy PROD

Repeat steps 1, 2, and 3 for prod:

Fleet Name: prod_us

Tags:

"stage:prod|region:us-east1|cluster:cluster1"

"stage:prod|region:us-east1|cluster:cluster2"

"stage:prod|region:us-west2|cluster:cluster1"

"stage:prod|region:us-west2|cluster:cluster2"

Step 6: Verify Deployment

Thoroughly test the deployment.

  1. Access Edge Delta: Log in to your Edge Delta account.
  2. Navigate to Fleets: Check that the newly created fleets correspond correctly to your environments.
  3. Monitor Agents: Ensure that agents are correctly reporting back from all clusters.
  4. Monitor Telemetry: Ensure that logs, metrics and events are being produced by all clusters and that they contain the attributes defined with edCustomTags:
{
  "_type": "log"
  ...
  "attributes": {
    "stage": "prod"
    "region": "us-east1"
    "cluster": "cluster1"
  }

Solution 3 - Tag Overrides

In this solution, only three unique pipeline configurations are required - one per stage. Therefore, a shared pipeline deployment is used, with tag overrides to identify the clusters.

Benefits

  • The dashboards in Edge Delta is natively segmented by fleet (agent tag).
  • You can filter by agent tag in Log Search, Metric Explorer, on the K8s Overview etc.
  • You can view aggregated data for each fleet on the Pipelines Dashboard.
  • You can make a change to a single pipeline and have it propagated across multiple fleets.

Drawbacks

  • It is harder to differentiate between fleets and the pipelines they implement because the pipelines are named for only their first fleet.
  • Deleting a fleet will delete all the other fleets that use the same pipeline configuration even if they have tag overrides. For example if you delete dev_us-east1_cluster1, then dev_us-east1_cluster2, dev_us-west2_cluster1, and dev_us-west2_cluster2 will also be deleted (those agents will no longer be able to communicate with the Edge Delta back end).
  • You might want to further customize pipelines on a per fleet basis, for example by having a different transformation type node depending on the region.

Deployment:

Step 1: Configure 1st Dev Fleet

Deploy a fleet for dev in us-east1 named dev_us-east1_cluster1. Configure its pipeline specifically for the Dev stage.

Step 2: Deploy dev_us-east1_cluster2

View the installation commands for dev_us-east1_cluster1 and execute them on us-east1 cluster2, while passing in the tag override variable to specify the name dev_us-east1_cluster2:

Helm Example:

  1. Click Pipelines and select the Fleet with the Pipeline you want to share.
  2. Click Add Agents.
  3. Select Helm.
  4. Copy and execute the commands, while passing in the following Helm value to the helm upgrade command:
--set edTagOverride=dev_us-east1_cluster2

Kubectl Example:

  1. Click Pipelines and select the dev_us-east1_cluster1 Fleet.
  2. Click Add Agents.
  3. Select Kubernetes.
  4. Open the file mentioned in the kubectl apply command and create a local copy of the manifest:
https://raw.githubusercontent.com/edgedelta/k8s/master/edgedelta-agent-k8s-from-helm.yaml

Add the following environment variable to the Processor DaemonSet, Rollup StatefulSet, and Compactor StatefulSet:

env:
  - name: ED_TAG_OVERRIDE
    value: "dev_us-east1_cluster2"

Step 3: Deploy dev_us-west2_cluster1

Repeat Step 2 but execute them on us-west2 cluster1, while specifying the tag override variable to specify the name dev_us-west2_cluster1

Step 4: Deploy dev_us-west2_cluster2

Repeat Step 2 but execute execute them on us-west2 cluster2, while specifying the tag override variable to specify the name dev_us-west2_cluster2

All the DEV environment agents have now been deployed.

  • dev_us-east1_cluster1
  • dev_us-east1_cluster2
  • dev_us-west2_cluster1
  • dev_us-west2_cluster2

Step 5 Deploy uat and prod

  1. Repeat steps 1 to 4 for uat
  • uat_us-east1_cluster1
  • uat_us-east1_cluster2
  • uat_us-west2_cluster1
  • uat_us-west2_cluster2
  1. Repeat steps 1 to 4 for prod
  • prod_us-east1_cluster1
  • prod_us-east1_cluster2
  • prod_us-west2_cluster1
  • prod_us-west2_cluster2

Step 6: Verify Deployment

Thoroughly test the deployment.

  1. Access Edge Delta: Log in to your Edge Delta account.
  2. Navigate to Fleets: Check that the newly created fleets correspond correctly to your environments.
  3. Monitor Agents: Ensure that agents are correctly reporting back from all clusters.
  4. Monitor Telemetry: Ensure that logs, metrics and events are being produced by all clusters and that they contain the ed.tag resource indicating which cluster they come from:
{
  "_type": "log"
  ...
  "resource": {
    "ed.tag": "dev_us-east1_cluster2"
    ...
  }
}

Solution 4 - Hybrid Approach

You can deploy a hybrid solution by using elements from each approach. For example by adding compound nodes to the Custom Tag approach.

Conclusion

You can configure Edge Delta for multi-region Kubernetes environments for high availability, consistent configuration, and efficient resource management. Using compound nodes, metadata tags, and tag overrides, enables you to tailor your observability pipeline to meet the specific needs of different architectures. Each approach offers unique benefits and potential drawbacks. A hybrid solution combining the advantages of multiple methods might be the most effective strategy. Thoroughly test your solution to ensure that all components, fleets, agents, and telemetry data are working as expected.

See Also