Integrating Edge Delta with Prometheus

For the V3 configuration, see here.

Agent Metrics

You can configure an Edge Delta agent to expose a metrics endpoint for Prometheus to pull. The following metrics are exposed:

The metric endpoint output should be similar to the following example:

# HELP edgedelta_incoming_bytes_total Sum of incoming bytes
# TYPE edgedelta_incoming_bytes_total counter
edgedelta_incoming_bytes_total{logical_source="K8s,default,CronJob,metric-migration-org-1,metric-migration-org-1"} 8.690409e+06
edgedelta_incoming_bytes_total{logical_source="K8s,default,CronJob,metric-migration-org-2,metric-migration-org-2"} 1.706539e+06
edgedelta_incoming_bytes_total{logical_source="K8s,default,CronJob,metric-migration-org-3,metric-migration-org-3"} 1.61757e+06
edgedelta_incoming_bytes_total{logical_source="K8s,default,CronJob,metric-migration-org-4,metric-migration-org-4"} 997915

In addition, Regex Processors can expose rule metrics to the metrics endpoint for Prometheus to scrape.

Configuration Overview

To integrate Edge Delta with Prometheus, take the following conceptual steps. The exact details depend on your environment. See the following tutorial for a more detailed example that uses Helm.

1. Configure an Edge Delta agent in the Edge Delta web app to expose a metrics endpoint. To do this you set internal_prom_stats_enabled to true, which configures the Edge Delta agent to create an endpoint on port 8087 by default.

agent_settings:
  internal_prom_stats_enabled: true

2. Upgrade or reinstall the Edge Delta agent in your environment to include a service monitor configuration. This creates a Kubernetes service that selects Edge Delta agents, and a Kubernetes Service Monitor that selects the service. Then Prometheus service discovery is able to find the endpoint and begin scaping it as a target. The service monitor configuration defines the metrics port with promPort as well as other environment variables such as the Prometheus namespace and a selector label. The selector label reflects how the Prometheus configuration matches ServiceMonitor. In the following example, assume Prometheus is configured to look for its release label, so the selector configured in the Edge Delta agent is set to the release name.

promPort: 8087
serviceMonitor:
  enabled: true
  namespace: <Prometheus namespace e.g. monitoring>
  scrapeTimeout: 10s
  jobLabel: edgedelta
  selector:
    release: <Prometheus release name e.g. prom-kube-stack>

If you want regex processors to expose rule metrics to the metrics endpoint for Prometheus to scrape, you set rule_metrics_prom_stats_enabled: to true in the agent_settings section.

agent_settings:
  rule_metrics_prom_stats_enabled: true

Tutorial: Install and Integrate Edge Delta with Prometheus using Helm

The following tutorial will demonstrate how to install and configure an Edge Delta agent to integrate with a Prometheus Stack on a local Kubernetes cluster using Helm. The Edge Delta agent will expose metrics on an endpoint and service discovery will enable Prometheus to define it as a target, scrape the metrics, and make them available to Grafana.

Prerequisites

To perform this tutorial, ensure you have the following prerequisites:

• Helm is installed,
• Docker Desktop is running,
• kind and kubectl are installed, and
• you have an Edge Delta subscription.

1. Create a Cluster

Start a local cluster on Docker Desktop using kind.

kind create cluster 

At the end of this tutorial you can delete this cluster with kind delete cluster

2. Download Charts

Download the Helm repositories for Edge Delta and Prometheus-Community, then update the repos.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo add edgedelta https://edgedelta.github.io/charts

helm repo update

3. Install the Prometheus Stack

Install the kube-prometheus-stack Helm chart which includes Prometheus Operator and Grafana. In this tutorial, it is installed in the monitoring namespace and the release is named prom-kube-stack. These details will be referred to later in the chart override for the Edge Delta installation, so if they change here you need to update the override values accordingly.

helm install --namespace monitoring --create-namespace prom-kube-stack prometheus-community/kube-prometheus-stack

4. Create and Configure an Agent

Create a new Kubernetes Helm agent in the Edge Delta Web Application.

1. In the web app, select Pipelines - Agent Settings.
2. Click Create Configuration
3. Select the Helm template
4. Set internal_prom_stats_enabled to true in the agent configuration.
5. Set rule_metrics_prom_stats_enabled to true so that regex processors can expose rule metrics to the metrics endpoint for Prometheus to scrape.

The agent settings section should look similar to this example:

#Global settings to apply to the agent
agent_settings:
  tag: helm_onboarding
  log:
    level: info
  anomaly_capture_size: 1000
  anomaly_confidence_period: 30m
  anomaly_tolerance: .15
  anomaly_coefficient: 5
  agent_stats_enabled: true
  internal_prom_stats_enabled: true

6. Click Save Edits.
7. Copy the agent ID in the first column of your agent row. This is also the secret API key.

5. Create a Helm Override Manifest

Create a helm override file. In this example it is named myvalues.yaml, enter the following contents, and save it in a local folder on your machine:

promPort: 8087
serviceMonitor:
  enabled: true
  namespace: monitoring
  scrapeTimeout: 10s
  jobLabel: edgedelta
  selector:
    release: prom-kube-stack

6. Install the Customized Edge Delta Agent

Navigate to the folder containing the myvalues.yaml override manifest. Install the Edge Delta agent, using the Edge Delta interface instructions, passing in the override file you created and setting the secret API key copied earlier from the Edge Delta web app. In the following example the API key is 12346789 but you should replace it with your key.

--values myvalues.yaml 

--set secretApiKey.value=123456789

An Edge Delta Agent is now running in Kubernetes and exposing metrics to a metrics endpoint. A Kubernetes service that selects Edge Delta agents, and k8s Service Monitor that selects this Service has also been created. The endpoint can now be scraped by Prometheus and viewed in Grafana.

Test the Metrics Endpoint (Optional)

You can test whether the Edge Delta agent is exposing a metrics endpoint by port forwarding an Edge Delta pod and viewing the output in a browser. To start, get the name of an Edge Delta pod.

kubectl get pod -n edgedelta

Copy the pod name, in this example edgedelta-bvfd,2 and ensure that the pod is running. Then port forward it to your browser:

kubectl port-forward edgedelta-bvfd2 8087:8087 -n edgedelta

Open localhost:8087 in your browser to confirm that metrics are being produced. They should look similar to this example:

# HELP edgedelta_open_files_total 0 or 1 Count if a file is closed or open
# TYPE edgedelta_open_files_total gauge
edgedelta_open_files_total{file_path="/var/log/pods/local-path-storage_local-path-provisioner-9bd9hk544-9kgfr_878a9b72-8098-1964-hj6f6-39b2c685924n/local-path-provisioner/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_alertmanager-prom-kube-stack-kube-prome-alertmanager-0_m9868b51-k5jh-4gg66-4893-d2ff1dd883c/alertmanager/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_alertmanager-prom-kube-stack-kube-prome-alertmanager-0_b19u8b24-j5mm-4jn5-1234-d3th8s3d951c/config-reloader/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_prom-kube-stack-grafana-655965m4n4-2gfdss_69f5x107-9cvb-4345-mnm7-9617835l44k4/grafana-sc-dashboard/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_prom-kube-stack-grafana-639284j4h4-2gfds_69f5r107-9tyu-4345-iio7-6163976o44p4/grafana-sc-datasources/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_prom-kube-stack-grafana-784562m4n4-2bvcx_69f5a107-9eba-4345-zas7-9189181s44d4/grafana/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_prom-kube-stack-kube-prome-operator-67fgh4ghb44-ghj5j_8jj37k93-lopi-4c19-uyt4-r12985969547/kube-prometheus-stack/0.log"} 1
edgedelta_open_files_total{file_path="/var/log/pods/monitoring_prom-kube-stack-kube-state-metrics-77qwert5ff-asdfg_y6hhj7g1-f58d-4s0x-cv92-3952264126584/kube-state-metrics/0.log"} 1
# HELP edgedelta_outgoing_bytes_total Sum of outgoing bytes
# TYPE edgedelta_outgoing_bytes_total counter
edgedelta_outgoing_bytes_total{data_type="",destination="Edgedelta-Backend-Influx",integration_name="",logical_source=""} 17339
# HELP edgedelta_outgoing_lines_total Count of outgoing lines
# TYPE edgedelta_outgoing_lines_total counter
edgedelta_outgoing_lines_total{data_type="",destination="Edgedelta-Backend-Influx",integration_name="",logical_source=""} 17
# HELP edgedelta_outgoing_raw_bytes_total Sum of outgoing bytes
# TYPE edgedelta_outgoing_raw_bytes_total counter
edgedelta_outgoing_raw_bytes_total{data_type="",destination="Edgedelta-Backend-Influx",integration_name="",logical_source=""} 0
# HELP edgedelta_outgoing_success_counts Count of successful outgoing streams
# TYPE edgedelta_outgoing_success_counts counter
edgedelta_outgoing_success_counts{destination="Edgedelta-Backend-Influx",integration_name=""} 2

You can stop port forwarding by pressing Control+C in the terminal.

Open the Prometheus UI (Optional)

To view the Prometheus UI you port forward its service. First you get the service name and port:

kubectl get services -n monitoring

In this example the service is prom-kube-stack-kube-prome-prometheus on port 9090. Next you port forward the service:

kubectl port-forward svc/prom-kube-stack-kube-prome-prometheus 9090:9090 -n monitoring

Open localhost:9090 in your browser. Select Status - Target. You should see the edgedelta target pod is up:

serviceMonitor/monitoring/edgedelta-service-monitor/0 (1/1 up)

Some Kube pods might not be available as targets due to port limitations with running a local kind Kubernetes cluster using default settings.

You can stop port forwarding by pressing Control+C in the terminal.

Open the Grafana UI (Optional)

To view the Grafana UI, first get the Grafana service name and port:

kubectl get services -n monitoring

In this example, it is prom-kube-stack-grafana on port 80. You need the Grafana username and password. In this example you can get their secret entries:

kubectl get secret --namespace monitoring prom-kube-stack-grafana -o jsonpath='{.data.admin-user}' | base64 -d

kubectl get secret --namespace monitoring prom-kube-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d

Bear in mind a production instance shouldn’t save unencrypted usernames and password secrets.

Port forward the Grafana service:

kubectl port-forward --namespace monitoring svc/prom-kube-stack-grafana 8080:80  

Open localhost:8080. Select Settings - Data Sources and confirm that Prometheus is a data source. Select Explore and confirm that Edge Delta metrics are available in the metric drop down within the query builder. If Prometheus is set up as a data source in Grafana, the scraped agent stats can be visualized with the following sample queries.

sum by(logical_source) (rate(edgedelta_incoming_lines_total[5m]))

sum by(destination) (rate(edgedelta_outgoing_bytes_total[5m]))

You can stop port forwarding by pressing Control+C in the terminal.

See Also:

GitOps Principles Deployment of Edge Delta