Edge Delta Metrics

Overview

Metrics, logs and traces are types of signals emitted by (properly instrumented) applications or systems to enable monitoring and troubleshooting. Metrics are numeric data or measurements of a system and the data is usually aggregated over time.

Metric Signals vs Log to Metric

There are two types of metrics in Edge Delta. Log to metrics are measurements that are derived by measuring the content of logs. For example, the rate of 4xx errors reported in logs that are emitted by NGINX servers can be determined by examining /var/log/nginx/error.log, identifying 4xx error logs, and counting the number of such errors for a specific duration.

Metric signals, on the other hand, are generated directly by the data source and emitted as metric signals. For example, the rate of 4xx errors being reported by NGINX servers can be determined by consuming metric signals from an nginx-prometheus-exporter.

Metric Signals Examples

• Golden Signals: CPU, RAM, Network
• Kubernetes Metrics: kubeapi, kubelet, cAdvisor
• Kube State Metrics: kube_secret_type{namespace="vm",secret="vm-grafana",type="Opaque"} 1
• Application Metrics:
  • Metrics that are generated by applications directly, either alongside or instead of log data such as nginxplus_upstream_server_responses{code=”4xx”,server=”192.168.45.119”,upstream”my_servers”} 23451
  • Raw state metrics such as Google Nest metrics: nest_ambient_temperature{id="abcd1234",label="Living-Room",scale=”fahrenheit”} 72

Metric Signal Sources

• Linux/VM
  • eBPF (node, syscall)
  • System (various exporters like node_exporter, /proc)
• Docker
  • Linux + cadvisor (container metrics)
• Kubernetes
  • Docker + cadvisor (kubelet) + KSM (Kube State Metrics, kubeapi)
• Custom Metrics
  • Any/all of the other metric signal sources
  • Custom exporters from infrastructure and app providers (Cilium, NGINX, Mongo, etc)

Edge Delta Metrics

The Edge Delta SaaS can ingest metric signals with full dimensions from the agent, including Kubernetes Source metrics. It handles metrics routing on the edge using Visual Pipelines and you can build custom dashboards in the Edge Delta web app.

Requirements

Metrics have the following prerequisites:

1. To ingest Kube State Metrics, KSM must be deployed in the same cluster as the Edge Delta agent.
2. To ingest node-exporter metrics, node-exporter must be deployed in the same cluster as the Edge Delta agent.
3. All metric processing nodes, such as logs to metrics nodes, must be connected to the Metrics Output node.

The Edge Delta agent uses eBPF to collect network metrics. Therefore, the following Kubernetes environment configuration is required for Kubernetes network metrics and eBPF to work:

• Linux kernel version 5.8 or later.
• Linux kernel built with the CONFIG_DEBUG_INFO_BTF=y flag.

To check for the flag, SSH into the node and examine the config, for example in minikube:

minikube sh
cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF

The output should show CONFIG_DEBUG_INFO_BTF=y.

In the case of minikube, the agent can run on minikube with the Docker driver. Docker must be at least v24.0.0 and it is started as follows:

minikube start --driver Docker