Edge Delta Metrics

Edge Delta ingests metrics signals and it calculates metrics from logs.

  3 minute read  

Overview

Metrics, logs and traces are types of signals emitted by (properly instrumented) applications or systems to enable monitoring and troubleshooting. Metrics are numeric data or measurements of a system and the data is usually aggregated over time.

Metric Signals vs Log to Metric

There are two types of metrics in Edge Delta. Log to metrics are measurements that are derived by the Edge Delta agent by evaluating the content of logs. For example, the rate of 4xx errors reported in logs that are emitted by NGINX servers is determined by the agent by by examining /var/log/nginx/error.log, identifying 4xx error logs, counting the number of such errors for a specific duration, and emitting a metric data item.

Metric signals, on the other hand, are generated directly by the data source. For example, the rate of 4xx errors being reported by NGINX servers can be determined by consuming metric signals from an nginx-prometheus-exporter.

Metric Signals Examples

  • Golden Signals: CPU, RAM, Network
  • Kubernetes Metrics: kubeapi, kubelet, cAdvisor
  • Kube State Metrics: kube_secret_type{namespace="vm",secret="vm-grafana",type="Opaque"} 1
  • Application Metrics:
    • Metrics that are generated by applications directly, either alongside or instead of log data such as nginxplus_upstream_server_responses{code=”4xx”,server=”192.168.45.119”,upstream”my_servers”} 23451
    • Raw state metrics such as Google Nest metrics: nest_ambient_temperature{id="abcd1234",label="Living-Room",scale=”fahrenheit”} 72

Metric Signal Sources

  • Linux/VM
    • eBPF (node, syscall)
    • System (various exporters like node_exporter, /proc)
  • Docker
  • Kubernetes
    • Docker + cAdvisor (kubelet) + KSM (Kube State Metrics, kubeapi)
  • Custom Metrics
    • Any/all of the other metric signal sources
    • Custom exporters from infrastructure and app providers (Cilium, NGINX, Mongo, etc)

Edge Delta Metrics

The Edge Delta agent can ingest metric signals with full dimensions, including Kubernetes Source metrics and it handles metrics routing on the edge using Visual Pipelines.

Requirements

Metrics have the following prerequisites:

  1. To ingest Kube State Metrics, KSM must be deployed in the same cluster as the Edge Delta agent.
  2. To ingest node-exporter metrics, node-exporter must be deployed in the same cluster as the Edge Delta agent.
  3. All metric processing nodes, such as logs to metrics nodes, must be connected to the Metrics destination node.

The Edge Delta agent uses eBPF to collect network metrics. Therefore, the following Kubernetes environment configuration is required for Kubernetes network metrics and eBPF to work:

  • Linux kernel version 5.8 or later.
  • Linux kernel built with the CONFIG_DEBUG_INFO_BTF=y and CONFIG_DEBUG_INFO_BTF_MODULES=y flags.

To check for the flag:

docker run -it --rm --privileged --pid=host ubuntu nsenter -t 1 -m -u -n -i sh -c 'cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF'

The output should show CONFIG_DEBUG_INFO_BTF=y and CONFIG_DEBUG_INFO_BTF_MODULES=y.

In the case of minikube, the agent can run on minikube with the Docker driver. Docker must be at least v26.0.0 (Docker Desktop v4.29.0) and it is started as follows:

minikube start --driver docker

If you want to disable eBPF:

  1. Delete the Kubernetes Traffic source node.
  2. Disable the tracer:

Helm Rerun the Helm upgrade command with the --set tracerProps.enabled=false flag.

Kubectl Update and re-apply the kubernetes manifest with the following parameter change:

ED_ENABLE_TRAFFIC_TRACER = "0"