Edge Delta Metrics
3 minute read
Overview
Metrics, logs and traces are types of signals emitted by (properly instrumented) applications or systems to enable monitoring and troubleshooting. Metrics are numeric data or measurements of a system and the data is usually aggregated over time.
Metric Signals vs Log to Metric
There are two types of metrics in Edge Delta. Log to metrics are measurements that are derived by the Edge Delta agent by evaluating the content of logs. For example, the rate of 4xx errors reported in logs that are emitted by NGINX servers is determined by the agent by by examining /var/log/nginx/error.log
, identifying 4xx error logs, counting the number of such errors for a specific duration, and emitting a metric data item.
Metric signals, on the other hand, are generated directly by the data source. For example, the rate of 4xx errors being reported by NGINX servers can be determined by consuming metric signals from an nginx-prometheus-exporter
.
Metric Signals Examples
- Golden Signals: CPU, RAM, Network
- Kubernetes Metrics: kubeapi, kubelet, cAdvisor
- Kube State Metrics:
kube_secret_type{namespace="vm",secret="vm-grafana",type="Opaque"} 1
- Application Metrics:
- Metrics that are generated by applications directly, either alongside or instead of log data such as
nginxplus_upstream_server_responses{code=”4xx”,server=”192.168.45.119”,upstream”my_servers”} 23451
- Raw state metrics such as Google Nest metrics:
nest_ambient_temperature{id="abcd1234",label="Living-Room",scale=”fahrenheit”} 72
- Metrics that are generated by applications directly, either alongside or instead of log data such as
Metric Signal Sources
- Linux/VM
- eBPF (node, syscall)
- System (various exporters like node_exporter, /proc)
- Docker
- Linux + cadvisor (container metrics)
- Kubernetes
- Docker + cadvisor (kubelet) + KSM (Kube State Metrics, kubeapi)
- Custom Metrics
- Any/all of the other metric signal sources
- Custom exporters from infrastructure and app providers (Cilium, NGINX, Mongo, etc)
Edge Delta Metrics
The Edge Delta agent can ingest metric signals with full dimensions, including Kubernetes Source metrics and it handles metrics routing on the edge using Visual Pipelines.
Requirements
Metrics have the following prerequisites:
- To ingest Kube State Metrics, KSM must be deployed in the same cluster as the Edge Delta agent.
- To ingest node-exporter metrics, node-exporter must be deployed in the same cluster as the Edge Delta agent.
- All metric processing nodes, such as logs to metrics nodes, must be connected to the Metrics Output node.
The Edge Delta agent uses eBPF to collect network metrics. Therefore, the following Kubernetes environment configuration is required for Kubernetes network metrics and eBPF to work:
- Linux kernel version 5.8 or later.
- Linux kernel built with the
CONFIG_DEBUG_INFO_BTF=y
andCONFIG_DEBUG_INFO_BTF_MODULES=y
flags.
To check for the flag:
docker run -it --rm --privileged --pid=host ubuntu nsenter -t 1 -m -u -n -i sh -c 'cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF'
The output should show CONFIG_DEBUG_INFO_BTF=y
and CONFIG_DEBUG_INFO_BTF_MODULES=y
.
In the case of minikube, the agent can run on minikube with the Docker driver. Docker must be at least v26.0.0 (Docker Desktop v4.29.0) and it is started as follows:
minikube start --driver docker
If you want to disable eBPF:
- Delete the Kubernetes Traffic Input node.
- Disable the tracer:
Helm
Rerun the Helm upgrade command with the --set tracerProps.enabled=false
flag.
Kubectl Update and re-apply the kubernetes manifest with the following parameter change:
ED_ENABLE_TRAFFIC_TRACER = "0"