Edge Delta Kubernetes Metrics Source

Scrape Kubernetes metrics.

  4 minute read  

Beta

Overview

You can use the Kubernetes Metrics source node to scrape certain Kubernetes metrics. All Kubernetes metrics families are scraped by default.

Metrics have the following prerequisites:

  1. To ingest Kube State Metrics, KSM must be deployed in the same cluster as the Edge Delta agent.
  2. To ingest node-exporter metrics, node-exporter must be deployed in the same cluster as the Edge Delta agent.
  3. All metric processing nodes, such as logs to metrics nodes, must be connected to the Metrics destination node.

The Edge Delta agent uses eBPF to collect network metrics. Therefore, the following Kubernetes environment configuration is required for Kubernetes network metrics and eBPF to work:

  • Linux kernel version 5.8 or later.
  • Linux kernel built with the CONFIG_DEBUG_INFO_BTF=y and CONFIG_DEBUG_INFO_BTF_MODULES=y flags.

To check for the flag:

docker run -it --rm --privileged --pid=host ubuntu nsenter -t 1 -m -u -n -i sh -c 'cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF'

The output should show CONFIG_DEBUG_INFO_BTF=y and CONFIG_DEBUG_INFO_BTF_MODULES=y.

In the case of minikube, the agent can run on minikube with the Docker driver. Docker must be at least v26.0.0 (Docker Desktop v4.29.0) and it is started as follows:

minikube start --driver docker

If you want to disable eBPF:

  1. Delete the Kubernetes Traffic source node.
  2. Disable the tracer:

Helm Rerun the Helm upgrade command with the --set tracerProps.enabled=false flag.

Kubectl Update and re-apply the kubernetes manifest with the following parameter change:

ED_ENABLE_TRAFFIC_TRACER = "0"

Metrics

The following metrics are ingested by the node:

Kube State Metrics (KSM)

To ingest Kube State Metrics, a version of KSM must be deployed in the same cluster as the Edge Delta agent to ingest Kube State Metrics into Edge Delta. Please see external KSM documentation and more info about installation here. The following two KSM metrics are required for Kubernetes requests and limits to be exposed via Edge Delta Golden Signals:

  • ed_k8s_metric_kube_pod_container_resource_requests_<resource>.value
  • ed_k8s_metric_kube_pod_container_resource_limits_<resource>.value

You will need to restart the Edge Delta fleet after installing Kube State Metrics:

kubectl rollout restart daemonset/edgedelta -n edgedelta

cAdvisor: See external documentation here.

  • ed_k8s_metric_container_cpu_cfs_throttled_periods.value
  • ed_k8s_metric_container_cpu_cfs_periods.value
  • ed_k8s_metric_container_cpu_cfs_throttled_seconds.value
  • ed_k8s_metric_container_cpu_usage_seconds.value
  • ed_k8s_metric_container_fs_writes_bytes.value
  • ed_k8s_metric_container_fs_reads_bytes.value
  • ed_k8s_metric_container_spec_cpu_period.value
  • ed_k8s_metric_container_last_seen.value
  • ed_k8s_metric_container_memory_usage_bytes.value
  • ed_k8s_metric_container_network_receive_bytes.value
  • ed_k8s_metric_container_network_receive_errors.value
  • ed_k8s_metric_container_network_receive_packets_dropped.value
  • ed_k8s_metric_container_network_receive_packets.value
  • ed_k8s_metric_container_network_transmit_bytes.value
  • ed_k8s_metric_container_network_transmit_errors.value
  • ed_k8s_metric_container_network_transmit_packets_dropped.value
  • ed_k8s_metric_container_network_transmit_packets.value
  • ed_k8s_metric_container_processes.value

Kubelet: See external documentation here.

  • ed_k8s_metric_kubelet_cgroup_manager_duration_seconds.histogram
  • ed_k8s_metric_kubelet_cgroup_manager_duration_seconds_sum.value
  • ed_k8s_metric_kubelet_cgroup_manager_duration_seconds_count.value
  • ed_k8s_metric_kubelet_node_config_error.value
  • ed_k8s_metric_kubelet_pleg_relist_duration_seconds.histogram
  • ed_k8s_metric_kubelet_pleg_relist_duration_seconds_sum.value
  • ed_k8s_metric_kubelet_pleg_relist_duration_seconds_count.value
  • ed_k8s_metric_kubelet_pleg_relist_interval_seconds.histogram
  • ed_k8s_metric_kubelet_pleg_relist_interval_seconds_sum.value
  • ed_k8s_metric_kubelet_pleg_relist_interval_seconds_count.value
  • ed_k8s_metric_kubelet_pod_start_duration_seconds.histogram
  • ed_k8s_metric_kubelet_pod_start_duration_seconds_sum.value
  • ed_k8s_metric_kubelet_pod_start_duration_seconds_count.value
  • ed_k8s_metric_kubelet_pod_worker_duration_seconds.histogram
  • ed_k8s_metric_kubelet_pod_worker_duration_seconds_sum.value
  • ed_k8s_metric_kubelet_pod_worker_duration_seconds_count.value
  • ed_k8s_metric_kubelet_running_containers.value
  • ed_k8s_metric_kubelet_running_pods.value
  • ed_k8s_metric_kubelet_runtime_operations_duration_seconds.histogram
  • ed_k8s_metric_kubelet_runtime_operations_duration_seconds_sum.value
  • ed_k8s_metric_kubelet_runtime_operations_duration_seconds_count.value
  • ed_k8s_metric_kubelet_runtime_operations_errors.value
  • ed_k8s_metric_kubelet_runtime_operations.value
  • ed_k8s_metric_process_cpu_seconds.value
  • ed_k8s_metric_process_resident_memory_bytes.value
  • ed_k8s_metric_rest_client_request_duration_seconds.histogram
  • ed_k8s_metric_rest_client_request_duration_seconds_sum.value
  • ed_k8s_metric_rest_client_request_duration_seconds_count.value
  • ed_k8s_metric_rest_client_requests.value
  • ed_k8s_metric_storage_operation_duration_seconds.histogram
  • ed_k8s_metric_storage_operation_duration_seconds_sum.value
  • ed_k8s_metric_storage_operation_duration_seconds_count.value
  • ed_k8s_metric_volume_manager_total_volumes.value

Node exporter:

To ingest node-exporter metrics, a version of node-exporter must be deployed in the same cluster as the Edge Delta agent to ingest node-exporter metrics into Edge Delta. Please see external node-exporter documentation and for more info about installation.

  • ed_k8s_metric_node_cpu_seconds.value
  • ed_k8s_metric_node_disk_read_bytes.value
  • ed_k8s_metric_node_disk_read_time_seconds.value
  • ed_k8s_metric_node_disk_write_time_seconds.value
  • ed_k8s_metric_node_disk_written_bytes.value
  • ed_k8s_metric_node_filesystem_avail_bytes.value
  • ed_k8s_metric_node_filesystem_free_bytes.value
  • ed_k8s_metric_node_filesystem_size_bytes.value
  • ed_k8s_metric_node_memory_mem_available_bytes.value
  • ed_k8s_metric_node_memory_mem_free_bytes.value
  • ed_k8s_metric_node_memory_mem_total_bytes.value
  • ed_k8s_metric_node_network_receive_bytes.value
  • ed_k8s_metric_node_network_receive_drop.value
  • ed_k8s_metric_node_network_receive_errs.value
  • ed_k8s_metric_node_network_transmit_bytes.value
  • ed_k8s_metric_node_network_transmit_drop.value
  • ed_k8s_metric_node_network_transmit_errs.value

Example Configuration

nodes:
  - name: my_k8s_metrics
    type: ed_k8s_metrics_input
    exclude:
      - node_exporter  
    scrape_interval: 1m

Example Output Metric

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x",  
    "instance_id":"i-1234567890abcdef0",   
    "instance_name":"test-name",   
  },
  "_stat_type": "count",  
  "name": "error.count",
  "kind": "sum",
  "sum": {
    "is_monotonic": false,
    "value": 10
  }
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ed_k8s_metrics_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Optional Parameters

exclude

The exclude parameter is used to specify which metrics not to scrape. At least one metric must be scraped. By default, the node scrapes the following:

  • cadvisor
  • node_exporter
  • kubelet
  • kube_state_metrics

It is specified as text array and it is optional.

nodes:
  - name: <node name>
    type: ed_k8s_metrics_input
    exclude:
      - node_exporter

scrape_interval

The scrape_interval parameter is used to specify the interval at which the Edge Delta agent scrapes metric endpoints. It is specified as a duration and it is optional. The default is 1m.

nodes:
  - name: <node name>
    type: ed_k8s_metrics_input
    scrape_interval: 1m