Edge Delta Kubernetes Traffic Source

Monitor Kubernetes pods and namespaces.

  4 minute read  

Beta

Overview

You can use the Kubernetes Traffic source node to enable Kubernetes metrics ingestion via eBPF and to specify which Kubernetes namespaces the agent should generate metrics from.

Note: This node is required to enable the service map.

Prerequisites

Metrics have the following prerequisites:

  1. To ingest Kube State Metrics, KSM must be deployed in the same cluster as the Edge Delta agent.
  2. To ingest node-exporter metrics, node-exporter must be deployed in the same cluster as the Edge Delta agent.
  3. All metric processing nodes, such as logs to metrics nodes, must be connected to the Metrics destination node.

The Edge Delta agent uses eBPF to collect network metrics. Therefore, the following Kubernetes environment configuration is required for Kubernetes network metrics and eBPF to work:

  • Linux kernel version 5.8 or later.
  • Linux kernel built with the CONFIG_DEBUG_INFO_BTF=y and CONFIG_DEBUG_INFO_BTF_MODULES=y flags.

To check for the flag:

docker run -it --rm --privileged --pid=host ubuntu nsenter -t 1 -m -u -n -i sh -c 'cat /proc/config.gz | gunzip | grep CONFIG_DEBUG_INFO_BTF'

The output should show CONFIG_DEBUG_INFO_BTF=y and CONFIG_DEBUG_INFO_BTF_MODULES=y.

In the case of minikube, the agent can run on minikube with the Docker driver. Docker must be at least v26.0.0 (Docker Desktop v4.29.0) and it is started as follows:

minikube start --driver docker

If you want to disable eBPF:

  1. Delete the Kubernetes Traffic source node.
  2. Disable the tracer:

Helm Rerun the Helm upgrade command with the --set tracerProps.enabled=false flag.

Kubectl Update and re-apply the kubernetes manifest with the following parameter change:

ED_ENABLE_TRAFFIC_TRACER = "0"

Metrics

  • ed_k8s_traffic_latency.avg
  • ed_k8s_traffic_latency.p95
  • ed_k8s_traffic_communication.count
  • ed_k8s_traffic_in.sum
  • ed_k8s_traffic_out.sum

Example Configuration

nodes:
  - name: ed_k8s_traffic
    type: k8s_traffic_input
    include:
      - "k8s.namespace.name=default"
    exclude:
      - "k8s.namespace.name=^kube-system$"
    other_peer_name_exclude_pattern:
      - ".*"
    pod_network_cidrs:
      - "10.244.0.0/16"
    report_unresolved_ssl_connections: false

Example Output Logs

{
   "_timestamp": 1681481446835,
   "_type": "metric",
   "_name": "ed_k8s_traffic_latency.value",
   "_value": 60.5,
   "_stat_type": "value",
   "client_pod": "unknown",
   "client_namespace": "unknown",
   "client_controller": "unknown",
   "server_pod": "test-pod-0",
   "server_namespace": "default",
   "server_controller": "StatefulSet/test-pod",
   "method": "GET",
   "URI": "/",
   "response_code": "200",
   "_ed": {
      "src_type": "K8sTraffic",
      "tag": "site-env"
      "config_id": "12345678-abcd-abcd-abcd-0123456789ab",
      "host": "site-agent",
      "ip": "10.0.0.0",
      "__group_name": "site-env|stat|site-agent|K8sTraffic|-",
      "__logical_source": "K8sTraffic,-",
      "__short_src_name": "-",
      "__src_name": "site-env|stat|site-agent|K8sTraffic|-",
   }
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: k8s_traffic_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Optional Parameters

exclude and include

The include and exclude parameters are used to filter ingestion of telemetry data from Kubernetes objects. They are specified with Golang regex or full text.

  • If exclude is specified it drops telemetry data from all objects in that namespace that match its Golang regex rules.
  • If neither include nor exclude are specified, then everything from the input is passed.
  • If include is specified as well as exclude, then telemetry data from the namespace matching in include is passed unless it is dropped by exclude.
nodes:
  - name: <node name>
    type: k8s_traffic_input
    exclude:
        - k8s.namespace.name=kube-system

NOTE: Kubernetes metrics data ingested via the k8s_traffic_input node can only be filtered by namespace using k8s.namespace.name. Filtering by any other object is not supported.

other_peer_name_exclude_pattern

The other_peer_name_exclude_pattern parameter is used to resolve the identity of unknown peers in the service map by specifying their domain. It is specified as a string list and is optional.

nodes:
  - name: <node name>
    type: k8s_traffic_input
    other_peer_name_exclude_pattern:
        - "amazonaws.com"

In this example, servers or clients ending in amazonaws.com will have their Fully Qualified Domain Names (FQDN) resolved in the service map, rather than being listed as an Unknown Service. See Service Map for more information.

pod_network_cidrs

The pod_network_cidrs parameter is used to define a range of IP addresses to trace. It is specified as a CIDR and is optional.

nodes:
  - name: ed_k8s_traffic
    type: k8s_traffic_input
    pod_network_cidrs:
      - "10.244.0.0/16"

report_unresolved_ssl_connections

The report_unresolved_ssl_connections parameter configure whether to report unresolved SSL connections. It is specified as a Boolean and the default is false.

nodes:
  - name: ed_k8s_traffic
    type: k8s_traffic_input
    report_unresolved_ssl_connections: false