Edge Delta Prometheus Output

Overview

The Edge Delta Prometheus Output Node collects metrics and exposes them on a particular port for Prometheus to scrape. You can pipe any metrics in Edge Delta to the prometheus output node to expose them to prometheus. If you pipe metrics to both the ed_metrics node and prometheus, the metrics will appear in both locations: Prometheus and the Metrics Explorer.

Metrics

In this example pipeline, all the default and hidden metrics emitting nodes are connected to the prometheus output node. The following metrics will be scraped:

• edgedelta_agent_cpu_millicores_value from ed_agent_stats_input.
• edgedelta_agent_mem_alloc_value from ed_agent_stats_input.
• edgedelta_agent_mem_to_be_freed_value from ed_agent_stats_input.
• edgedelta_agent_mem_virtual_value from ed_agent_stats_input.
• edgedelta_agent_num_goroutine_value from ed_agent_stats_input.
• edgedelta_error_monitoring_anomaly1 from the error_monitoring log to metric node.
• edgedelta_error_monitoring_count from the error_monitoring log to metric node.
• edgedelta_incoming_bytes_sum from the ed_pipeline_io_stats_input node.
• edgedelta_incoming_lines_count from the ed_pipeline_io_stats_input node.
• edgedelta_negative_sentiment_monitoring_anomaly1 from the negative_sentiment log to metric node.
• edgedelta_negative_sentiment_monitoring_count from the negative_sentiment log to metric node.
• edgedelta_outgoing_bytes_sum from the ed_pipeline_io_stats_input node.
• edgedelta_outgoing_lines_count from the ed_pipeline_io_stats_input node.
• edgedelta_outgoing_raw_bytes_sum from the ed_pipeline_io_stats_input node.
• edgedelta_outgoing_uncompressed_bytes_sum from the ed_pipeline_io_stats_input node.

Note: Click Show Diagnostic Nodes to view all hidden nodes in the pipeline.

Example Configuration

nodes:
- name: prometheus_exporter_output
  type: prometheus_exporter_output
  port: 8087

Configuring Prometheus

A ServiceMonitor custom resource extends service discovery capability specifically for Kubernetes services. When you configure a ServiceMonitor, you define how Prometheus should discover the services from which it should collect metrics. This includes specifying criteria such as the namespace where the service exists and the labels that the service must have for it to be included in the discovery process.

When the Prometheus Operator sees a ServiceMonitor resource with these configurations, it alters the Prometheus instance’s configuration so that Prometheus starts scraping metrics from the services that match these criteria. This allows for automatic updating of the scraping targets as new services are deployed or existing services are updated within the Kubernetes cluster.

You need to pass in a values file when installing the Edge Delta agent to create a ServiceMonitor:

promPort: 8087
serviceMonitor:
  enabled: true
  namespace: monitoring
  scrapeTimeout: 10s
  jobLabel: edgedelta
  selector:
    release: prom-kube-stack

• promPort: 8087 - This sets a specific port (8087) for Prometheus to collect metrics from Edge Delta. Prometheus will scrape the metrics exposed on this port. It must match the port specified in the Prometheus Exporter pipeline configuration.
• serviceMonitor: This section contains configurations for a ServiceMonitor, which is a custom resource defined by the Prometheus operator used in Kubernetes environments. This operator makes the Prometheus configuration easier and more adaptable to dynamic cluster environments.
  • enabled: true - This enables the creation of a ServiceMonitor resource, which means Prometheus is configured to automatically discover services and scrape metrics from them.
  • namespace: monitoring - This defines the namespace where the ServiceMonitor resource will be created. The monitoring namespace is typically used for monitoring related resources in a Kubernetes cluster.
  • scrapeTimeout: 10s - This sets the timeout for a scrape attempt by Prometheus. If the scrape takes longer than 10 seconds, it will be considered as a failure.
  • jobLabel: edgedelta - This label is used to identify the collection of metrics “job” within Prometheus. Jobs in Prometheus configuration define a set of targets (endpoints) to scrape metrics from. By setting this label, Prometheus can differentiate the metrics scraped from Edge Delta vs. other jobs.
  • selector: - This subsection specifies the label selectors required to identify the Service object associated with Edge Delta that Prometheus needs to scrape.
    • release: prom-kube-stack - This indicates that the service to be scraped must have the specified labels. The ServiceMonitor will select the service that has a label release set to prom-kube-stack. If you are using a different Prometheus setup other than the prom-kube-stack, you need to change the release value in this file to match the name of the Prometheus release you are using.

If this file is called values.yaml, add the following flag to the helm installation command:

--values values.yaml 

Note: Instead of passing in a separate values file, you can alternatively append the following parameters to the Helm command:

  --set promPort=8087 \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.namespace=monitoring \
  --set serviceMonitor.scrapeTimeout=10s \
  --set serviceMonitor.jobLabel=edgedelta \
  --set serviceMonitor.selector.release=prom-kube-stack

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: prometheus_exporter_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

port

The port parameter defines the port that Edge Delta will expose a metrics endpoint to for Prometheus. It specifies which port the Edge Delta agent will listen on to expose metrics for Prometheus to scrape. This should match the promPort parameter specified in the Helm chart values file. It is specified as a string and is required for a Prometheus Output.

nodes:
- name: prometheus_exporter_output
  type: prometheus_exporter_output
  port: 8087

Optional Parameters

interval

The interval parameter is used to define how long the exporter should collect metrics per interval. The Prometheus server should be configured to collect metrics at the same interval. It is specified as a duration, the default is 1m and it is optional.

nodes:
- name: prometheus_exporter_output
  type: prometheus_exporter_output
  port: 8087
  interval: 1m

labels

The labels parameter is used to define labels to append to the metrics. This is useful for differentiating between metrics generated by different Edge Delta agents and for differentiating between metrics with capture groups that are saved as metric attributes. Labels are defined with a name for the field, and a path to the source field containing the label value. See here for information on how to reference fields. Labels are optional.

nodes:
- name: prometheus_exporter_output
  type: prometheus_exporter_output
  port: 8087
  labels:
    - name: k8s_namespace
      path: item["resource"]["k8s.namespace.name"]
    - name: k8s_container_name
      path: item["resource"]["k8s.container.name"]
    - name: k8s_pod_name
      path: item["resource"]["k8s.pod.name"]
    - name: k8s_controller_kind
      path: item["resource"]["k8s.cronjob.name"]
    - name: k8s_controller_kind
      path: item["resource"]["k8s.daemonset.name"]
    - name: k8s_controller_kind
      path: item["resource"]["k8s.deployment.name"]
    - name: k8s_controller_kind
      path: item["resource"]["k8s.job.name"]
    - name: k8s_controller_kind
      path: item["resource"]["k8s.replicaset.name"]
    - name: k8s_controller_kind
      path: item["resource"]["k8s.statefulset.name"]
    - name: platform_label
      path: item["attributes"]["platform"]
    - name: platform_label
      path: item["attributes"]["status_code"]

retire_period

The retire_period parameter is used to define how long to wait before retiring unused metric vectors. It is specified as a duration, the default is 15m and it is optional.

nodes:
- name: prometheus_exporter_output
  type: prometheus_exporter_output
  port: 8087
  retire_period: