Use Edge Delta to Ingest from Prometheus
6 minute read
Overview
The Prometheus Scraper input node can scrape metrics from any target that supports the OpenMetrics format. It functions as a drop-in replacement: you can migrate from Prometheus to the ED Agent using the same configuration.
There are three different ways to implement Edge Delta prometheus scraping:
Method 1: Target Allocator with Dynamic Configuration
This method uses a Target Allocator component to communicate with Edge Delta Agents. Its function is to discover Prometheus Custom Resources and distribute Prometheus targets among the pool of Edge Delta Agent Scrapers.
The target allocator is configured using a values YAML file that is passed in when the Edge Delta agent is installed.
Enable the Prometheus Custom Resource
Configure a YAML file (named values.yaml in this example) to enable the Prometheus Custom Resource:
targetAllocator:
configuration:
prometheus_cr:
enabled: true
pod_monitor_selector:
matchLabels: {}
matchExpressions: []
scrape_interval: 30s
service_monitor_selector:
matchLabels: {}
matchExpressions: []
You can alter the pod_monitor_selector and service_monitor_selector to define the scope of metrics targets to scrape, for example
service_monitor_selector:
matchlabels: {}
matchexpressions:
- {key: app.kubernetes.io/component, operator: In, values: [application-controller]}
Configure the Prometheus Input Node:
In the Edge Delta pipeline, add a Prometheus input node and connect it to the Edge Delta destination. In this method, the node takes no parameters other than name and type:
Note: While this node is experimental, you can only add it using the YAML editor.
- name: my_prometheus_input
type: prometheus_input
Deploy an Edge Delta Agent with a Target Allocator
To include the target allocator in the Edge Delta installation, along with it’s dynamic values yaml, include --set targetAllocator.enabled=true to set the targetAllocator value in the Helm command and pass in the values.yaml file with -f values.yaml.
Method 2: Target Allocator with Static Configuration
Rather than a dynamic scraper configuration, you can specify a static scrape configuration by including it in the values.yaml file within a configuration parameter as follows:
targetAllocator:
configuration:
config:
scrape_configs:
# if you want to scrape TA metrics also otherwise you can drop "ed-agent-collector" job
- job_name: ed-agent-collector
scrape_interval: 10s
static_configs:
- targets:
- 0.0.0.0:8888
- job_name: "myapp"
scrape_interval: 10s
static_configs:
- targets: ["myapp-application-controller-metrics.myapp.svc.cluster.local:8082"]
metrics_path: "/metrics"
scheme: "http"
- job_name: serviceMonitor/myapp/myapp-application-controller/0
honor_timestamps: true
track_timestamps_staleness: false
scrape_interval: 30s
scrape_timeout: 10s
scrape_protocols:
- OpenMetricsText1.0.0
- OpenMetricsText0.0.1
- PrometheusText0.0.4
metrics_path: /metrics
scheme: http
enable_compression: true
follow_redirects: true
enable_http2: true
relabel_configs:
- source_labels: [job]
separator: ;
regex: (.*)
target_label: __tmp_prometheus_job_name
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_component, __meta_kubernetes_service_labelpresent_app_kubernetes_io_component]
separator: ;
regex: (application-controller);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (myapp);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (myapp-metrics);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: http-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_phase]
separator: ;
regex: (Failed|Succeeded)
replacement: $1
action: drop
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: http-metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
enable_http2: true
namespaces:
own_namespace: false
names:
- myapp
Configure the Prometheus Input Node:
In the Edge Delta pipeline, add a Prometheus input node and connect it to the Edge Delta destination. In this method, the node takes no parameters other than name and type:
Note: While this node is experimental, you can only add it using the YAML editor.
- name: my_prometheus_input
type: prometheus_input
Deploy an Edge Delta Agent with a Target Allocator
To include the target allocator in the Edge Delta installation, along with it’s static values yaml, include --set targetAllocator.enabled=true to set the targetAllocator value in the Helm command and pass in the values.yaml file with -f values.yaml.
Method 3: No Target Allocator with Static Configuration
In this example, you configure the static scrape configuration in the Prometheus input node using the config parameter. In this scenario a target allocator is not created during agent installation. This would normally result in duplicate metrics being scraped as the Edge Delta processor agents would all scrape the same metric sources. To avoid this issue, you must enable the should_only_leader_ingest parameter in the node configuration. Bear in mind you lose the performance benefits of a target allocator sharing the scraping load between agents in a high volume environment.
Note: While this node is experimental, you can only add it using the YAML editor.
- name: my_prometheus_input
type: prometheus_input
should_only_leader_ingest: true
config: |
global:
scrape_interval: 30s
scrape_configs:
- job_name: "myapp"
scrape_interval: 10s
static_configs:
- targets: ["myapp-application-controller-metrics.myapp.svc.cluster.local:8082"]
metrics_path: "/metrics"
scheme: "http"
- job_name: serviceMonitor/myapp/myapp-application-controller/0
honor_timestamps: true
track_timestamps_staleness: false
scrape_interval: 30s
scrape_timeout: 10s
scrape_protocols:
- OpenMetricsText1.0.0
- OpenMetricsText0.0.1
- PrometheusText0.0.4
metrics_path: /metrics
scheme: http
enable_compression: true
follow_redirects: true
enable_http2: true
relabel_configs:
- source_labels: [job]
separator: ;
regex: (.*)
target_label: __tmp_prometheus_job_name
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_component, __meta_kubernetes_service_labelpresent_app_kubernetes_io_component]
separator: ;
regex: (application-controller);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
separator: ;
regex: (myapp);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
separator: ;
regex: (myapp-metrics);true
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_port_name]
separator: ;
regex: http-metrics
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Node;(.*)
target_label: node
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
separator: ;
regex: Pod;(.*)
target_label: pod
replacement: ${1}
action: replace
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: service
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name]
separator: ;
regex: (.*)
target_label: pod
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_container_name]
separator: ;
regex: (.*)
target_label: container
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_phase]
separator: ;
regex: (Failed|Succeeded)
replacement: $1
action: drop
- source_labels: [__meta_kubernetes_service_name]
separator: ;
regex: (.*)
target_label: job
replacement: ${1}
action: replace
- separator: ;
regex: (.*)
target_label: endpoint
replacement: http-metrics
action: replace
- source_labels: [__address__]
separator: ;
regex: (.*)
modulus: 1
target_label: __tmp_hash
replacement: $1
action: hashmod
- source_labels: [__tmp_hash]
separator: ;
regex: "0"
replacement: $1
action: keep
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
enable_http2: true
namespaces:
own_namespace: false
names:
- myapp
Deploy an Edge Delta Agent
With this method, no custom components are included so you install the agent normally.