Trigger a Metric Threshold with Edge Delta
3 minute read
Overview
The Threshold node in Edge Delta serves a critical role in activating alerts based on specific conditions met within the metrics derived from log data. By defining set thresholds, this node triggers when metrics exceed or fall below the given parameters, ensuring that teams are promptly alerted to potential issues that require attention. In the context of an observability pipeline monitoring web server health, a Threshold node can be integral in detecting abnormal error rates, resource usage spikes, or performance anomalies.
The scenario continues from the log to metric how-to. Imagine an e-commerce platform where it’s vital to maintain an optimal user experience. The Threshold node could be configured to watch for an unusual increase in HTTP 5xx errors within the NGINX server logs. When the error count per minute surpasses a pre-defined threshold, the node fires an alert to signify immediate investigation is needed. This capability is essential for maintaining service stability and swiftly resolving issues that could otherwise lead to user dissatisfaction or revenue loss.
In this scenario you have configured a Log to Metric node to count 5xx logs. After considering key performance indicators and service level agreements, you want to configure a Threshold node to alert on an elevated count of NGINX 5xx errors—indicative of server-side issues. The pipeline handles 2300 NGINX logs per minute. If the node encounters 23 or more 5xx errors per minute (1% of all NGINX logs), a signal should be sent to a Webhook destination node. The webhook destination node, in turn, dispatches a custom alert to a notification application, prompting the team to act swiftly.
This is a sample of the generated logs:
Prerequisites
To use trigger alerts you need an Edge Delta account with a Pipeline configuration already created. This is the configuration in which you will create the alert configuration. You have configured a Log to Metric node to count 5xx logs You also need a webhook URL for a notification application such as Slack, Microsoft Teams, Discord, PagerDuty or Zapier to name a few.
Configure the Threshold Node
Create a Threshold Node that evaluates the metrics generated by the count_5xx log to metric node and issues an alert if the 5xx_per_minute.count metric is 23 or higher.
- Click Edit mode
- Click Add Processor.
- Expand Analytics and select Threshold.
- Enter
23 5xx per minuteor another suitable name. - Enter the following CEL macro as the filter:
item.name == "5xx_per_minute.count"
- Enter the following Condition:
value > 23
- Click Save Changes.
- Connect the count_5xx Log to Metric node’s output to the 23 5xx per minute Threshold node’s input.
Next, you configure a webhook destination to consume the threshold alert.