Monitor Notifications

Configure notifications for monitors in the Edge Delta web application.

Overview

You configure the Notifications section when setting up monitors to specify the recipients and to customize the content of monitor state change notifications.

You can create customized notification messages using the standard markdown format. This allows you to:

  • Provide additional context
  • Include troubleshooting guidelines
  • Offer resolution steps for specific issues

Note: Some markdown elements may not be rendered as expected depending on whether the receiving application supports them. For example, Slack currently does not fully support heading elements.

General Usage

By default, all top-level recipients added to the Notifications section will receive the notification, including the following information:

  • Monitor name
  • Summary message about the scope
  • For some monitor types, Charts illustrating the monitor state change that triggered the monitor.

For monitors defined with a group-by configuration, the summary message also includes information about the affected group.

Recipients

You can have the monitor email a specific user with the notify function. It can also deliver notification to a trigger-type legacy integration (Slack, Webhook, Email or PagerDuty). Autocomplete can be used to select a user or integration by entering @. Autocomplete lists users registered for your organization as well as configured destination integrations.

Specifying recipients at top level allows you to deliver notification for all monitor state changes.

The following example is a notification mail for a metric threshold monitor. It was configured with only the recipients.

Advanced Configurations

The Notifications section offers flexibility for advanced use cases such as:

  • Different notification content for specific recipients
  • Tailored messages based on certain states
  • Custom alerts for specific monitor groups where monitor state change is triggered
  • Any combination of above

These advanced configurations use conditional formatting variables for precise and relevant communication.

Monitor Variables

You can reference monitor group attributes directly when you define a monitor with a group-by configuration:

{{host.name}}

Conditional Formatting Variables

You can use Boolean logic to display parts of the customised message.

{{#var}} this text will show {{/var}} if the variable var is true. {{^var}} this text will show {{/var}} if the variable var is not true.

Also any recipients specified within conditional variable tags receive the notification if the variable is true.

Available Condiitional Formatting Variables

You can select from the following conditions:

Opening Tag Description Closing Tag
{{#is_alert}} This section renders when the monitor detects a condition that meets the alert threshold. {{/is_alert}}
{{^is_alert}} This section renders when the monitor does not detect any condition that meets the alert threshold. {{/is_alert}}
{{#is_warning}} This section renders when the monitor detects a condition that meets the warning threshold but not the alert threshold. {{/is_warning}}
{{^is_warning}} This section renders when the monitor does not detect any condition that meets the warning threshold. {{/is_warning}}
{{#is_recovery}} This section renders when a previously triggered alert or warning condition has returned to normal (recovered). {{/is_recovery}}
{{^is_recovery}} This section renders when an alert or warning condition has not recovered and is still active. {{/is_recovery}}
{{#is_renotify}} This section renders when the monitor is set to re-notify about a persisting alert or warning condition. This trigger is configured in the Renotification section. {{/is_renotify}}
{{^is_renotify}} This section renders when the trigger is not a re-notify and initial notification about a new alert or warning condition. {{/is_renotify}}
{{#is_exact_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name matches any of the matched_value(s) exactly. Multiple values can be separated by commas. {{/is_exact_match}}
{{#is_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name contains any of the matched_value(s) as a substring. Multiple values can be separated by commas. {{/is_match}}
{{^is_exact_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name does not match any of the matched_value(s) exactly. {{/is_exact_match}}
{{^is_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name does not contain any of the matched_value(s) as a substring. {{/is_match}}

Variable Autocomplete

The notification pane uses autocomplete so you can explore the conditions available. To get started, open a condition with a double curly brace {{.

When you select a condition from autocomplete, it adds opening and closing tags for you. Within them, you specify the actions to take if that condition is true.

Renotification

To be reminded about persistent alerts you can set a renotification period. After the first notification the monitor will wait for the duration configured. If after that duration the monitor is still in the alert state, another notification will be sent.

Example Notifications

Only Send Alert Notifications

{{#is_alert}}
  **ALERT**: An alert notification has been triggered for your system.
  - Check https://acme.com dashboard
  - Run playbook-123
  @monitor@team.com
{{/is_alert}}

{{#is_alert}}...{{/is_alert}} block is executed only if monitor state is changed to alert and notification is delivered to monitor@team.com mail address.

Only Send Alert Notifications for a specific resource group

{{#is_alert}}
  {{#is_exact_match "k8s.namespace.name" "production"}}
    **ALERT**: An alert notification has been triggered for your system.
    The issue is within the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
    @monitor@team.com
  {{/is_exact_match}}
{{/is_alert}}
  • {{#is_exact_match "k8s.namespace.name" "production"}}: Within the {{#is_alert}} block, this section further filters alerts to those that are specific to the production namespace. If the namespace is production, the following message is sent: The issue is within the production namespace, with average error or warn logs per 3-hour window exceeding 50.
  • {{k8s.namespace.name}} variable will be also replaced for the originating monitor group. This requires a monitor query statement with group by set to k8s.namespace.name.

Send warning notifications to different recipients based on resource groups

{{#is_warning}}
  {{#is_exact_match "k8s.namespace.name" "production"}}
    @sre@team.com
    @oncall@team.com
    The issue is within the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
  {{/is_exact_match}}

  {{^is_exact_match "k8s.namespace.name" "production"}}
    @developers@team.com
    An issue detected in the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
  {{/is_exact_match}}
{{/is_warning}}

{{#is_alert}}
    @sre@team.com
    @oncall@team.com
    @developers@team.com
{{/is_alert}}
  • This configuration sends a warning notification to sre@team.com and oncall@team.com only if k8s.namespace.name is production.
  • {{#is_exact_match "k8s.namespace.name" "production"}} ensures warnings for all other namespaces go to developers@team.com.
  • Alerts go to both teams with the default body.