Monitor Notifications

Overview

You configure the Notifications section when setting up monitors to specify the recipients and to customize the content of monitor state change notifications.

You can create customized notification messages using the standard markdown format. This allows you to:

• Provide additional context
• Include troubleshooting guidelines
• Offer resolution steps for specific issues

Note: Some markdown elements may not be rendered as expected depending on whether the receiving application supports them. For example, Slack currently does not fully support heading elements.

General Usage

By default, all top-level recipients added to the Notifications section will receive the notification, including the following information:

• Monitor name
• Summary message about the scope
• For some monitor types, Charts illustrating the monitor state change that triggered the monitor.

For monitors defined with a group-by configuration, the summary message also includes information about the affected group.

Recipients

You can have the monitor email a specific user with the notify function. It can also deliver notification to a trigger-type legacy integration (Slack, Webhook, Email or PagerDuty). Autocomplete can be used to select a user or integration by entering @. Autocomplete lists users registered for your organization as well as configured destination integrations. You can click Test Notifications to trigger a test alert using the current configuration.

Specifying recipients at top level allows you to deliver notification for all monitor state changes.

The following example is a notification mail for a metric threshold monitor. It was configured with only the recipients.

Advanced Configurations

The Notifications section offers flexibility for advanced use cases such as:

• Different notification content for specific recipients
• Tailored messages based on certain states
• Custom alerts for specific monitor groups where monitor state change is triggered
• Any combination of above

These advanced configurations use conditional formatting variables for precise and relevant communication.

Monitor Variables

You can reference monitor group attributes directly when you define a monitor with a group-by configuration:

{{host.name}}

Conditional Formatting Variables

You can use Boolean logic to display parts of the customized message.

{{#var}} this text will show {{/var}} if the variable var is true. {{^var}} this text will show {{/var}} if the variable var is not true.

Also any recipients specified within conditional variable tags receive the notification if the variable is true.

Available Condiitional Formatting Variables

You can select from the following conditions:

Variable Autocomplete

The notification pane uses autocomplete so you can explore the conditions available. To get started, open a condition with a double curly brace {{.

When you select a condition from autocomplete, it adds opening and closing tags for you. Within them, you specify the actions to take if that condition is true.

Renotification

To be reminded about persistent alerts you can set a renotification period. After the first notification the monitor will wait for the duration configured. If after that duration the monitor is still in the alert state, another notification will be sent.

Example Notifications

Only Send Alert Notifications

{{#is_alert}}
  **ALERT**: An alert notification has been triggered for your system.
  - Check https://acme.com dashboard
  - Run playbook-123
  @monitor@team.com
{{/is_alert}}

{{#is_alert}}...{{/is_alert}} block is executed only if monitor state is changed to alert and notification is delivered to monitor@team.com mail address.

Only Send Alert Notifications for a specific resource group

{{#is_alert}}
  {{#is_exact_match "k8s.namespace.name" "production"}}
    **ALERT**: An alert notification has been triggered for your system.
    The issue is within the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
    @monitor@team.com
  {{/is_exact_match}}
{{/is_alert}}

• {{#is_exact_match "k8s.namespace.name" "production"}}: Within the {{#is_alert}} block, this section further filters alerts to those that are specific to the production namespace. If the namespace is production, the following message is sent: The issue is within the production namespace, with average error or warn logs per 3-hour window exceeding 50.
• {{k8s.namespace.name}} variable will be also replaced for the originating monitor group. This requires a monitor query statement with group by set to k8s.namespace.name.

Send warning notifications to different recipients based on resource groups

{{#is_warning}}
  {{#is_exact_match "k8s.namespace.name" "production"}}
    @sre@team.com
    @oncall@team.com
    The issue is within the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
  {{/is_exact_match}}

  {{^is_exact_match "k8s.namespace.name" "production"}}
    @developers@team.com
    An issue detected in the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
  {{/is_exact_match}}
{{/is_warning}}

{{#is_alert}}
    @sre@team.com
    @oncall@team.com
    @developers@team.com
{{/is_alert}}

• This configuration sends a warning notification to sre@team.com and oncall@team.com only if k8s.namespace.name is production.
• {{#is_exact_match "k8s.namespace.name" "production"}} ensures warnings for all other namespaces go to developers@team.com.
• Alerts go to both teams with the default body.