Monitor Notifications

Overview

You configure the Notifications section when setting up monitors to specify the recipients and to customize the content of monitor state change notifications.

You can create customized notification messages using the standard markdown format. This allows you to:

• Provide additional context
• Include troubleshooting guidelines
• Offer resolution steps for specific issues

Note: Some markdown elements may not be rendered as expected depending on whether the receiving application supports them. For example, Slack currently does not fully support heading elements.

General Usage

By default, all top-level recipients added to the Notifications section will receive the notification, including the following information:

• Monitor name
• Summary message about the scope
• For some monitor types, Charts illustrating the monitor state change that triggered the monitor.

For monitors defined with a group-by configuration, the summary message also includes information about the affected group.

Recipients

You can have the monitor email a specific user with the notify function. It can also deliver notification to a trigger-type legacy integration (Slack, Webhook, Email or PagerDuty). Autocomplete can be used to select a user or integration by entering @. Autocomplete lists users registered for your organization as well as configured destination integrations. See Send Monitor Events to an API for an example integration.

Specifying recipients at top level allows you to deliver notification for all monitor state changes.

The following example is a notification mail for a metric threshold monitor. It was configured with only the recipients.

Notification Options

You can attach a dashboard to the notification message with a particular lookback period. Design the dashboard to reflect the monitor’s query scope to illustrate the events leading up to the event.

You can choose to hide information from the message, such as the receiver handles (other recipients), the monitor query, and any default charts.

The Notify creator if there are no receivers option serves as a safety mechanism. When enabled, if a monitor changes state (such as WARN, RECOVERED, or NO DATA) and no recipients are configured to receive notifications for that specific state, the system automatically sends an email notification to the person who created the monitor. This ensures critical alerts are not missed due to incomplete notification configurations.

For example, if you have wrapped recipient configurations in conditional tags like {{#is_alert}} for Slack notifications, but the monitor triggers a WARN, RECOVERED, or NO DATA state without matching recipients, the creator will receive an email alert. If you’re receiving unexpected notifications as the monitor creator, review your conditional notification rules to ensure all monitor states have appropriate recipients configured, or disable this option if fallback notifications are not desired.

To be reminded about persistent alerts you can set a renotification period. After the first notification the monitor will wait for the duration configured. If after that duration the monitor is still in the alert state, another notification will be sent.

You can click Test Notifications to trigger a test alert using the current configuration.

Advanced Configurations

The Notifications section offers flexibility for advanced use cases such as:

• Different notification content for specific recipients
• Tailored messages based on certain states
• Custom alerts for specific monitor groups where monitor state change is triggered
• Any combination of above

These advanced configurations use conditional formatting variables for precise and relevant communication.

Monitor Variables

You can reference monitor group attributes directly when you define a monitor with a group-by configuration:

{{host.name}}

Other values are also available depending on the monitor type. Use auto-complete to explore them all.

Threshold and State Variables

• alert threshold: {{alert_threshold}}
• warn threshold: {{warn_threshold}}
• comparator: {{comparator}}
• value: {{value}}

Time-Related Variables

• evaluated from: {{evaluated_from}}
• evaluated to: {{evaluated_to}}
• first triggered at: {{first_triggered_at}}
• last triggered at: {{last_triggered_at}}
• triggered duration seconds: {{triggered_duration_sec}}

Context Variables

• organization ID: {{org_id}}
• pipeline name: {{ed.tag}}
• service name: {{service.name}}
• top patterns: {{top_patterns}}

Conditional Formatting Variables

You can use Boolean logic to display parts of the customized message.

{{#var}} this text will show {{/var}} if the variable var is true. {{^var}} this text will show {{/var}} if the variable var is not true.

Also any recipients specified within conditional variable tags receive the notification if the variable is true.

Available Conditional Formatting Variables

You can select from the following conditions:

Variable Autocomplete

The notification pane uses autocomplete so you can explore the conditions available. To get started, open a condition with a double curly brace {{.

When you select a condition from autocomplete, it adds opening and closing tags for you. Within them, you specify the actions to take if that condition is true.

Comments

You can add comments to the notification section. These are not rendered in the message:

{{! This is a comment }}

<!-- This is also a comment. -->

Example Notifications

Only Send Alert Notifications

{{#is_alert}}
  **ALERT**: An alert notification has been triggered for your system.
  - Check https://acme.com dashboard
  - Run playbook-123
  @monitor@team.com
{{/is_alert}}

{{#is_alert}}...{{/is_alert}} block is executed only if monitor state is changed to alert and notification is delivered to monitor@team.com mail address.

Handle No Data Conditions

{{#is_no_data}}
  **NO DATA**: Monitor {{monitor.name}} has stopped receiving telemetry.
  - Last data received: {{last_data_timestamp}}
  - Check agent connectivity
  - Verify data source is operational
  @oncall@team.com
{{/is_no_data}}

{{#is_no_data_recovery}}
  **RECOVERED**: Monitor {{monitor.name}} is receiving data again.
  - Data flow restored at: {{recovery_timestamp}}
  @devops@team.com
{{/is_no_data_recovery}}

Use {{#is_no_data}}...{{/is_no_data}} to alert when telemetry stops flowing and {{#is_no_data_recovery}}...{{/is_no_data_recovery}} when data resumes.

Handle State Transitions

{{#is_alert_to_warning}}
  **DE-ESCALATION**: Monitor {{monitor.name}} improved from ALERT to WARNING
  - Current value: {{value}}
  - Alert threshold: {{alert_threshold}}
  - Warning threshold: {{warn_threshold}}
  @sre@team.com
{{/is_alert_to_warning}}

{{#is_alert_recovery}}
  **RESOLVED**: Alert condition cleared for {{monitor.name}}
  - Duration: {{triggered_duration_sec}} seconds
  - Evaluated period: {{evaluated_from}} to {{evaluated_to}}
  @sre@team.com
{{/is_alert_recovery}}

{{#is_warning_recovery}}
  **CLEARED**: Warning condition resolved for {{monitor.name}}
  - Back to normal at: {{last_triggered_at}}
  @devops@team.com
{{/is_warning_recovery}}

These conditions help track specific state transitions, providing more granular control over notifications.

Only Send Alert Notifications for a specific resource group

{{#is_alert}}
  {{#is_exact_match "k8s.namespace.name" "production"}}
    **ALERT**: An alert notification has been triggered for your system.
    The issue is within the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
    @monitor@team.com
  {{/is_exact_match}}
{{/is_alert}}

• {{#is_exact_match "k8s.namespace.name" "production"}}: Within the {{#is_alert}} block, this section further filters alerts to those that are specific to the production namespace. If the namespace is production, the following message is sent: The issue is within the production namespace, with average error or warn logs per 3-hour window exceeding 50.
• {{k8s.namespace.name}} variable will be also replaced for the originating monitor group. This requires a monitor query statement with group by set to k8s.namespace.name.

Send warning notifications to different recipients based on resource groups

{{#is_warning}}
  {{#is_exact_match "k8s.namespace.name" "production"}}
    @sre@team.com
    @oncall@team.com
    The issue is within the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
  {{/is_exact_match}}

  {{^is_exact_match "k8s.namespace.name" "production"}}
    @developers@team.com
    An issue detected in the {{k8s.namespace.name}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
  {{/is_exact_match}}
{{/is_warning}}

{{#is_alert}}
    @sre@team.com
    @oncall@team.com
    @developers@team.com
{{/is_alert}}

• This configuration sends a warning notification to sre@team.com and oncall@team.com only if k8s.namespace.name is production.
• {{#is_exact_match "k8s.namespace.name" "production"}} ensures warnings for all other namespaces go to developers@team.com.
• Alerts go to both teams with the default body.