Monitor Notifications

Configure notifications for monitors in the Edge Delta web application.

Overview

When configuring metric, log or pattern anomaly monitors, you create a message using a combination of markdown and a simple conditional logic language.

Syntax

Syntax: {{var}}

You can use Boolean logic:

{{#var}} this text will show {{/var}} if the variable var is true. {{^var}} this text will show {{/var}} if the variable var is not true.

You can also reference event attributes directly:

{{host.name}}

Variables

You can select from the following conditions:

Opening Tag Description Closing Tag
{{#is_alert}} This section renders when the monitor detects a condition that meets the alert threshold. {{/is_alert}}
{{^is_alert}} This section renders when the monitor does not detect any condition that meets the alert threshold. {{/is_alert}}
{{#is_warning}} This section renders when the monitor detects a condition that meets the warning threshold but not the alert threshold. {{/is_warning}}
{{^is_warning}} This section renders when the monitor does not detect any condition that meets the warning threshold. {{/is_warning}}
{{#is_recovery}} This section renders when a previously triggered alert or warning condition has returned to normal (recovered). This trigger is configured in the Renotification section. {{/is_recovery}}
{{^is_recovery}} This section renders when an alert or warning condition has not recovered and is still active. This trigger is configured in the Renotification section. {{/is_recovery}}
{{#is_renotify}} This section renders when the monitor is set to re-notify about a persisting alert or warning condition. This trigger is configured in the Renotification section. {{/is_renotify}}
{{^is_renotify}} This section renders when the trigger is re-fired rather than a re-notify about a persisting alert or warning condition. This trigger is configured in the Renotification section. {{/is_renotify}}
{{#is_exact_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name matches matched_value(s) exactly. Multiple values can be separated by commas. {{/is_exact_match}}
{{#is_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name contains any of the matched_value(s) as a substring. Multiple values can be separated by commas. {{/is_match}}
{{^is_exact_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name does not match any of the matched_value(s) exactly. {{/is_exact_match}}
{{^is_match "variable_name" "matched_value(s)"}} This section renders when the value of variable_name does not contain any of the matched_value(s) as a substring. {{/is_match}}

Autocomplete

The notification pane uses autocomplete so you can explore the conditions available. To get started, open a condition with a double curly brace {{.

When you select a condition from autocomplete, it adds opening and closing tags for you. Within them you specify the actions to take if that condition is true.

Recipients

You can have the monitor email a specific user with the notify function. It can also trigger a trigger-type legacy integration (Slack, Webhook or PagerDuty). Autocomplete can be used to select a user or integration by entering @. Autocomplete lists users registered for your organization as well as configured destination integrations.

Renotification

To prevent alerts from flooding recipients you can set a renotification period. After the first notification the monitor will wait for the duration configured. If after that duration the monitor is still in the alert state, another notification will be sent.

Example Notification

{{#is_alert}}
  **ALERT**: An alert notification has been triggered for your system.
  @monitor@team.com
  
  {{#is_exact_match "k8s.namespace" "production"}}
    The issue is within the {{k8s.namespace}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
    
    {{#is_warning}}
      - Warning: Average `error` or `warn` logs per 3-hour window are nearing the alert threshold.
    {{/is_warning}}
    
    {{#is_recovery}}
      - Recovery: The average log count has returned below the alert threshold.
    {{/is_recovery}}

    {{^is_recovery}}
      - The system has *not* recovered from the alert condition yet.
    {{/is_recovery}}

  {{/is_exact_match}}

  {{^is_exact_match "k8s.namespace" "production"}}
    An issue detected in the {{k8s.namespace}} namespace, with average `error` or `warn` logs per 3-hour window exceeding 50.
    
    {{#is_warning}}
      - Warning: Average log count per 3-hour window is approaching the alert threshold.
    {{/is_warning}}

  {{/is_exact_match}}

  {{#is_renotify}}
    - Reminder: The alert condition persists.
  {{/is_renotify}}
{{/is_alert}}

{{^is_alert}}
  **INFO**: There are currently no alerts.
  
  {{#is_warning}}
    - Warning: Average `error` or `warn` logs per 3-hour window are nearing the alert threshold in some namespaces.
  {{/is_warning}}

  {{^is_warning}}
    No alerts or warnings at this time.
  {{/is_warning}}
{{/is_alert}}

Example Walkthrough

{{#is_alert}}...{{/is_alert}} Block:

This top-level block is executed only when the monitor identifies an alert condition.

  • When there is an alert, the notification is sent to the user: monitor@team.com
    • {{#is_exact_match "k8s.namespace" "production"}}: Within the {{#is_alert}} block, this section further filters alerts to those that are specific to the production namespace. If the namespace is production, the following message is sent: The issue is within the production namespace, with average errororwarn logs per 3-hour window exceeding 50.
      • Nested Check for Warnings: This condition checks if the alert also has a warning status. If it does, it sends this message: Warning: Average errororwarn logs per 3-hour window are nearing the alert threshold.
      • Nested Check for Recovery: This condition handles whether the system has recovered from an alert or warning. If recovered, it sends this message: Recovery: The average log count has returned below the alert threshold.
      • If not recovered, this message is sent: The system has *not* recovered from the alert condition yet.
    • {{^is_exact_match "k8s.namespace" "production"}} Block: This section is nested within {{#is_alert}}. This block handles alert scenarios where the namespace is anything other than production. If the namespace is not production, this message is sent: An issue detected in the {{k8s.namespace}} namespace, with average error or warn logs per 3-hour window exceeding 50. The namespace is dynamically populated.
      • Nested Check for Warnings: Similar to the previous section but scoped to non-production namespaces: Warning: Average log count per 3-hour window is approaching the alert threshold.
    • {{#is_renotify}}...{{/is_renotify}} Block: This section is nested within {{#is_alert}} and only executes if the alert is a renotify event. This typically happens in scenarios where ongoing issues need repeated notifications: Reminder: The alert condition persists.

{{^is_alert}}...{{/is_alert}} Block:

This block is executed when there is no alert condition. It helps provide information on the system status when everything is running normally or only warnings are active:

  • When there is no alert, This message is sent: There are currently no alerts.
    • Nested Check for Warnings: if there’s a non-critical warning, this message is sent: Warning: Average errororwarn logs per 3-hour window are nearing the alert threshold in some namespaces.
    • If there are no warnings: this message is sent: No alerts or warnings at this time.