Send Events from Edge Delta Monitors to PagerDuty

Send events to PagerDuty.

  3 minute read  

Overview

You can create a webhook or API integration, which can be triggered by a monitor in the backend, and which does not need to be deployed in a particular pipeline like a webhook node.

Pagerduty

In this example, a PagerDuty integration is triggered to send an event to a PagerDuty API.

To create an integration:

  1. Click Admin.
  2. Click Legacy Integrations.
  3. Select Available.
  4. Search for and select the PagerDuty destination.

To receive events from Edge Delta you need to create an Events API v2 integration in PagerDuty for your specific PagerDuty service. During this process you are provided with the endpoint (probably https://events.pagerduty.com/v2/enqueue) and an integration key.

The key must be included as a routing_key in the event payload, and a custom header Content-Type with the value application/json must be configured:

Payload Example 1

The payload in this example illustrates many event variables: Integration name: pagerduty

{
  "payload": {
    "summary": "Example alert",
    "id": "$EVENT_ID",
    "eventType": "$EVENT_TYPE",
    "eventURL": "$EVENT_URL",
    "eventTime": "$EVENT_DATE",
    "eventQuery": "$EVENT_QUERY",
    "eventMetric": "$EVENT_METRIC",
    "eventMessage": "$EVENT_MSG",
    "eventStatus": "$EVENT_STATUS",
    "OrgID": "$EVENT_ORG_ID",
    "eventGroupALL": "$EVENT_GROUP_ALL",
    "eventSpecificGroup": "$EVENT_GROUP.k8s.namespace.name",
    "eventTitle": "$EVENT_TITLE",
    "eventValue": "$EVENT_EVALUATED_VALUE",
    "source": "Edge Delta Monitor"
  },
  "routing_key": "<routing_key>",
  "dedup_key": "samplekeyhere",
  "images": [
    {
      "src": "https://www.pagerduty.com/wp-content/uploads/2016/05/pagerduty-logo-green.png",
      "href": "https://example.com/",
      "alt": "Example text"
    }
  ],
  "links": [
    {
      "href": "$EVENT_URL",
      "text": "Link text"
    }
  ],
  "event_action": "trigger",
  "client": "Edge Delta",
  "client_url": "https://edgedelta.com"
}

Replace <routing_key> with the integration key provided by PagerDuty.

Payload Example 2

These examples uses more static values. There is one integration for alerts and another for alert resolutions. Static values assume the scope of the monitor is more restricted to, for example, errors.

Upon alert for incoming lines: Integration name: incoming-lines

{
  "payload": {
      "summary": "Incoming sum for fleet $EVENT_GROUP.ed.tag is 0",
      "severity": "error",
      "source": "Edge Delta Monitor",
      "component": "Incoming lines",
      "group": "$EVENT_GROUP.ed.tag"
  },
  "routing_key": "****",
  "dedup_key": "$EVENT_GROUP.ed.tag",
  "event_action": "trigger"
}

Upon resolution: Integration name: incoming-lines-resolve

{
  "payload": {
      "summary": "Incoming sum for fleet $EVENT_GROUP.ed.tag is not 0 anymore",
      "severity": "error",
      "source": "Edge Delta Monitor",
      "component": "Incoming lines",
      "group": "$EVENT_GROUP.ed.tag"
  },
  "routing_key": "****",
  "dedup_key": "$EVENT_GROUP.ed.tag",
  "event_action": "resolve"
}

Event Variables

This configuration will attempt to populate the payload with available event variables from the incoming signal’s metadata. You reference event variables using the $ character:

    "id": "$EVENT_ID",
    "eventType": "$EVENT_TYPE",
    "eventURL": "$EVENT_URL",
    "eventTime": "$EVENT_DATE",
    "eventQuery": "$EVENT_QUERY",
    "eventMetric": "$EVENT_METRIC",
    "eventMessage": "$EVENT_MSG",
    "eventStatus": "$EVENT_STATUS",
    "OrgID": "$EVENT_ORG_ID",
    "eventGroupALL": "$EVENT_GROUP_ALL",
    "eventSpecificGroup": "$EVENT_GROUP.k8s.namespace.name",
    "eventTitle": "$EVENT_TITLE",
    "eventValue": "$EVENT_EVALUATED_VALUE"

Monitor Integration

These integrations can now be triggered by a monitor by including their names, prefixed with @pagerduty- in the monitor Notification configuration:

@pagerduty-<integration-name>

Payload Example 1:

{{#is_alert}}
@pagerduty-pagerduty
{{/is_alert}}

Payload Example 2:

{{#is_alert}}
@pagerduty-incoming-lines
{{/is_alert}}

{{#is_recovery}}
@pagerduty-incoming-lines-resolve
{{/is_recovery}}