Send PagerDuty alerts with Sensu

Sensu checks are commands the Sensu agent executes that generate observability data in a status or metric event. Sensu handlers define the actions the Sensu backend executes on the events. Follow this guide to create a check that looks for a specific file and a handler that sends an alert to PagerDuty if the file is not found.

To follow this guide, you’ll need to install the Sensu backend, have at least one Sensu agent running, and install and configure sensuctl. You’ll also need your PagerDuty API integration key to set up the handler in this guide.

Configure a Sensu entity

Every Sensu agent has a defined set of subscriptions that determine which checks the agent will execute. For an agent to execute a specific check, you must specify the same subscription in the agent configuration and the check definition. To run the check_cpu check, you’ll need a Sensu entity with the subscription system.

First, find your entity name:

sensuctl entity list

The ID in the response is the name of your entity.

Replace <entity_name> with the name of your entity in the sensuctl command below. Then run the command to add the system subscription to your entity:

sensuctl entity update <entity_name>

• For Entity Class, press enter.
• For Subscriptions, type system and press enter.

Confirm both Sensu services are running:

systemctl status sensu-backend && systemctl status sensu-agent

The response should indicate active (running) for both the Sensu backend and agent.

Add the file_exists check

Before Sensu can send alerts to your PagerDuty account, you’ll need a check to generate events. In this guide, you will use a file_exists check, which is included in Sensu’s Nagios Foundation dynamic runtime asset.

Before you can add the check, you need to add the Nagios Foundation asset to your Sensu configuration:

sensuctl asset add ncr-devops-platform/nagiosfoundation

To confirm that the asset was added to your Sensu backend, run:

sensuctl asset info ncr-devops-platform/nagiosfoundation

The list should include === ncr-devops-platform/nagiosfoundation followed by a list of available builds for the asset.

Now that you’ve added the Nagios Foundation dynamic runtime asset, you can add its file_exists check to your Sensu backend. Use sensuctl to add the check:

sensuctl check create file_exists \
--command "check_file_exists --pattern /tmp/my-file.txt" \
--subscriptions system \
--handlers pagerduty \
--interval 10 \
--runtime-assets ncr-devops-platform/nagiosfoundation

To confirm that the check was added to your Sensu backend and view the check definition in YAML or JSON format, run:

sensuctl check info file_exists --format yaml

sensuctl check info file_exists --format wrapped-json

The response will list the complete check resource definition:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: file_exists
spec:
  check_hooks: null
  command: check_file_exists --pattern /tmp/my-file.txt
  env_vars: null
  handlers:
  - pagerduty
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - ncr-devops-platform/nagiosfoundation
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "file_exists"
  },
  "spec": {
    "check_hooks": null,
    "command": "check_file_exists --pattern /tmp/my-file.txt",
    "env_vars": null,
    "handlers": [
      "pagerduty"
    ],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "ncr-devops-platform/nagiosfoundation"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

The check command includes the path for the file that the check will look for on your system, /tmp/my-file.txt. For this guide, you’ll add /tmp/my-file.txt as a temporary file:

touch /tmp/my-file.txt

In production, you could replace /tmp/my-file.txt with the file path or globbing pattern for any existing file or files on your system.

Well done! In the next step, you’ll configure your workflow for sending Sensu alerts to your PagerDuty account.

Add the PagerDuty handler

The Sensu PagerDuty Handler dynamic runtime asset includes the scripts you will need to send events to PagerDuty.

To add the PagerDuty handler asset, run:

sensuctl asset add sensu/sensu-pagerduty-handler

Once again, it’s a good idea to confirm that the asset was added to your Sensu backend. Run:

sensuctl asset info sensu/sensu-pagerduty-handler

The response will list the available builds for the PagerDuty handler dynamic runtime asset.

Now that you’ve added the Sensu PagerDuty Handler dynamic runtime asset, you can create a handler that uses the asset to send non-OK events to PagerDuty. This requires you to update the handler command by adding your PagerDuty API key.

In the following command, replace <pagerduty_key> with your PagerDuty API integration key. Then run the updated command:

sensuctl handler create pagerduty \
--type pipe \
--filters is_incident,not_silenced \
--runtime-assets sensu/sensu-pagerduty-handler \
--command "sensu-pagerduty-handler -t <pagerduty_key>"

Make sure that your handler was added by retrieving the complete handler definition in YAML or JSON format:

sensuctl handler info pagerduty --format yaml

sensuctl handler info pagerduty --format wrapped-json

The response will list the complete handler resource definition:

---
type: Handler
api_version: core/v2
metadata:
  name: pagerduty
spec:
  command: sensu-pagerduty-handler -t <pagerduty_key>
  env_vars: null
  filters:
  - is_incident
  - not_silenced
  handlers: null
  runtime_assets:
  - sensu/sensu-pagerduty-handler
  secrets: null
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "pagerduty"
  },
  "spec": {
    "command": "sensu-pagerduty-handler -t <pagerduty_key>",
    "env_vars": null,
    "filters": [
      "is_incident",
      "not_silenced"
    ],
    "handlers": null,
    "runtime_assets": [
      "sensu/sensu-pagerduty-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}

Trigger an event

With your file_exists check and PagerDuty handler workflow in place, you can remove a file to cause Sensu to send a non-OK event.

Remove the file /tmp/my-file.txt:

rm /tmp/my-file.txt

This will make sure the file is not there for Sensu to find the next time the file_exists check runs. After about 10 seconds, Sensu will detect that my-file.txt is missing and reflect that in an event.

To view the event with sensuctl, run:

sensuctl event list

The response should show that the file removal resulted in a CRITICAL (2) event:

     Entity         Check                                      Output                                   Status   Silenced             Timestamp                             UUID                  
 ────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ────────────────────────────────────── 
  host01         file_exists   CheckFileExists CRITICAL - 0 files matched pattern /tmp/my-file.txt           2   false      2021-03-15 19:28:21 +0000 UTC   1b4266ae-7200-4728-a0n4-2f50f7a56613

Open the Sensu web UI to view the events the file_exists check is generating. Visit http://127.0.0.1:3000, and log in as the admin user (created during initialization when you installed the Sensu backend). The failing check’s events will be listed on the Events page.

Observe the alert in PagerDuty

After Sensu detects the non-OK event, the handler you set up will send the alert to PagerDuty. Log in to your PagerDuty account to view an event similar to this one:

Resolve the alert in PagerDuty

To complete your workflow, restore the file that you removed so Sensu sends a resolution to PagerDuty:

touch /tmp/my-file.txt

In your PagerDuty account, the alert should now be listed under the “Resolved” tab.

To view the resolved event with sensuctl, run:

sensuctl event list

The response should show that the file is restored, with an OK (0) status:

     Entity         Check                                      Output                                   Status   Silenced             Timestamp                             UUID                  
 ────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ────────────────────────────────────── 
  host01         file_exists   CheckFileExists OK - 1 files matched pattern /tmp/my-file.txt                 0   false      2021-03-15 19:58:31 +0000 UTC   73f382fe-4290-48e8-955a-6efd54347b43

The web UI Events page will also show that the file_check check is passing.

Next steps

You should now have a working set-up with a file_exists check and a handler that sends alerts to your PagerDuty account. To share and reuse the check and handler like code, save them to files and start building a monitoring as code repository.

You can customize your PagerDuty handler with configuration options like severity mapping, PagerDuty team-based routing and authentication via check and agent annotations, and event-based templating. Learn more about the Sensu PagerDuty integration and our curated, configurable quick-start template for incident management to integrate Sensu with your existing PagerDuty workflows.