Send PagerDuty alerts with Sensu
Sensu checks are commands the Sensu agent executes that generate observability data in a status or metric event. Sensu handlers define the actions the Sensu backend executes on the events. Follow this guide to create a check that looks for a specific file and a handler that sends an alert to PagerDuty if the file is not found.
To follow this guide, you’ll need to install the Sensu backend, have at least one Sensu agent running, and install and configure sensuctl. You’ll also need your PagerDuty API integration key to set up the handler in this guide.
Configure a Sensu entity
Every Sensu agent has a defined set of subscriptions that determine which checks the agent will execute.
For an agent to execute a specific check, you must specify the same subscription in the agent configuration and the check definition.
To run the check_cpu
check, you’ll need a Sensu entity with the subscription system
.
First, find your entity name:
sensuctl entity list
The ID
in the response is the name of your entity.
Replace <entity_name>
with the name of your entity in the sensuctl command below.
Then run the command to add the system
subscription to your entity:
sensuctl entity update <entity_name>
- For
Entity Class
, press enter. - For
Subscriptions
, typesystem
and press enter.
Confirm both Sensu services are running:
systemctl status sensu-backend && systemctl status sensu-agent
The response should indicate active (running)
for both the Sensu backend and agent.
Add the file_exists check
Before Sensu can send alerts to your PagerDuty account, you’ll need a check to generate events.
In this guide, you will use a file_exists
check, which is included in Sensu’s Nagios Foundation dynamic runtime asset.
Before you can add the check, you need to add the Nagios Foundation asset to your Sensu configuration:
sensuctl asset add ncr-devops-platform/nagiosfoundation
To confirm that the asset was added to your Sensu backend, run:
sensuctl asset info ncr-devops-platform/nagiosfoundation
The list should include === ncr-devops-platform/nagiosfoundation
followed by a list of available builds for the asset.
Now that you’ve added the Nagios Foundation dynamic runtime asset, you can add its file_exists
check to your Sensu backend.
Use sensuctl to add the check:
sensuctl check create file_exists \
--command "check_file_exists --pattern /tmp/my-file.txt" \
--subscriptions system \
--handlers pagerduty \
--interval 10 \
--runtime-assets ncr-devops-platform/nagiosfoundation
To confirm that the check was added to your Sensu backend and view the check definition in YAML or JSON format, run:
sensuctl check info file_exists --format yaml
sensuctl check info file_exists --format wrapped-json
The response will list the complete check resource definition:
---
type: CheckConfig
api_version: core/v2
metadata:
name: file_exists
spec:
check_hooks: null
command: check_file_exists --pattern /tmp/my-file.txt
env_vars: null
handlers:
- pagerduty
high_flap_threshold: 0
interval: 10
low_flap_threshold: 0
output_metric_format: ""
output_metric_handlers: null
proxy_entity_name: ""
publish: true
round_robin: false
runtime_assets:
- ncr-devops-platform/nagiosfoundation
secrets: null
stdin: false
subdue: null
subscriptions:
- system
timeout: 0
ttl: 0
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"name": "file_exists"
},
"spec": {
"check_hooks": null,
"command": "check_file_exists --pattern /tmp/my-file.txt",
"env_vars": null,
"handlers": [
"pagerduty"
],
"high_flap_threshold": 0,
"interval": 10,
"low_flap_threshold": 0,
"output_metric_format": "",
"output_metric_handlers": null,
"proxy_entity_name": "",
"publish": true,
"round_robin": false,
"runtime_assets": [
"ncr-devops-platform/nagiosfoundation"
],
"secrets": null,
"stdin": false,
"subdue": null,
"subscriptions": [
"system"
],
"timeout": 0,
"ttl": 0
}
}
The check command includes the path for the file that the check will look for on your system, /tmp/my-file.txt
.
For this guide, you’ll add /tmp/my-file.txt
as a temporary file:
touch /tmp/my-file.txt
In production, you could replace /tmp/my-file.txt
with the file path or globbing pattern for any existing file or files on your system.
Well done! In the next step, you’ll configure your workflow for sending Sensu alerts to your PagerDuty account.
Add the PagerDuty handler
The Sensu PagerDuty Handler dynamic runtime asset includes the scripts you will need to send events to PagerDuty.
To add the PagerDuty handler asset, run:
sensuctl asset add sensu/sensu-pagerduty-handler
Once again, it’s a good idea to confirm that the asset was added to your Sensu backend. Run:
sensuctl asset info sensu/sensu-pagerduty-handler
The response will list the available builds for the PagerDuty handler dynamic runtime asset.
Now that you’ve added the Sensu PagerDuty Handler dynamic runtime asset, you can create a handler that uses the asset to send non-OK events to PagerDuty. This requires you to update the handler command by adding your PagerDuty API key.
In the following command, replace <pagerduty_key>
with your PagerDuty API integration key.
Then run the updated command:
sensuctl handler create pagerduty \
--type pipe \
--filters is_incident,not_silenced \
--runtime-assets sensu/sensu-pagerduty-handler \
--command "sensu-pagerduty-handler -t <pagerduty_key>"
NOTE: For checks whose handlers use the Sensu PagerDuty Handler dynamic runtime asset (like the one you’ve created in this guide), you can use an alternative method for authenticating and routing alerts based on PagerDuty teams.
To use this option, list the teams’ PagerDuty API integration keys in the handler definition as environment variables or secrets or in the /etc/default/sensu-backend
configuration file as environment variables.
Then, add check or agent annotations to specify which PagerDuty teams should receive alerts based on check events.
Sensu will look up the key in the handler definition or backend configuration file that corresponds to the team name in the check annotation to authenticate and send alerts.
Make sure that your handler was added by retrieving the complete handler definition in YAML or JSON format:
sensuctl handler info pagerduty --format yaml
sensuctl handler info pagerduty --format wrapped-json
The response will list the complete handler resource definition:
---
type: Handler
api_version: core/v2
metadata:
name: pagerduty
spec:
command: sensu-pagerduty-handler -t <pagerduty_key>
env_vars: null
filters:
- is_incident
- not_silenced
handlers: null
runtime_assets:
- sensu/sensu-pagerduty-handler
secrets: null
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "pagerduty"
},
"spec": {
"command": "sensu-pagerduty-handler -t <pagerduty_key>",
"env_vars": null,
"filters": [
"is_incident",
"not_silenced"
],
"handlers": null,
"runtime_assets": [
"sensu/sensu-pagerduty-handler"
],
"secrets": null,
"timeout": 0,
"type": "pipe"
}
}
Trigger an event
With your file_exists
check and PagerDuty handler workflow in place, you can remove a file to cause Sensu to send a non-OK event.
Remove the file /tmp/my-file.txt
:
rm /tmp/my-file.txt
This will make sure the file is not there for Sensu to find the next time the file_exists
check runs.
After about 10 seconds, Sensu will detect that my-file.txt
is missing and reflect that in an event.
To view the event with sensuctl, run:
sensuctl event list
The response should show that the file removal resulted in a CRITICAL (2) event:
Entity Check Output Status Silenced Timestamp UUID
────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ──────────────────────────────────────
host01 file_exists CheckFileExists CRITICAL - 0 files matched pattern /tmp/my-file.txt 2 false 2021-03-15 19:28:21 +0000 UTC 1b4266ae-7200-4728-a0n4-2f50f7a56613
Open the Sensu web UI to view the events the file_exists
check is generating.
Visit http://127.0.0.1:3000, and log in as the admin user (created during initialization when you installed the Sensu backend).
The failing check’s events will be listed on the Events page.
Observe the alert in PagerDuty
After Sensu detects the non-OK event, the handler you set up will send the alert to PagerDuty. Log in to your PagerDuty account to view an event similar to this one:

Resolve the alert in PagerDuty
To complete your workflow, restore the file that you removed so Sensu sends a resolution to PagerDuty:
touch /tmp/my-file.txt
In your PagerDuty account, the alert should now be listed under the “Resolved” tab.
To view the resolved event with sensuctl, run:
sensuctl event list
The response should show that the file is restored, with an OK (0) status:
Entity Check Output Status Silenced Timestamp UUID
────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ──────────────────────────────────────
host01 file_exists CheckFileExists OK - 1 files matched pattern /tmp/my-file.txt 0 false 2021-03-15 19:58:31 +0000 UTC 73f382fe-4290-48e8-955a-6efd54347b43
The web UI Events page will also show that the file_check
check is passing.
Next steps
You should now have a working set-up with a file_exists
check and a handler that sends alerts to your PagerDuty account.
To share and reuse the check and handler like code, save them to files and start building a monitoring as code repository.
You can customize your PagerDuty handler with configuration options like severity mapping, PagerDuty team-based routing and authentication via check and agent annotations, and event-based templating. Learn more about the Sensu PagerDuty integration and our curated, configurable quick-start template for incident management to integrate Sensu with your existing PagerDuty workflows.