Send Slack alerts with a pipeline

Pipelines are Sensu resources composed of observation event processing workflows that include filters, mutators, and handlers. You can use pipelines to send email alerts, create or resolve incidents (in PagerDuty, for example), or store metrics in a time-series database like InfluxDB.

This guide will help you send alerts to Slack in the channel monitoring by configuring a pipeline and adding it to a check named check_cpu. If you don’t already have this check in place, as well as an entity with a system subscription, follow Monitor server resources to add the check and the correct entity subscription.

Register the dynamic runtime asset

Dynamic runtime assets are shareable, reusable packages that help you deploy Sensu plugins. In this guide, you’ll use the Sensu Slack Handler dynamic runtime asset to power a slack handler.

Use sensuctl asset add to register the Sensu Slack Handler dynamic runtime asset:

sensuctl asset add sensu/sensu-slack-handler:1.5.0 -r sensu-slack-handler

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-slack-handler.

The response will indicate that the asset was added:

fetching bonsai asset: sensu/sensu-slack-handler:1.5.0
added asset: sensu/sensu-slack-handler:1.5.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-slack-handler"].

You can also download the latest dynamic runtime asset definition for your platform from Bonsai and register the asset with sensuctl create --file filename.yml or sensuctl create --file filename.json.

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution. Read the asset reference for more information about dynamic runtime asset builds.

Get a Slack webhook

If you’re already the admin of a Slack, visit https://YOUR_WORKSPACE_NAME_HERE.slack.com/services/new/incoming-webhook and follow the steps to add the Incoming WebHooks integration, choose a channel, and save the settings. If you’re not yet a Slack admin, create a new workspace and then create and save your webhook.

After you save your webhook, you can find the webhook URL under Integration Settings.

Create a handler

Use sensuctl to create a handler called slack that pipes observation data (events) to Slack using the sensu-slack-handler dynamic runtime asset. Edit the sensuctl command below to include your Slack webhook URL and the channel where you want to receive events.

sensuctl handler create slack \
--type pipe \
--env-vars "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX" \
--command "sensu-slack-handler --channel '#monitoring'" \
--runtime-assets sensu-slack-handler

You should receive a confirmation message:

Created

The sensuctl handler create slack command creates a handler resource. To view the slack handler definition, run:

sensuctl handler info slack --format yaml
sensuctl handler info slack --format wrapped-json

The slack handler resource definition will be similar to this example:

---
type: Handler
api_version: core/v2
metadata:
  created_by: admin
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#monitoring'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX
  filters: null
  handlers: null
  runtime_assets:
  - sensu-slack-handler
  secrets: null
  timeout: 0
  type: pipe
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-slack-handler --channel '#monitoring'",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T0000/B000/XXXXXXXX"
    ],
    "filters": null,
    "handlers": null,
    "runtime_assets": [
      "sensu-slack-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}

You can share and reuse this handler like code — save it to a file and start building a monitoring as code repository.

Create a pipeline that includes the handler

With your handler configured, you can add it to a pipeline workflow. A single pipeline workflow can include one or more filters, one mutator, and one handler.

For now, the pipeline includes only the slack handler so that you receive an alert for every event the check generates (including events with OK status). To create the pipeline, run:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: cpu_check_alerts
spec:
  workflows:
  - name: slack_alerts
    handler:
      name: slack
      type: Handler
      api_version: core/v2
EOF
cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "cpu_check_alerts"
  },
  "spec": {
    "workflows": [
      {
        "name": "slack_alerts",
        "handler": {
          "name": "slack",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

Assign the pipeline to a check

To use the cpu_check_alerts pipeline, list it in a check definition’s pipelines array (in this case, the check_cpu check created in Monitor server resources). All the observability events that the check produces will be processed according to the pipeline’s workflows.

Assign your cpu_check_alerts pipeline to the check_cpu check to receive Slack alerts when the CPU usage of your system reaches the specific thresholds set in the check command.

To open the check definition in your text editor, run:

sensuctl edit check check_cpu

Replace the pipelines: [] line with the following array and save the updated check definition:

  pipelines:
  - type: Pipeline
    api_version: core/v2
    name: cpu_check_alerts
  "pipelines": [
    {
      "type": "Pipeline",
      "api_version": "core/v2",
      "name": "cpu_check_alerts"
    }
  ]

You should see a response to confirm the update:

Updated /api/core/v2/namespaces/default/checks/check_cpu

To view the updated check_cpu resource definition, run:

sensuctl check info check_cpu --format yaml
sensuctl check info check_cpu --format wrapped-json

The updated check definition will be similar to this example:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  name: check_cpu
  namespace: default
spec:
  check_hooks: null
  command: check-cpu-usage -w 75 -c 90
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  pipelines:
  - api_version: core/v2
    name: cpu_check_alerts
    type: Pipeline
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - check-cpu-usage
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "check_cpu",
    "namespace": "default",
    "created_by": "admin"
  },
  "spec": {
    "check_hooks": null,
    "command": "check-cpu-usage -w 75 -c 90",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "pipelines": [
      {
        "api_version": "core/v2",
        "name": "cpu_check_alerts",
        "type": "Pipeline"
      }
    ],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "check-cpu-usage"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

Validate the pipeline

It might take a few moments after you add the pipeline to the check for the check to be scheduled on entities with the system subscription and the result sent back to Sensu backend. After an event is handled, you should receive a message like this in Slack:

Example Slack message

Verify proper handler behavior with sensu-backend logs. Read Troubleshoot Sensu for log locations by platform.

Whenever an event is being handled, a log entry is added with the message "handler":"slack","level":"debug","msg":"sending event to handler", followed by a second log entry with the message "msg":"event pipe handler executed","output":"","status":0.

Add an event filter to the pipeline

At this point, the cpu_check_alerts pipeline has probably sent quite a few Slack messages for events with OK (0) status. To receive alerts for events with only warning (1) or critical (2) status, add the built-in is_incident event filter to the pipeline:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: cpu_check_alerts
spec:
  workflows:
  - name: slack_alerts
    filters:
    - name: is_incident
      type: EventFilter
      api_version: core/v2
    handler:
      name: slack
      type: Handler
      api_version: core/v2
EOF
cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "cpu_check_alerts"
  },
  "spec": {
    "workflows": [
      {
        "name": "slack_alerts",
        "filters": [
          {
            "name": "is_incident",
            "type": "EventFilter",
            "api_version": "core/v2"
          }
        ],
        "handler": {
          "name": "slack",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

Adding the is_incident filter to your pipeline should quickly reduce the number of alerts you receive in Slack.

Next steps

Now that you know how to apply a pipeline to a check and take action on events, read the pipeline reference for in-depth documentation. Read Route alerts with event filters for a more complex example with multiple filters and handlers organized into several pipeline workflows.

For more information about customizing your Slack alerts, read the Sensu Slack Handler page in Bonsai.

Follow Send PagerDuty alerts with Sensu to configure a check that generates status events and a handler that sends Sensu alerts to PagerDuty for non-OK events.