Handlers

How do Sensu handlers work?

Handlers actions are executed by the Sensu backend on events, and there are several types of handlers available. The most common handler type is the pipe handler, which works very similarly to how checks work, enabling Sensu to interact with almost any computer program via standard streams.

  • Pipe handlers. Pipe handlers pipe event data into arbitrary commands via STDIN.
  • TCP/UDP handlers. TCP and UDP handlers send event data to a remote socket.
  • Handler sets. Handler sets (also called “set handlers”) are used to group event handlers, making it easy to manage groups of actions that should be executed for certain types of events.

Pipe handlers

Pipe handlers are external commands that can consume event data via STDIN.

Pipe handler command

Pipe handler definitions include a command attribute, which is a command to be executed by the Sensu backend.

Pipe handler command arguments

Pipe handler command attributes may include command line arguments for controlling the behavior of the command executable.

TCP/UDP handlers

TCP and UDP handlers enable Sensu to forward event data to arbitrary TCP or UDP sockets for external services to consume.

Handler sets

Handler set definitions allow groups of handlers (individual collections of actions to take on event data) to be referenced via a single named handler set.

NOTE: Attributes defined on handler sets do not apply to the handlers they include. For example, filters, and mutator attributes defined in a handler set will have no effect.

New and improved handlers

Default handler

Sensu no longer attempts to handle events using a handler named default, which caused confusion as this default handler was only a reference, since Sensu did not provide a built-in default handler.

Transport handlers

Sensu architecture considerably changed between the 1.x and Sensu Go versions, and a dedicated message bus (like RabbitMQ) is no longer used. Therefore, transport handlers have been removed but a similar functionality could be achieved using a pipe handler that connects to a message bus and injects event data into a queue.

Each and every check results are handled

All check results are now considered to be events, and therefore these events are automatically sent to the check handlers, no matter their status, whereas only non-zero check results were considered as events and sent to handlers in Sensu 1.x.

That being said, 1.x behavior can be replicated in Sensu Go by using the built-in is_incident filter.

Handler specification

Top-level attributes

type
description Top-level attribute specifying the sensuctl create resource type. Handlers should always be of type Handler.
required Required for handler definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example
"type": "Handler"
api_version
description Top-level attribute specifying the Sensu API group and version. For handlers in Sensu backend version 5.0, this attribute should always be core/v2.
required Required for handler definitions in wrapped-json or yaml format for use with sensuctl create.
type String
example
"api_version": "core/v2"
metadata
description Top-level collection of metadata about the handler, including the name and namespace as well as custom labels and annotations. The metadata map is always at the top level of the handler definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. See the metadata attributes reference for details.
required Required for handler definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example
"metadata": {
  "name": "handler-slack",
  "namespace": "default",
  "labels": {
    "region": "us-west-1"
  },
  "annotations": {
    "slack-channel" : "#monitoring"
  }
}
spec
description Top-level map that includes the handler spec attributes.
required Required for handler definitions in wrapped-json or yaml format for use with sensuctl create.
type Map of key-value pairs
example
"spec": {
  "type": "tcp",
  "socket": {
    "host": "10.0.1.99",
    "port": 4444
  },
  "metadata" : {
    "name": "tcp_handler",
    "namespace": "default"
  }
}

Spec attributes

type
description The handler type.
required true
type String
allowed values pipe, tcp, udp & set
example
"type": "pipe"
filters
description An array of Sensu event filters (names) to use when filtering events for the handler. Each array item must be a string.
required false
type Array
example
"filters": ["occurrences", "production"]
mutator
description The Sensu event mutator (name) to use to mutate event data for the handler.
required false
type String
example
"mutator": "only_check_output"
timeout
description The handler execution duration timeout in seconds (hard stop). Only used by pipe and tcp handler types.
required false
type Integer
default 60 (for tcp handler)
example
"timeout": 30
command
description The handler command to be executed. The event data is passed to the process via STDIN.NOTE: the command attribute is only supported for Pipe handlers (i.e. handlers configured with "type": "pipe").
required true (if type equals pipe)
type String
example
"command": "/etc/sensu/plugins/pagerduty.go"
env_vars
description An array of environment variables to use with command execution.NOTE: the env_vars attribute is only supported for Pipe handlers (i.e. handlers configured with "type": "pipe").
required false
type Array
example
"env_vars": ["API_KEY=0428d6b8nb51an4d95nbe28nf90865a66af5"]
socket
description The socket definition scope, used to configure the TCP/UDP handler socket.NOTE: the socket attribute is only supported for TCP/UDP handlers (i.e. handlers configured with "type": "tcp" or "type": "udp").
required true (if type equals tcp or udp)
type Hash
example
"socket": {}
handlers
description An array of Sensu event handlers (names) to use for events using the handler set. Each array item must be a string.NOTE: the handlers attribute is only supported for handler sets (i.e. handlers configured with "type": "set").
required true (if type equals set)
type Array
example
"handlers": ["pagerduty", "email", "ec2"]
runtime_assets
description An array of Sensu assets (names), required at runtime for the execution of the command
required false
type Array
example
"runtime_assets": ["ruby-2.5.0"]

Metadata attributes

name
description A unique string used to identify the handler. Handler names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z). Each handler must have a unique name within its namespace.
required true
type String
example
"name": "handler-slack"
namespace
description The Sensu RBAC namespace that this handler belongs to.
required false
type String
default default
example
"namespace": "production"
labels
description Custom attributes to include with event data, which can be queried like regular attributes. You can use labels to organize handlers into meaningful collections that can be selected using filters and tokens.
required false
type Map of key-value pairs. Keys and values can be any valid UTF-8 string.
default null
example
"labels": {
  "environment": "development",
  "region": "us-west-2"
}
annotations
description Arbitrary, non-identifying metadata to include with event data. In contrast to labels, annotations are not used internally by Sensu and cannot be used to identify handlers. You can use annotations to add data that helps people or external tools interacting with Sensu.
required false
type Map of key-value pairs. Keys and values can be any valid UTF-8 string.
default null
example
 "annotations": {
  "managed-by": "ops",
  "slack-channel": "#monitoring",
  "playbook": "www.example.url"
}

socket attributes

host
description The socket host address (IP or hostname) to connect to.
required true
type String
example
"host": "8.8.8.8"
port
description The socket port to connect to.
required true
type Integer
example
"port": 4242

Handler examples

Sending slack alerts

This handler will send alerts to a channel named monitoring with the configured webhook URL, using the handler-slack executable command.

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata" : {
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "type": "pipe",
    "command": "handler-slack --webhook-url https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX --channel monitoring"
  }
}

Sending event data to a TCP socket

This handler will forward event data to a TCP socket (10.0.1.99:4444) and will timeout if an acknowledgement (ACK) is not received within 30 seconds.

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata" : {
    "name": "tcp_handler",
    "namespace": "default"
  },
  "spec": {
    "type": "tcp",
    "socket": {
      "host": "10.0.1.99",
      "port": 4444
    },
    "metadata" : {
      "name": "tcp_handler",
      "namespace": "default"
    }
  }
}

Sending event data to a UDP socket

The following example will also forward event data but to UDP socket instead (ex: 10.0.1.99:4444).

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata" : {
    "name": "udp_handler",
    "namespace": "default"
  },
  "spec": {
    "type": "udp",
    "socket": {
      "host": "10.0.1.99",
      "port": 4444
    }
  }
}

Executing multiple handlers

The following example handler will execute three handlers: slack, tcp_handler, and udp_handler.

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata" : {
    "name": "notify_all_the_things",
    "namespace": "default"
  },
  "spec": {
    "type": "set",
    "handlers": [
      "slack",
      "tcp_handler",
      "udp_handler"
    ]
  }
}