How to collect and extract metrics using Sensu checks

What are Sensu checks?

In short, Sensu checks are commands (or scripts), executed by the Sensu agent, that output data and produce an exit code to indicate a state. If you are unfamiliar with checks, or would like to learn how to configure one first, take a look through the check reference doc and guide before you continue.

Extracting metrics from check output

In order to extract metrics from check output, you’ll need to do the following:

  1. Configure the check command such that the command execution outputs metrics in one of the supported output metric formats.
  2. Configure the check output_metric_format to one of the supported output metric formats.
  3. Configure the check output_metric_handlers (optional) to a Sensu handler that is equipped to handle Sensu metrics (see handlers or influx-db handler to learn more).

You can configure the check with these fields at creation, or use the commands below assuming you have a check named collect-metrics. In this example, we’ll be using graphite_plaintext format and sending the metrics to a handler named influx-db.

sensuctl check set-command collect-metrics collect_metrics.sh
sensuctl check set-output-metric-format collect-metrics graphite_plaintext
sensuctl check set-output-metric-handlers collect-metrics influx-db

Supported output metric formats

The output metric formats that Sensu currently supports for check output metric extraction are nagios, influxdb, graphite, and opentsdb.

nagios
output_metric_format nagios_perfdata
documentation Nagios Performance Data
example
PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0, rta=0.80
graphite
output_metric_format graphite_plaintext
documentation Graphite Plaintext Protocol
example
local.random.diceroll 4 123456789
influxdb
output_metric_format influxdb_line
documentation InfluxDB Line Protocol
example
weather,location=us-midwest temperature=82 1465839830100400200
opentsdb
output_metric_format opentsdb_line
documentation OpenTSDB Data Specification
example
sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0

Validating the metrics

If the check output is formatted correctly according to its output_metric_format, the metrics will be extracted in Sensu Metric Format, and saved within the event. You should expect to see logged errors if Sensu is unable to parse the check output. You can validate that metrics have been extracted from your check through your handler, or through the resulting event. The example check we used would yield an event similar to the one below:

{
  "type": "Event",
  "api_version": "core/v2",
  "metadata": {},
  "spec": {
    "check": {
      "metadata": {
        "name": "collect-metrics",
        "namespace": "default"
      },
      "command": "collect_metrics.sh",
      "output": "cpu.idle_percentage 61 1525462242\nmem.sys 104448 1525462242",
      "output_metric_format": "graphite_plaintext",
      "output_metric_handlers": [
        "influx-db"
      ]
    },
    "metrics": {
      "handlers": [
        "influx-db"
      ],
      "points": [
        {
          "name": "cpu.idle_percentage",
          "value": 61,
          "timestamp": 1525462242,
          "tags": []
        },
        {
          "name": "mem.sys",
          "value": 104448,
          "timestamp": 1525462242,
          "tags": []
        }
      ]
    }
  }
}

Next steps

Now you know how to extract metrics from check output! Check out the below resources for some further reading: