Collect metrics with Sensu checks

Sensu checks are commands (or scripts) that the Sensu agent executes that output data and produce an exit code to indicate a state. If you are unfamiliar with checks or want to learn how to configure a check before reading this guide, read the check reference and Monitor server resources.

Extract metrics from check output

To extract metrics from check output, you’ll need to:

  1. Configure the check command so that the command execution outputs metrics in one of the supported output metric formats.
  2. Configure the check output_metric_format to one of the supported output metric formats.

You can also configure the check output_metric_handlers to a Sensu handler that is equipped to handle Sensu metrics if you wish. See handlers or influx-db handler to learn more.

You can configure the check with these fields at creation or use the commands in this guide (assuming you have a check named collect-metrics). This example uses graphite_plaintext format and sends the metrics to a handler named influx-db.

sensuctl check set-command collect-metrics collect_metrics.sh
sensuctl check set-output-metric-format collect-metrics graphite_plaintext
sensuctl check set-output-metric-handlers collect-metrics influx-db

Supported output metric formats

The output metric formats that Sensu currently supports for check output metric extraction are nagios, influxdb, graphite, and opentsdb.

nagios
output_metric_format nagios_perfdata
documentation Nagios Performance Data
example
PING ok - Packet loss = 0%, RTA = 0.80 ms | percent_packet_loss=0, rta=0.80
graphite
output_metric_format graphite_plaintext
documentation Graphite Plaintext Protocol
example
local.random.diceroll 4 123456789
influxdb
output_metric_format influxdb_line
documentation InfluxDB Line Protocol
example
weather,location=us-midwest temperature=82 1465839830100400200
opentsdb
output_metric_format opentsdb_line
documentation OpenTSDB Data Specification
example
sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0

Validate the metrics

If the check output is formatted correctly according to its output_metric_format, the metrics will be extracted in Sensu metric format and passed to the event pipeline. You should expect to see errors logged by sensu-agent if it is unable to parse the check output. To confirm that metrics have been extracted from your check, inspect the event passed to the handler.

See Troubleshoot Sensu for an example debug handler that writes events to a file for inspection.

The example check would yield an event similar to this:

type: Event
api_version: core/v2
metadata: {}
spec:
  check:
    command: collect_metrics.sh
    metadata:
      name: collect-metrics
      namespace: default
    output: |-
      cpu.idle_percentage 61 1525462242
      mem.sys 104448 1525462242
    output_metric_format: graphite_plaintext
    output_metric_handlers:
    - influx-db
  metrics:
    handlers:
    - influx-db
    points:
    - name: cpu.idle_percentage
      tags: []
      timestamp: 1525462242
      value: 61
    - name: mem.sys
      tags: []
      timestamp: 1525462242
      value: 104448
{
  "type": "Event",
  "api_version": "core/v2",
  "metadata": {},
  "spec": {
    "check": {
      "metadata": {
        "name": "collect-metrics",
        "namespace": "default"
      },
      "command": "collect_metrics.sh",
      "output": "cpu.idle_percentage 61 1525462242\nmem.sys 104448 1525462242",
      "output_metric_format": "graphite_plaintext",
      "output_metric_handlers": [
        "influx-db"
      ]
    },
    "metrics": {
      "handlers": [
        "influx-db"
      ],
      "points": [
        {
          "name": "cpu.idle_percentage",
          "value": 61,
          "timestamp": 1525462242,
          "tags": []
        },
        {
          "name": "mem.sys",
          "value": 104448,
          "timestamp": 1525462242,
          "tags": []
        }
      ]
    }
  }
}

Next steps

Now you know how to extract metrics from check output! Check out these resources for more information about scheduling checks and using handlers: