Populate metrics in InfluxDB with handlers

A Sensu event handler is an action the Sensu backend executes when a specific event occurs. In this guide, you’ll use a handler to populate the time-series database InfluxDB with Sensu observability event data.

Metrics can be collected from check output (in this guide, a check that generates Prometheus metrics) or the Sensu StatsD Server.

To follow this guide, you’ll need to install the Sensu backend, have at least one Sensu agent running, and install and configure sensuctl.

Configure a Sensu entity

Every Sensu agent has a defined set of subscriptions that determine which checks the agent will execute. For an agent to execute a specific check, you must specify the same subscription in the agent configuration and the check definition.

The example in this guide uses the prometheus_metrics check from Collect Prometheus metrics with Sensu, which includes the subscription app_tier. Use sensuctl to add an app_tier subscription to one of your entities.

Before you run the following code, replace <ENTITY_NAME> with the name of the entity on your system.

sensuctl entity update <ENTITY_NAME>

• For Entity Class, press enter.
• For Subscriptions, type app_tier and press enter.

Run this command to confirm both Sensu services are running:

systemctl status sensu-backend && systemctl status sensu-agent

The response should indicate active (running) for both the Sensu backend and agent.

Register the dynamic runtime asset

Dynamic runtime assets are shareable, reusable packages that make it easier to deploy Sensu plugins. This example uses the sensu/sensu-influxdb-handler dynamic runtime asset to power an InfluxDB handler.

Use sensuctl asset add to register the sensu/sensu-influxdb-handler dynamic runtime asset:

sensuctl asset add sensu/sensu-influxdb-handler:3.7.0 -r sensu-influxdb-handler

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-influxdb-handler:3.7.0 -r sensu-influxdb-handler
added asset: sensu/sensu-influxdb-handler:3.7.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-influxdb-handler"].

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-influxdb-handler.

You can also download the latest dynamic runtime asset definition for your platform from Bonsai and register the asset with sensuctl create --file filename.yml or sensuctl create --file filename.json.

Run sensuctl asset list to confirm that the dynamic runtime asset is ready to use.

Create the handler

Now that you have registered the dynamic runtime asset, use sensuctl to create a handler called influxdb-handler that pipes observation data (events) to InfluxDB with the sensu/sensu-influxdb-handler dynamic runtime asset. Edit the command below to replace the placeholders for database name, address, username, and password with the information for your own InfluxDB database. For more information about the Sensu InfluxDB handler, read the asset page in Bonsai.

sensuctl handler create influxdb-handler \
--type pipe \
--command "sensu-influxdb-handler -d sensu" \
--env-vars "INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086, INFLUXDB_USER=sensu, INFLUXDB_PASS=password" \
--filters not_silenced \
--runtime-assets sensu-influxdb-handler

You should receive a confirmation message:

Created

To review the complete resource definition for the handler resource you just created with sensuctl, run:

sensuctl handler info influxdb-handler --format yaml

sensuctl handler info influxdb-handler --format wrapped-json

The influxdb-handler resource definition will be similar to this example:

---
type: Handler
api_version: core/v2
metadata:
  name: influxdb-handler
spec:
  command: sensu-influxdb-handler -d sensu
  env_vars:
  - INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086
  - INFLUXDB_USER=sensu
  - INFLUXDB_PASS=password
  filters:
  - not_silenced
  handlers: null
  runtime_assets:
  - sensu-influxdb-handler
  secrets: null
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "influxdb-handler"
  },
  "spec": {
    "command": "sensu-influxdb-handler -d sensu",
    "env_vars": [
      "INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086",
      "INFLUXDB_USER=sensu",
      "INFLUXDB_PASS=password"
    ],
    "filters": [
      "not_silenced"
    ],
    "handlers": null,
    "runtime_assets": [
      "sensu-influxdb-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}

You can share, reuse, and maintain this handler just like you would code: save it to a file and start building a monitoring as code repository.

Assign the InfluxDB handler to a check

With the influxdb-handler resource created, you can assign it to a check for check output metric extraction. For example, if you followed Collect Prometheus metrics with Sensu, you created the prometheus_metrics check. The prometheus_metrics check already uses the influxdb_line output metric format, but you will need to update the output metric handlers to extract the metrics with the influxdb-handler.

To update the output metric handlers, run:

sensuctl check set-output-metric-handlers prometheus_metrics influxdb-handler

You should receive a confirmation message:

Updated

To review the updated check resource definition, run:

sensuctl check info prometheus_metrics --format yaml

sensuctl check info prometheus_metrics --format wrapped-json

The updated prometheus_metrics check definition will be similar to this example:

---
type: CheckConfig
api_version: core/v2
metadata:
  name: prometheus_metrics
spec:
  check_hooks: null
  command: sensu-prometheus-collector -prom-url http://localhost:9090 -prom-query up
  env_vars: null
  handlers: []
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: influxdb_line
  output_metric_handlers:
  - influxdb-handler
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - sensu-prometheus-collector
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - app_tier
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "prometheus_metrics"
  },
  "spec": {
    "check_hooks": null,
    "command": "sensu-prometheus-collector -prom-url http://localhost:9090 -prom-query up",
    "env_vars": null,
    "handlers": [],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "output_metric_format": "influxdb_line",
    "output_metric_handlers": [
      "influxdb-handler"
    ],
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "sensu-prometheus-collector"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "app_tier"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

Assign the InfluxDB handler to the Sensu StatsD listener

To assign your influxdb-handler resource to the Sensu StatsD listener at agent startup and pass all StatsD metrics into InfluxDB:

sensu-agent start --statsd-event-handlers influxdb-handler

Validate the InfluxDB handler

It might take a few moments after you assign the handler to the check or StatsD server for Sensu to receive the metrics, but after an event is handled, metrics should start populating InfluxDB. You can verify proper handler behavior with sensu-backend logs. Read Troubleshoot Sensu for log locations by platform.

Whenever an event is being handled, a log entry is added with the message "handler":"influxdb-handler","level":"debug","msg":"sending event to handler", followed by a second log entry with the message "msg":"pipelined executed event pipe handler","output":"","status":0.

Next steps

Now that you know how to apply an InfluxDB handler to metrics, read Aggregate metrics with the Sensu StatsD listener to learn more about using Sensu to implement StatsD and take action on observability events.