Collect Prometheus metrics with Sensu

The Sensu Prometheus Collector is a check plugin that collects metrics from a Prometheus exporter or the Prometheus query API. This allows Sensu to route the collected metrics to one or more time-series databases, such as InfluxDB or Graphite.

The Prometheus ecosystem contains a number of actively maintained exporters, such as the node exporter for reporting hardware and operating system metrics or Google’s cAdvisor exporter for monitoring containers. These exporters expose metrics that Sensu can collect and route to one or more time-series databases. Sensu and Prometheus can run in parallel, complementing each other and making use of environments where Prometheus is already deployed.

This guide uses CentOS 7 as the operating system with all components running on the same compute resource. Commands and steps may change for different distributions or if components are running on different compute resources.

At the end of this guide, Prometheus will be scraping metrics. The Sensu Prometheus Collector will then query the Prometheus API as a Sensu check and send the metrics to an InfluxDB Sensu handler, which will send metrics to an InfluxDB instance. Finally, Grafana will query InfluxDB to display the collected metrics.

Install and configure Prometheus

Download and extract Prometheus with these commands:

wget https://github.com/prometheus/prometheus/releases/download/v2.6.0/prometheus-2.6.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*

Replace the default prometheus.yml configuration file with the following configuration:

global:
  scrape_interval: 15s
  external_labels:
    monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Start Prometheus in the background:

nohup ./prometheus --config.file=prometheus.yml > prometheus.log 2>&1 &

Ensure Prometheus is running:

ps -ef | grep "[p]rometheus"

The response should be similar to this example:

vagrant   7647  3937  2 22:23 pts/0    00:00:00 ./prometheus --config.file=prometheus.yml

Install and configure Sensu

Follow the RHEL/CentOS install instructions for the Sensu backend, the Sensu agent, and sensuctl.

Use sensuctl to add an app_tier subscription to the entity the Sensu agent is observing. Before you run the following code, Replace <entity_name> with the name of the entity on your system.

NOTE: To find your entity name, run sensuctl entity list. The ID is the name of your entity.

sensuctl entity update <entity_name>
  • For Entity Class, press enter.
  • For Subscriptions, type app_tier and press enter.

Run this command to confirm both Sensu services are running:

systemctl status sensu-backend && systemctl status sensu-agent

Install and configure InfluxDB

Add an InfluxDB repo:

echo "[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key" | sudo tee /etc/yum.repos.d/influxdb.repo

Install InfluxDB:

sudo yum -y install influxdb

Open /etc/influxdb/influxdb.conf and uncomment the http API line:

[http]
  # Determines whether HTTP endpoint is enabled.
  enabled = true

Start InfluxDB:

sudo systemctl start influxdb

Add the Sensu user and database with these commands:

influx -execute "CREATE DATABASE sensu"
influx -execute "CREATE USER sensu WITH PASSWORD 'sensu'"
influx -execute "GRANT ALL ON sensu TO sensu"

Install and configure Grafana

Install Grafana:

sudo yum install -y https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.1.4-1.x86_64.rpm

Change Grafana’s listen port so that it does not conflict with the Sensu web UI:

sudo sed -i 's/^;http_port = 3000/http_port = 4000/' /etc/grafana/grafana.ini

Create a /etc/grafana/provisioning/datasources/influxdb.yaml file, and add an InfluxDB data source:

apiVersion: 1

deleteDatasources:
  - name: InfluxDB
    orgId: 1

datasources:
  - name: InfluxDB
    type: influxdb
    access: proxy
    orgId: 1
    database: sensu
    user: grafana
    password: grafana
    url: http://localhost:8086

Start Grafana:

sudo systemctl start grafana-server

Create a Sensu InfluxDB handler

Add the Sensu InfluxDB handler asset

To add the Sensu InfluxDB Handler dynamic runtime asset to Sensu, run the following command:

sensuctl asset add sensu/sensu-influxdb-handler:3.7.0 -r sensu-influxdb-handler

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-influxdb-handler:3.7.0
added asset: sensu/sensu-influxdb-handler:3.7.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-influxdb-handler"].

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-influxdb-handler.

To confirm that the sensu-influxdb-handler asset is ready to use, run:

sensuctl asset list

The response should list the sensu-influxdb-handler dynamic runtime asset:

             Name                                                     URL                                            Hash    
───────────────────────────── ──────────────────────────────────────────────────────────────────────────────────── ──────────
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_386.tar.gz           6719527  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_amd64.tar.gz         d05650d  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_armv7.tar.gz         38918c1  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_arm64.tar.gz         944075f  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_windows_amd64.tar.gz       8228cbc  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_darwin_amd64.tar.gz        7c73e1d  

Add an InfluxDB handler

To add the handler definition that uses the Sensu InfluxDB Handler dynamic runtime asset, run:

cat << EOF | sensuctl create
---
type: Handler
api_version: core/v2
metadata:
  name: influxdb
spec:
  command: sensu-influxdb-handler -a 'http://127.0.0.1:8086' -d sensu -u sensu -p sensu
  timeout: 10
  type: pipe
  runtime_assets:
  - sensu-influxdb-handler
EOF
cat << EOF | sensuctl create
{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "influxdb"
  },
  "spec": {
    "command": "sensu-influxdb-handler -a 'http://127.0.0.1:8086' -d sensu -u sensu -p sensu",
    "timeout": 10,
    "type": "pipe",
    "runtime_assets": [
      "sensu-influxdb-handler"
    ]
  }
}
EOF

PRO TIP: sensuctl create --file also accepts files that contain multiple resources’ definitions. You could save both the asset and handler definitions in a single file and use sensuctl create --file FILE_NAME.EXT to add them.

Create a pipeline that includes the InfluxDB handler

Add your handler to a pipeline workflow. A single pipeline workflow can include one or more filters, one mutator, and one handler.

In this case, the pipeline includes only the InfluxDB handler you’ve already configured. To create the pipeline, run:

cat << EOF | sensuctl create
---
type: Pipeline
api_version: core/v2
metadata:
  name: prometheus_metrics_workflows
spec:
  workflows:
  - name: influxdb_metrics
    handler:
      name: influxdb
      type: Handler
      api_version: core/v2
EOF
cat << EOF | sensuctl create
{
  "type": "Pipeline",
  "api_version": "core/v2",
  "metadata": {
    "name": "prometheus_metrics_workflows"
  },
  "spec": {
    "workflows": [
      {
        "name": "influxdb_metrics",
        "handler": {
          "name": "influxdb",
          "type": "Handler",
          "api_version": "core/v2"
        }
      }
    ]
  }
}
EOF

Now you can add the prometheus_metrics_workflows pipeline to a check for check output metric extraction.

Collect Prometheus metrics with Sensu

Add the Sensu Prometheus Collector asset

To add the Sensu Prometheus Collector dynamic runtime asset to Sensu, run the following command:

sensuctl asset add sensu/sensu-prometheus-collector:1.3.2 -r sensu-prometheus-collector

The response will confirm that the asset was added:

fetching bonsai asset: sensu/sensu-prometheus-collector:1.3.2
added asset: sensu/sensu-prometheus-collector:1.3.2

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["sensu-prometheus-collector"].

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: sensu-prometheus-collector.

To confirm that the sensu-prometheus-collector asset is ready to use, run:

sensuctl asset list

The response should list the sensu-prometheus-collector dynamic runtime asset along with the previously added sensu-influxdb-handler asset:

             Name                                                     URL                                            Hash    
───────────────────────────── ──────────────────────────────────────────────────────────────────────────────────── ──────────
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_386.tar.gz           6719527  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_amd64.tar.gz         d05650d  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_armv7.tar.gz         38918c1  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_linux_arm64.tar.gz         944075f  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_windows_amd64.tar.gz       8228cbc  
  sensu-influxdb-handler       //assets.bonsai.sensu.io/.../sensu-influxdb-handler_3.7.0_darwin_amd64.tar.gz        7c73e1d  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_windows_amd64.tar.gz   77f47c9  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_darwin_amd64.tar.gz    5e25a41  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_linux_armv7.tar.gz     2ae6727  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_linux_armv6.tar.gz     acad256  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_linux_arm64.tar.gz     6bfdbfc  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_linux_386.tar.gz       69e6d02  
  sensu-prometheus-collector   //assets.bonsai.sensu.io/.../sensu-prometheus-collector_1.3.2_linux_amd64.tar.gz     aca56fa  

Add a Sensu check that references the pipeline

To add the check definition that uses the Sensu Prometheus Collector dynamic runtime asset and your prometheus_metrics_workflows pipeline, run:

cat << EOF | sensuctl create
---
type: CheckConfig
api_version: core/v2
metadata:
  name: prometheus_metrics
spec:
  command: sensu-prometheus-collector -prom-url http://localhost:9090 -prom-query up
  handlers: []
  interval: 10
  publish: true
  output_metric_format: influxdb_line
  pipelines:
  - name: prometheus_metrics_workflows
    type: Pipeline
    api_version: core/v2
  subscriptions:
  - app_tier
  timeout: 0
  runtime_assets:
  - sensu-prometheus-collector
EOF
cat << EOF | sensuctl create
{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "name": "prometheus_metrics"
  },
  "spec": {
    "command": "sensu-prometheus-collector -prom-url http://localhost:9090 -prom-query up",
    "handlers": [],
    "interval": 10,
    "publish": true,
    "output_metric_format": "influxdb_line",
    "pipelines": [
      {
        "name": "prometheus_metrics_workflows",
        "type": "Pipeline",
        "api_version": "core/v2"
      }
    ],
    "subscriptions": [
      "app_tier"
    ],
    "timeout": 0,
    "runtime_assets": [
      "sensu-prometheus-collector"
    ]
  }
}
EOF

The check subscription matches the subscription you added to your entity during set-up. The Sensu backend will coordinate check execution for you by comparing the subscriptions in your checks and entities. Sensu automatically executes a check when the check definition includes a subscription that matches a subscription for a Sensu entity.

Open the Sensu web UI to view the events generated by the prometheus_metrics check. Visit http://127.0.0.1:3000, and log in as the admin user (created during the initialization step when you installed the Sensu backend).

You can also view the metric event data using sensuctl. Run:

sensuctl event list

The response should be similar to this example:

     Entity            Check                                          Output                                   Status   Silenced             Timestamp                             UUID                  
─────────────── ──────────────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ───────────────────────────────────────
  sensu-centos   keepalive            Keepalive last sent from sensu-centos at 2022-01-14 15:23:00 +0000 UTC        0   false      2022-01-14 15:23:00 +0000 UTC   a9kr7kf8-21h8-459k-v6f8-ad93mf82mkfd  
  sensu-centos   prometheus_metrics   up,instance=localhost:9090,job=prometheus value=1 1642173795                  0   false      2022-01-14 15:23:15 +0000 UTC   sd8j4ls9-34gf-fr77-456g-92384738jd72

Visualize metrics with Grafana

Configure a dashboard in Grafana

Download the Grafana dashboard configuration file from the Sensu docs:

curl -O https://docs.sensu.io/sensu-go/latest/files/up_or_down_dashboard.json

Using the downloaded file, add the dashboard to Grafana with an API call:

curl  -XPOST -H 'Content-Type: application/json' -d@up_or_down_dashboard.json HTTP://admin:admin@127.0.0.1:4000/api/dashboards/db

View metrics in Grafana

Confirm metrics in Grafana: log in at http://127.0.0.1:4000. Use admin for both username and password.

Click Home in the upper left corner, then click the Up or Down Sample 2 dashboard. The page should include a graph with initial metrics, similar to this example:

Example Grafana dashboard for visualizing Prometheus metrics from Sensu

Next steps

You should now have a working observability pipeline with Prometheus scraping metrics. The Sensu Prometheus Collector dynamic runtime asset runs via the prometheus_metrics Sensu check and collects metrics from the Prometheus API.

The check sends metrics to the prometheus_metrics_workflows pipeline, and the influxdb handler sends the metrics to InfluxDB. You can visualize the metrics in a Grafana dashboard.

Add the Sensu Prometheus Collector to your Sensu ecosystem and include it in your monitoring as code repository. Use Prometheus to gather metrics and use Sensu to send them to the proper final destination. Prometheus has a comprehensive list of additional exporters to pull in metrics.