How to monitor server resources with checks

What are Sensu checks?

Sensu checks are commands (or scripts), executed by the Sensu agent, that output data and produce an exit code to indicate a state. Sensu checks use the same specification as Nagios, therefore, Nagios check plugins may be used with Sensu.

Why use a check?

You can use checks to monitor server resources, services, and application health (for example: is Nginx running?) as well as collect and analyze metrics (for example: how much disk space do I have left?).

Using checks to monitor a service

The purpose of this guide is to help you monitor server resources, more specifically the CPU usage, by configuring a check named check-cpu with a subscription named system, in order to target all entities subscribed to the system subscription. This guide requires a Sensu backend and at least one Sensu agent running on Linux.

Registering assets

To power the check, we’ll use the Sensu CPU checks asset and the Sensu Ruby runtime asset.

Use the following sensuctl example to register the sensu-plugins-cpu-checks asset for CentOS, or download the asset definition for Debian or Alpine from Bonsai and register the asset using sensuctl create --file filename.yml.

sensuctl asset create sensu-plugins-cpu-checks --url "https://assets.bonsai.sensu.io/68546e739d96fd695655b77b35b5aabfbabeb056/sensu-plugins-cpu-checks_4.0.0_centos_linux_amd64.tar.gz" --sha512 "518e7c17cf670393045bff4af318e1d35955bfde166e9ceec2b469109252f79043ed133241c4dc96501b6636a1ec5e008ea9ce055d1609865635d4f004d7187b"

Then use the following sensuctl example to register the sensu-ruby-runtime asset for CentOS, or download the asset definition for Debian or Alpine from Bonsai and register the asset using sensuctl create --file filename.yml.

sensuctl asset create sensu-ruby-runtime --url "https://assets.bonsai.sensu.io/03d08cdfc649500b7e8cd1708bb9bb93d91fea9e/sensu-ruby-runtime_0.0.8_ruby-2.4.4_centos_linux_amd64.tar.gz" --sha512 "7b254d305af512cc524a20a117c601bcfae0d51d6221bbfc60d8ade180cc1908081258a6eecfc9b196b932e774083537efe748c1534c83d294873dd3511e97a3"

You can use sensuctl to confirm that both the sensu-plugins-cpu-checks and sensu-ruby-runtime assets are ready to use.

sensuctl asset list
          Name                                                URL                                       Hash    
────────────────────────── ─────────────────────────────────────────────────────────────────────────── ───────── 
 sensu-plugins-cpu-checks   //assets.bonsai.sensu.io/.../sensu-plugins-cpu-checks_4.0.0_centos_linux_amd64.tar.gz          518e7c1  
 sensu-ruby-runtime         //assets.bonsai.sensu.io/.../sensu-ruby-runtime_0.0.10_ruby-2.4.4_centos_linux_amd64.tar.gz     338b88b

Creating the check

Now that the assets are registered, we’ll create a check named check-cpu, which runs the command check-cpu.rb -w 75 -c 90 using the sensu-plugins-cpu-checks and sensu-ruby-runtime assets, at an interval of 60 seconds, for all entities subscribed to the system subscription. This checks generates a warning event (-w) when CPU usage reaches 75% and a critical alert (-c) at 90%.

sensuctl check create check-cpu \
--command 'check-cpu.rb -w 75 -c 90' \
--interval 60 \
--subscriptions system \
--runtime-assets sensu-plugins-cpu-checks,sensu-ruby-runtime

Configuring the subscription

To run the check, we’ll need a Sensu agent with the subscription system. After installing an agent, open /etc/sensu/agent.yml and add the system subscription so the subscription configuration looks like:

subscriptions:
  - system

Then restart the agent.

sudo service sensu-agent restart

Validating the check

We can use sensuctl to see that Sensu is monitoring CPU usage using the check-cpu, returning an OK status (0). It might take a few moments, once the check is created, for the check to be scheduled on the entity and the event returned to Sensu backend.

sensuctl event list
    Entity        Check                                                                    Output                                                                   Status   Silenced             Timestamp            
────────────── ─────────── ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── 
 sensu-centos   check-cpu   CheckCPU TOTAL OK: total=0.2 user=0.0 nice=0.0 system=0.2 idle=99.8 iowait=0.0 irq=0.0 softirq=0.0 steal=0.0 guest=0.0 guest_nice=0.0        0   false      2019-04-23 16:42:28 +0000 UTC

Next steps

You now know how to run a simple check to monitor CPU usage. From this point, here are some recommended resources: