How to plan maintenance windows using silencing
What is Sensu silencing?
As check results are processed by a Sensu server, the server executes event
handlers to send alerts to personnel or otherwise relay event data to
external services. Sensu’s built-in silencing, along with the built-in
not_silenced
filter, provides the means to suppress execution of event
handlers on an ad hoc basis.
When to use silencing
Silencing is used to prevent handlers configured with the not_silenced
filter
from being triggered based on the check name present in a check result or the
subscriptions associated with the entity that published the check result. This
can be desirable in many scenarios, giving operators the ability to quiet
incoming alerts while coordinating their response.
Sensu silencing entries make it possible to:
- Silence all checks on a specific entity
- Silence a specific check on a specific entity
- Silence all checks on entities with a specific subscription
- Silence a specific check on entities with a specific subscription
- Silence a specific check on every entity
Using silencing to plan maintenance
The purpose of this guide is to help you plan a maintenance window, by creating
a silenced entry for a specific entity named i-424242
and its check named
check-http
, in order to prevent alerts as you restart or redeploy the
services associated with this entity.
Creating the silenced entry
The first step is to create a silenced entry that will silence the check
check-http
on an entity named i-424242
, for a planned maintenance window
that starts at 01:00, on Sunday, and ends 1 hour later. Your
username will automatically be added as the creator of the silenced entry.
sensuctl silenced create \
--subscription 'entity:i-424242' \
--check 'check-http' \
--begin '2018-03-16 01:00:00 -04:00' \
--expire 3600 \
--reason 'Server upgrade'
See the sensuctl documentation for the supported time formats in the
begin
flag.
Validating the silenced entry
You can verify that the silenced entry against our entity, here named
i-424242
, has been properly created, by using sensuctl
.
sensuctl silenced info 'entity:i-424242:check-http'
Once the silenced entry starts to take effect, events that are silenced will be
marked as so in sensuctl events
.
sensuctl event list
Entity Check Output Status Silenced Timestamp
────────────── ───────── ───────── ──────────── ────────── ───────────────────────────────
i-424242 check-http 0 true 2018-03-16 13:22:16 -0400 EDT
WARNING: By default, a silenced event will be handled unless the handler uses
the not_silenced
filter to discard silenced events.
Next steps
You now know how to create silenced entries to plan a maintenance and hopefully avoid false positive. From this point, here are some recommended resources:
- Read the silencing reference for in-depth documentation on silenced entries.