Sensu backend

The Sensu backend is a service that manages check requests and event data. Every Sensu backend includes an integrated transport for scheduling checks using subscriptions, an event processing pipeline that applies filters, mutators, and handlers, an embedded etcd datastore for storing configuration and state, a Sensu API, Sensu dashboard, and sensu-backend command-line tool. The Sensu backend is available for Ubuntu/Debian and RHEL/CentOS distributions of Linux. See the installation guide to install the backend.

Event pipeline

The backend processes event data and executes filters, mutators, and handlers. These pipelines are powerful tools to automate your monitoring workflows. To learn more about filters, mutators, and handlers, see:

Check scheduling

The backend is responsible for storing check definitions and scheduling check requests. Check scheduling is subscription-based; the backend sends check requests to subscriptions where they’re picked up by subscribing agents.

For information about creating and managing checks, see:

Operation and service management

NOTE: Commands in this section may require administrative privileges.

Starting the service

Use the sensu-backend tool to start the backend and apply configuration flags.

To start the backend with configuration flags:

sensu-backend start --state-dir /var/lib/sensu/sensu-backend --log-level debug

To see available configuration flags and defaults:

sensu-backend start --help

If no configuration flags are provided, the backend loads configuration from /etc/sensu/backend.yml by default.

To start the backend using a service manager:

service sensu-backend start

Stopping the service

To stop the backend service using a service manager:

service sensu-backend stop

Restarting the service

You must restart the backend to implement any configuration updates.

To restart the backend using a service manager:

service sensu-backend restart

Enabling on boot

To enable the backend to start on system boot:

systemctl enable sensu-backend

To disable the backend from starting on system boot:

systemctl disable sensu-backend

NOTE: On older distributions of Linux, use sudo chkconfig sensu-server on to enable the backend and sudo chkconfig sensu-server off to disable.

Getting service status

To see the status of the backend service using a service manager:

service sensu-backend status

Getting service version

To get the current backend version using the sensu-backend tool:

sensu-backend version

Getting help

The sensu-backend tool provides general and command-specific help flags:

# Show sensu-backend commands
sensu-backend help

# Show options for the sensu-backend start subcommand
sensu-backend start --help

Clustering

You can run the backend as a standalone service, but running a cluster of backends makes Sensu more highly available, reliable, and durable. Sensu backend clusters build on the clustering system used by etcd. Clustering lets you synchronize data between backends and get the benefits of a highly available configuration. To configure a cluster, see:

Time synchronization

System clocks between agents and the backend should be synchronized to a central NTP server. Out of sync system time may cause issues with keepalive, metric, and check alerts.

Configuration

You can specify the backend configuration using a /etc/sensu/backend.yml file or using sensu-backend start configuration flags. The backend requires that the state-dir flag be set before starting; all other required flags have default values. See the example config file for flags and defaults. The backend loads configuration upon startup, so you must restart the backend for any configuration updates to take effect.

Configuration summary

$ sensu-backend start --help
start the sensu backend

Usage:
  sensu-backend start [flags]

General Flags:
      --agent-host string               agent listener host (default "[::]")
      --agent-port int                  agent listener port (default 8081)
      --api-listen-address string       address to listen on for api traffic (default "[::]:8080")
      --api-url string                  url of the api to connect to (default "http://localhost:8080")
      --cache-dir string                path to store cached data (default "/var/cache/sensu/sensu-backend")
      --cert-file string                TLS certificate in PEM format
  -c, --config-file string              path to sensu-backend config file
      --dashboard-cert-file string      dashboard TLS certificate in PEM format
      --dashboard-host string           dashboard listener host (default "[::]")
      --dashboard-key-file string       dashboard TLS certificate key in PEM format
      --dashboard-port int              dashboard listener port (default 3000)
      --debug                           enable debugging and profiling features
      --deregistration-handler string   default deregistration handler
      --event-log-buffer-size int       buffer size of the event logger (default 100000)
      --event-log-file string           path to the event log file
      --eventd-buffer-size int          number of incoming events that can be buffered (default 100)
      --eventd-workers int              number of workers spawned for processing incoming events (default 100)
  -h, --help                            help for start
      --insecure-skip-tls-verify        skip TLS verification (not recommended!)
      --keepalived-buffer-size int      number of incoming keepalives that can be buffered (default 100)
      --keepalived-workers int          number of workers spawned for processing incoming keepalives (default 100)
      --key-file string                 TLS certificate key in PEM format
      --log-level string                logging level [panic, fatal, error, warn, info, debug] (default "warn")
      --pipelined-buffer-size int       number of events to handle that can be buffered (default 100)
      --pipelined-workers int           number of workers spawned for handling events through the event pipeline (default 100)
  -d, --state-dir string                path to sensu state storage (default "/var/lib/sensu/sensu-backend")
      --trusted-ca-file string          TLS CA certificate bundle in PEM format used for etcd client (mutual TLS)

Store Flags:
      --etcd-advertise-client-urls strings         list of this member's client URLs to advertise to the rest of the cluster. (default [http://localhost:2379])
      --etcd-cert-file string                      path to the client server TLS cert file
      --etcd-cipher-suites strings                 list of ciphers to use for etcd TLS configuration
      --etcd-client-cert-auth                      enable client cert authentication
      --etcd-initial-advertise-peer-urls strings   list of this member's peer URLs to advertise to the rest of the cluster (default [http://127.0.0.1:2380])
      --etcd-initial-cluster string                initial cluster configuration for bootstrapping (default "default=http://127.0.0.1:2380")
      --etcd-initial-cluster-state string          initial cluster state ("new" or "existing") (default "new")
      --etcd-initial-cluster-token string          initial cluster token for the etcd cluster during bootstrap
      --etcd-key-file string                       path to the client server TLS key file
      --etcd-listen-client-urls strings            list of URLs to listen on for client traffic (default [http://127.0.0.1:2379])
      --etcd-listen-peer-urls strings              list of URLs to listen on for peer traffic (default [http://127.0.0.1:2380])
      --etcd-max-request-bytes uint                maximum etcd request size in bytes (use with caution) (default 1572864)
      --etcd-name string                           human-readable name for this member (default "default")
      --etcd-peer-cert-file string                 path to the peer server TLS cert file
      --etcd-peer-client-cert-auth                 enable peer client cert authentication
      --etcd-peer-key-file string                  path to the peer server TLS key file
      --etcd-peer-trusted-ca-file string           path to the peer server TLS trusted CA file
      --etcd-quota-backend-bytes int               maximum etcd database size in bytes (use with caution) (default 4294967296)
      --etcd-trusted-ca-file string                path to the client server TLS trusted CA cert file
      --no-embed-etcd                              don't embed etcd, use external etcd instead

General configuration flags

cache-dir
description Path to store cached data
type String
default /var/cache/sensu/sensu-backend
example 
# Command line example
sensu-backend start --cache-dir /cache/sensu-backend

# /etc/sensu/backend.yml example
cache-dir: "/cache/sensu-backend"
config-file
description Path to Sensu backend config file
type String
default /etc/sensu/backend.yml
example 
# Command line example
sensu-backend start --config-file /etc/sensu/backend.yml
sensu-backend start -c /etc/sensu/backend.yml

# /etc/sensu/backend.yml example
config-file: "/etc/sensu/backend.yml"
debug
description Enable debugging and profiling features
type Boolean
default false
example 
# Command line example
sensu-backend start --debug

# /etc/sensu/backend.yml example
debug: true
deregistration-handler
description Default event handler to use when processing agent deregistration events.
type String
default ""
example 
# Command line example
sensu-backend start --deregistration-handler /path/to/handler.sh

# /etc/sensu/backend.yml example
deregistration-handler: "/path/to/handler.sh"
log-level
description Logging level: panic, fatal, error, warn, info, or debug
type String
default warn
example 
# Command line example
sensu-backend start --log-level debug

# /etc/sensu/backend.yml example
log-level: "debug"
state-dir
description Path to Sensu state storage: /var/lib/sensu/sensu-backend.
type String
required true
example 
# Command line example
sensu-backend start --state-dir /var/lib/sensu/sensu-backend
sensu-backend start -d /var/lib/sensu/sensu-backend

# /etc/sensu/backend.yml example
state-dir: "/var/lib/sensu/sensu-backend"
api-listen-address
description Address the API daemon will listen for requests on
type String
default [::]:8080
example 
# Command line example
sensu-backend start --api-listen-address [::]:8080

# /etc/sensu/backend.yml example
api-listen-address: "[::]:8080"
api-url
description URL used to connect to the API
type String
default http://localhost:8080
example 
# Command line example
sensu-backend start --api-url http://localhost:8080

# /etc/sensu/backend.yml example
api-url: "http://localhost:8080"

Agent communication configuration flags

agent-host
description agent listener host, listens on all IPv4 and IPv6 addresses by default
type String
default [::]
example 
# Command line example
sensu-backend start --agent-host 127.0.0.1

# /etc/sensu/backend.yml example
agent-host: "127.0.0.1"
agent-port
description agent listener port
type Integer
default 8081
example 
# Command line example
sensu-backend start --agent-port 8081

# /etc/sensu/backend.yml example
agent-port: 8081

Security configuration flags

cert-file
description Path to the primary backend certificate file, as well as specifies a fallback SSL/TLS certificate if the flag dashboard-cert-file is not use. This certificate secures communications between Sensu Dashboard and end user web browsers, as well as communication between sensuctl and the Sensu API.
type String
default ""
example 
# Command line example
sensu-backend start --cert-file /path/to/ssl/cert.pem

# /etc/sensu/backend.yml example
cert-file: "/path/to/ssl/cert.pem"
key-file
description Path to the primary backend key file, as well as specifies a fallback SSL/TLS key if the flag dashboard-key-file is not use. This key secures communication between Sensu Dashboard and end user web browsers, as well as communication between sensuctl and the Sensu API.
type String
default ""
example 
# Command line example
sensu-backend start --key-file /path/to/ssl/key.pem

# /etc/sensu/backend.yml example
key-file: "/path/to/ssl/key.pem"
trusted-ca-file
description Path to the primary backend CA file, as well as specifies a fallback SSL/TLS certificate authority in PEM format used for etcd client (mutual TLS) communication if the etcd-trusted-ca-file is not used. This CA file is used in communication between Sensu Dashboard and end user web browsers, as well as communication between sensuctl and the Sensu API.
type String
default ""
example 
# Command line example
sensu-backend start --trusted-ca-file /path/to/trusted-certificate-authorities.pem

# /etc/sensu/backend.yml example
trusted-ca-file: "/path/to/trusted-certificate-authorities.pem"
insecure-skip-tls-verify
description Skip SSL verification. WARNING: This configuration flag is intended for use in development systems only. Do not use this flag in production.
type Boolean
default false
example 
# Command line example
sensu-backend start --insecure-skip-tls-verify

# /etc/sensu/backend.yml example
insecure-skip-tls-verify: true

Dashboard configuration flags

dashboard-cert-file
description Dashboard TLS certificate in PEM format. This certificate secures communication with the Sensu Dashboard. If the dashboard-cert-file is not provided in the backend configuration, Sensu uses the certificate specified in the cert-file flag for the dashboard.
type String
default ""
example 
# Command line example
sensu-backend start --dashboard-cert-file /path/to/tls/cert.pem

# /etc/sensu/backend.yml example
dashboard-cert-file: "/path/to/tls/cert.pem"
dashboard-key-file
description Dashboard TLS certificate key in PEM format. This key secures communication with the Sensu Dashboard. If the dashboard-key-file is not provided in the backend configuration, Sensu uses the key specified in the key-file flag for the dashboard.
type String
default ""
example 
# Command line example
sensu-backend start --dashboard-key-file /path/to/tls/key.pem

# /etc/sensu/backend.yml example
dashboard-key-file: "/path/to/tls/key.pem"
dashboard-host
description Dashboard listener host
type String
default [::]
example 
# Command line example
sensu-backend start --dashboard-host 127.0.0.1

# /etc/sensu/backend.yml example
dashboard-host: "127.0.0.1"
dashboard-port
description Dashboard listener port
type Integer
default 3000
example 
# Command line example
sensu-backend start --dashboard-port 4000

# /etc/sensu/backend.yml example
dashboard-port: 4000

Datastore and cluster configuration flags

etcd-advertise-client-urls
description List of this member’s client URLs to advertise to the rest of the cluster.
type List
default http://localhost:2379
example 
# Command line examples
sensu-backend start --etcd-advertise-client-urls http://localhost:2378,http://localhost:2379
sensu-backend start --etcd-advertise-client-urls http://localhost:2378 --etcd-advertise-client-urls http://localhost:2379

# /etc/sensu/backend.yml example
etcd-advertise-client-urls:
  - http://localhost:2378
  - http://localhost:2379
etcd-cert-file
description Path to the etcd client API TLS cert file. Secures communication between the embedded etcd client API and any etcd clients.
type String
default ""
example 
# Command line example
sensu-backend start --etcd-cert-file ./client.pem

# /etc/sensu/backend.yml example
etcd-cert-file: "./client.pem"
etcd-client-cert-auth
description Enable client cert authentication
type Boolean
default false
example 
# Command line example
sensu-backend start --etcd-client-cert-auth

# /etc/sensu/backend.yml example
etcd-client-cert-auth: true
etcd-initial-advertise-peer-urls
description List of this member’s peer URLs to advertise to the rest of the cluster
type List
default http://127.0.0.1:2380
example 
# Command line examples
sensu-backend start --etcd-listen-peer-urls https://10.0.0.1:2380,https://10.1.0.1:2380
sensu-backend start --etcd-listen-peer-urls https://10.0.0.1:2380 --etcd-listen-peer-urls https://10.1.0.1:2380

# /etc/sensu/backend.yml example
etcd-listen-peer-urls:
  - https://10.0.0.1:2380
  - https://10.1.0.1:2380
etcd-initial-cluster
description Initial cluster configuration for bootstrapping
type String
default default=http://127.0.0.1:2380
example 
# Command line example
sensu-backend start --etcd-initial-cluster backend-0=https://10.0.0.1:2380,backend-1=https://10.1.0.1:2380,backend-2=https://10.2.0.1:2380

# /etc/sensu/backend.yml example
etcd-initial-cluster: "backend-0=https://10.0.0.1:2380,backend-1=https://10.1.0.1:2380,backend-2=https://10.2.0.1:2380"
etcd-initial-cluster-state
description Initial cluster state (new or existing)
type String
default new
example 
# Command line example
sensu-backend start --etcd-initial-cluster-state existing

# /etc/sensu/backend.yml example
etcd-initial-cluster-state: "existing"
etcd-initial-cluster-token
description Initial cluster token for the etcd cluster during bootstrap
type String
default ""
example 
# Command line example
sensu-backend start --etcd-initial-cluster-token sensu

# /etc/sensu/backend.yml example
etcd-initial-cluster-token: "sensu"
etcd-key-file
description Path to the etcd client API TLS key file. Secures communication between the embedded etcd client API and any etcd clients.
type String
example 
# Command line example
sensu-backend start --etcd-key-file ./client-key.pem

# /etc/sensu/backend.yml example
etcd-key-file: "./client-key.pem"
etcd-listen-client-urls
description List of URLs to listen on for client traffic
type List
default http://127.0.0.1:2379
example 
# Command line examples
sensu-backend start --etcd-listen-client-urls https://10.0.0.1:2379,https://10.1.0.1:2379
sensu-backend start --etcd-listen-client-urls https://10.0.0.1:2379 --etcd-listen-client-urls https://10.1.0.1:2379

# /etc/sensu/backend.yml example
etcd-listen-client-urls:
  - https://10.0.0.1:2379
  - https://10.1.0.1:2379
etcd-listen-peer-urls
description List of URLs to listen on for peer traffic
type List
default http://127.0.0.1:2380
example 
# Command line examples
sensu-backend start --etcd-listen-peer-urls https://10.0.0.1:2380,https://10.1.0.1:2380
sensu-backend start --etcd-listen-peer-urls https://10.0.0.1:2380 --etcd-listen-peer-urls https://10.1.0.1:2380

# /etc/sensu/backend.yml example
etcd-listen-peer-urls:
  - https://10.0.0.1:2380
  - https://10.1.0.1:2380
etcd-name
description Human-readable name for this member
type String
default default
example 
# Command line example
sensu-backend start --etcd-name backend-0

# /etc/sensu/backend.yml example
etcd-name: "backend-0"
etcd-peer-cert-file
description Path to the peer server TLS cert file
type String
example 
# Command line example
sensu-backend start --etcd-peer-cert-file ./backend-0.pem

# /etc/sensu/backend.yml example
etcd-peer-cert-file: "./backend-0.pem"
etcd-peer-client-cert-auth
description Enable peer client cert authentication
type Boolean
default false
example 
# Command line example
sensu-backend start --etcd-peer-client-cert-auth

# /etc/sensu/backend.yml example
etcd-peer-client-cert-auth: true
etcd-peer-key-file
description Path to the etcd peer API TLS key file. Secures communication between etcd cluster members.
type String
example 
# Command line example
sensu-backend start --etcd-peer-key-file ./backend-0-key.pem

# /etc/sensu/backend.yml example
etcd-peer-key-file: "./backend-0-key.pem"
etcd-peer-trusted-ca-file
description Path to the etcd peer API server TLS trusted CA file. This certificate secures communication between etcd cluster members.
type String
example 
# Command line example
sensu-backend start --etcd-peer-trusted-ca-file ./ca.pem

# /etc/sensu/backend.yml example
etcd-peer-trusted-ca-file: "./ca.pem"
etcd-trusted-ca-file
description Path to the client server TLS trusted CA cert file. Secures communication with the etcd client server.
type String
default ""
example 
# Command line example
sensu-backend start --etcd-trusted-ca-file ./ca.pem

# /etc/sensu/backend.yml example
etcd-trusted-ca-file: "./ca.pem"
no-embed-etcd
description Don’t embed etcd, use external etcd instead
type Boolean
default false
example 
# Command line example
sensu-backend start --no-embed-etcd

# /etc/sensu/backend.yml example
no-embed-etcd: true

etcd-cipher-suites
description List of allowed cipher suites for etcd TLS configuration. Sensu supports TLS 1.0-1.2 cipher suites as listed in the Go TLS documentation. You can use this attribute to defend your TLS servers from attacks on weak TLS ciphers. The default cipher suites are determined by Go, based on the hardware used. NOTE: To use TLS 1.3, add the following environment variable: GODEBUG="tls13=1".
recommended 
etcd-cipher-suites:
  - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
  - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
type List
example 
# Command line examples
sensu-backend start --etcd-cipher-suites TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
sensu-backend start --etcd-cipher-suites TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 --etcd-cipher-suites TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

# /etc/sensu/backend.yml example
etcd-cipher-suites:
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Advanced configuration options

etcd-max-request-bytes
description Maximum etcd request size in bytes that can be sent to an etcd server by a client. Increasing this value allows etcd to process events with large outputs at the cost of overall latency. WARNING: Use with caution. This configuration option requires familiarity with etcd. Improper use of this option can result in a non-functioning Sensu instance.
type Integer
default 1572864
example 
# Command line example
sensu-backend start --etcd-max-request-bytes 1572864

# /etc/sensu/backend.yml example
etcd-max-request-bytes: 1572864
etcd-quota-backend-bytes
description Maximum etcd database size in bytes. Increasing this value allows for a larger etcd database at the cost of performance. WARNING: Use with caution. This configuration option requires familiarity with etcd. Improper use of this option can result in a non-functioning Sensu instance.
type Integer
default 4294967296
example 
# Command line example
sensu-backend start --etcd-quota-backend-bytes 4294967296

# /etc/sensu/backend.yml example
etcd-quota-backend-bytes: 4294967296
eventd-buffer-size
description Number of incoming events that can be buffered before being processed by an eventd worker. WARNING: Modify with caution. Increasing this value may result in higher memory usage.
type Integer
default 100
example 
# Command line example
sensu-backend start --eventd-buffer-size 100


# /etc/sensu/backend.yml example
eventd-buffer-size: 100
eventd-workers
description Number of workers spawned for processing incoming events that are stored in the eventd buffer. WARNING: Modify with caution. Increasing this value may result in higher CPU usage.
type Integer
default 100
example 
# Command line example
sensu-backend start --eventd-workers 100

# /etc/sensu/backend.yml example
eventd-workers: 100
keepalived-buffer-size
description Number of incoming keepalives that can be buffered before being processed by a keepalived worker. WARNING: Modify with caution. Increasing this value may result in higher memory usage.
type Integer
default 100
example 
# Command line example
sensu-backend start --keepalived-buffer-size 100

# /etc/sensu/backend.yml example
keepalived-buffer-size: 100
keepalived-workers
description Number of workers spawned for processing incoming keepalives that are stored in the keepalived buffer. WARNING: Modify with caution. Increasing this value may result in higher CPU usage.
type Integer
default 100
example 
# Command line example
sensu-backend start --keepalived-workers 100

# /etc/sensu/backend.yml example
keepalived-workers: 100
pipelined-buffer-size
description Number of events to handle that can be buffered before being processed by a pipelined worker. WARNING: Modify with caution. Increasing this value may result in higher memory usage.
type Integer
default 100
example 
# Command line example
sensu-backend start --pipelined-buffer-size 100

# /etc/sensu/backend.yml example
pipelined-buffer-size: 100
pipelined-workers
description Number of workers spawned for handling events through the event pipeline that are stored in the pipelined buffer. WARNING: Modify with caution. Increasing this value may result in higher CPU usage.
type Integer
default 100
example 
# Command line example
sensu-backend start --pipelined-workers 100

# /etc/sensu/backend.yml example
pipelined-workers: 100

Event logging

LICENSED TIER: Unlock event logging in Sensu Go with a Sensu license. To activate your license, see the getting started guide.

All Sensu events can be optionally logged to a file in JSON format. This file can then be used as an input source for your favorite data lake solution. Using the event logging functionality provides better performance and reliability than using event handlers.

event-log-file
description Path to the event log file. WARNING: The log file should be located on a local drive. Logging directly to network drives is not supported.
type String
example 
# Command line example
sensu-backend start --event-log-file /var/log/sensu/events.log


# /etc/sensu/backend.yml example
event-log-file: "/var/log/sensu/events.log"
event-log-buffer-size
description Buffer size of the event logger. Corresponds to the maximum number of events kept in memory in case the log file is temporarily unavailable or more events have been received than what can be written to the log file.
type Integer
default 100000
example 
# Command line example
sensu-backend start --event-log-buffer-size 100000


# /etc/sensu/backend.yml example
event-log-buffer-size: 100000

Log rotation

Event logging supports log rotation via the SIGHUP signal. The current log file should first be renamed (moved) and then, this signal should be sent to the sensu-backend process so it re-creates a new log file and starts logging to it. Here are some logrotate sample configurations:

systemd
/var/log/sensu/events.log
{
  rotate 3
  hourly
  missingok
  notifempty
  compress
  postrotate
    /bin/systemctl reload sensu-backend.service > /dev/null 2>/dev/null || true
  endscript
}
sysvinit
/var/log/sensu/events.log
{
  rotate 3
  hourly
  missingok
  notifempty
  compress
  postrotate
    kill -HUP `cat /var/run/sensu/sensu-backend.pid 2> /dev/null` 2> /dev/null || true
  endscript
}