monit

Official docs

https://mmonit.com/monit/documentation/monit.html

Main docs.

https://mmonit.com/wiki/Monit/ConfigurationExamples

Real-world configuration examples.

https://mmonit.com/wiki/Monit/FAQ

FAQ

Debugging monit

Run standalone:

monit -c /path/to/monitrc

Check syntax:

monit -t
monit -t -c /path/to/monitrc

Override log file location in config:

set log /var/log/monit.log

or with CLI option:

monit -l /var/log/monit.log

Do not go into background:

monit -I

Make verbose output:

monit -v
monit -vv

Full length commant may look like:

sudo monit -v status
sudo monit -Iv validate
sudo monit -Ivv -c /path/to/monitrc

To debug start/stop scripts write wrapper that redirects STDIO to file:

#/bin/sh
exec 1>my.log
exec 2>my.log
echo "$@"
exec "$@"

Limiting server to localhost:

set httpd port 2812
  use address localhost
  allow localhost
  allow admin:monit

Monit modes

• In active mode (the default), Monit will pro-actively monitor a service and in case of problems raise alerts and/or restart the service.
• In passive mode, Monit will passively monitor a service and will raise alerts, but will not try to fix a problem by executing start, stop or restart.
• In manual mode, Monit will enter active mode only if a service was started via Monit.

Example:

check process App with pidfile /var/run/app.pid
  start program = "/usr/bin/app start"
  stop program = "/usr/bin/app stop"
  mode passive

Alerting

check process memcached with match memcached
  start program = "/usr/bin/systemctl start memcached"
  stop program = "/usr/bin/systemctl stop memcached"
  if failed host 127.0.0.1 port 11211 protocol MEMCACHE then restart
  if cpu > 70% for 2 cycles then alert
  if cpu > 98% for 5 cycles then restart
  if 2 restarts within 3 cycles then timeout

check process myapp with pidfile /run/myapp.pid
  if does not exist then alert

check filesystem Ubuntu with path /dev/sda1
  if space usage > 90% then alert
check filesystem Home with path /dev/sda3
  if space usage > 90% then alert

check host app_name with address 127.0.0.1
  start "/sbin/start app_name"
  stop "/sbin/stop app_name"
  if failed port 80 protocol HTTP
    request /ok
    with timeout 5 seconds
    then restart

Running monit by systemd

[Unit]
Description=Pro-active monitoring utility for unix systems
After=network.target
Documentation=man:monit(1) https://mmonit.com/wiki/Monit/HowTo

[Service]
Type=simple
KillMode=process
ExecStart=/usr/local/bin/monit -I
ExecStop=/usr/local/bin/monit quit
ExecReload=/usr/local/bin/monit reload
Restart = on-abnormal
StandardOutput=null

[Install]
WantedBy=multi-user.target

https://mmonit.com/wiki/Monit/Systemd

Official docs.

Monitor systemd process

With pid-file:

check process nginx with pidfile /var/run/nginx.pid
  start program = "/bin/systemctl start nginx"
  stop program = "/bin/systemctl stop nginx"

If process without pid-file:

check program MyApp with path "systemctl --quiet is-active MyApp"
  if status != 0 then ...

There is a way to tell systemd to create pid-file if process doesn't to it itself:

ExecStartPost=/bin/sh -c "echo $MAINPID > /run/myapp.pid"

By matching program name:

check process MyApp matching 'myapp.*'
   start program = /bin/app
   stop program = something..

Can systemd replace monit?

No. Systemd has only basic check if process is running. Imaging that process is stuck. You need some active probes, like HTTP health endpoint.

Systemd has limited capabilities for notifying with OnFailure:

[Service]
Restart=always
RestartSecs=30

[Unit]
OnFailure=...

Reverse proxy for monit

Ngnix:

server {
  listen   80;
  server_name  my.server.name;
  location /monit/ {
    allow 127.0.0.1;
    allow 192.0.0.0/8;
    deny all;

    proxy_pass http://127.0.0.1:2812;
    proxy_set_header Host $host;
    rewrite ^/monit/(.*) /$1 break;
    proxy_ignore_client_abort on;
  }
}