Systemd watchdog with Python

This discussion comes about after seeing several long running daemons fall over “inelegantly” and seeking solutions.

There are several ways to control the operation of system daemons – programs that run relentlessly and unattended. Here we’ll use systemd and its management tool systemctl. The target environment is a raspberry Pi 4 running a Python data monitoring program. Python isn’t my preferred environment, but for this kind of application it’s a sensible choice.

The program and the daemon

The program in this case is a simple Python script that waits for MQTT messages and sends that data to an SQL database. The program can be tested of course using:

python3 myscript.py

To automate this with systemd on the Raspberry Pi, we create a service file mqtt2mysql.service in /etc/systemd/system.
Basic contents of this file are:

[Unit]
Description=mqtt2mysql

[Service]
User=pi
Restart=on-failure
RestartSec=5s
ExecStart=python3 /path-to-script/mqtt2mysql.py
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target

All of this is pretty standard noting that
Environment=PYTHONUNBUFFERED=1
will make sure that Python’s output will be correctly send to journalctl and systemd for logging.

Assuming that the Python program works correctly, it can be controlled as a daemon using systemctl commands:

systemctl commands

systemctl start myscript.service - Start the daemon.
systemctl stop myscript.service - Stop it.
systemctl restart myscript.service - Restart it.
systemctl status myscript.service - Status of the daemon.
systemctl daemon-reload - Refresh the daemon scripts. (Run each time a system file is changed)
systemctl enable myscript.service - Make this service auto start at boot time.
systemctl disable myscript.service - Prevent the service auto starting.

(Watch the running outcome in syslog or journalctl)

Watchdog – listening

Watchdog is a powerful function available to systemd.
With the following lines added to the service file, systemd will be expecting regular watchdog messages from the target service. If these don’t appear in the given WatchdogSec time, failure is assumed and the service will be restarted.

Type=notify
WatchdogSec=45

Watchdog – sdnotify

In our example, the python script myscript.py will be expected to send a regular Watchdog message in time before systemd (WatchdogSec) expects it. In my example myscript is supposed to have an activity every 30 seconds.

There are several Python implementations of sdnotify, all pretty similar that will perform this function in your script.
https://github.com/bb4242/sdnotify

sudo pip3 install sdnotify

Use sdnotify commands to talk to systemd.

# NB THIS IS NOT A USEFUL PROGRAM - syntax demo ONLY
import sdnotify

n = sdnotify.SystemdNotifier()
n.notify("READY=1")

count = 1
while True:
	print("Running... {}".format(count))
	n.notify("STATUS=Count is {}".format(count))
	count += 1
        if (gotdata==true):
            n.notify("WATCHDOG=1")

If n.notify(“WATCHDOG=1”) is not communicated to systemd within the 45seconds defined, then systemd will take charge and restart the service.