And other Linux delights.
Often there is a need to configure processes to start on startup and to restart on failure. On Linux systems this process is controlled via the
systemd init system and system manager.
Now, as with most things Linux, there are a lot of sources of information out there. Several of them are patchy and out-of-date. Also they tend to play around with things at the root level, e.g. which you need to do when configuring existing packages rather than your own code. This post will thus briefly look at implementing
systemd daemons within safer user space.
systemctl command is the main command-line tool for interacting with
systemd processes. There are several good guides to using
systemctl including one here: https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units.
Lesser known is that there is a flag for
systemctl commands that allows everything to be set in user space within messing with root administrator settings. This is the
Working in user space avoids having to mess with system files in the
/lib directories (
/etc for user added services and
/lib for installed package services). Instead, user services can be added within the current user’s home directory. Here is a guide to doing that: https://computingforgeeks.com/how-to-run-systemd-service-without-root-sudo/.
In short, you can add custom service definitions to a
~/.config/systemd/user/ directory. If this folder does not exist, you can easily make it via:
mkdir -p ~/.config/systemd/user/
How Services Work
Services are another name for daemons – processes that run in the background. In Linux these are defined via a
<service_name>.service file. For custom user services, these can live in the
<service_name>.service has a number of sections defined by square brackets with the main sections being
[Install]. In the
[Service] section you can define the command or script you want to run, and various options like environment variables and working directories. If you are familiar with tools such as Docker, these service files can be thought of as similar to composition and deployment files in those tools.
A basic service file might look like:
[Unit] Description=My Service - this is what it does [Service] ExecStart=<path_to_command_or_script>/<command> <command-line-options> [Install] WantedBy=default.target
For example, the ExecStart field could be: /home/$USER/my_app/my_executable.sh or python /home/$USER/my_app/.
A warning: the service may run in a more limited environment than your standard bash shell (i.e. the terminal). I found I needed to add some paths to my PATH variable so I could get the right conda command to run.
You can either code the service file within the user folder using a text editor:
Or use an Interactive Development Environment such as VSCode or PyCharm to write and then save in that folder.
Configuring for Start on Reboot
Once the service is defined, you need to restart the user system manager:
systemctl --user daemon-reload
And then enable the service:
systemctl --user enable --now <service_name>.service
You can check the status using:
systemctl --user status <service_name>.service
And use normal
systemctl commands with the
--user flag, such as
In all honesty, it took me a while to get a simple service running. Common problems involved changes in running environment, typos in names and paths, and failures to run. I found root services were often a little trickier and riskier to configure than user services.
A good tip is to check that your executable or command runs fine if the service fails. A very useful command is:
systemctl --user show-environment
that shows the environment in which the command is run. I found that additional paths were added to my normal shell environment on startup (via
~/.bashrc), so I added some changes to the PATH environment variable in the service file (e.g.
Environment="PATH=<copy_and_paste_output_of_$PATH>"). I also found the
WorkingDirectory variable useful to set the working directory as my project directory (both are defined in the
[Service] field is:
This automatically restarts the service if it crashes.
You can also get the systemd logs via:
journalctl --user --unit <service_name>.service
Configuring useful logging is another task, which we will address on another day.
You can also bundle a set of commands to run into a shell script. To do this, first create a shell script (.sh file):
#!/bin/bash # Script to setup environment and run webserver on startup <command 1> <command 2>
Then make this file executable:
sudo chmod +x ~/my_project/api_start.sh
Then, in the
[Service] section, you can set:
When implementing complex machine-learning environments, I often find using
conda helps a lot. However, conda can complicate running custom processes as you need to activate and configure the environment beforehand.
Now, initially I had problems running the standard
conda activate <env_name> command within a script being run as a service. To get around this I found that you can use the
conda run command that allows certain commands to be run in a specific environment:
conda run -n <env_name> <command>
All of this can be very useful for configuring machine-learning APIs or background processes without worrying you are going to delete all the Linux services by accident.