Running Scripts at Startup

And other Linux delights.

Often there is a need to configure processes to start on startup and to restart on failure. On Linux systems this process is controlled via the systemd init system and system manager.

Now, as with most things Linux, there are a lot of sources of information out there. Several of them are patchy and out-of-date. Also they tend to play around with things at the root level, e.g. which you need to do when configuring existing packages rather than your own code. This post will thus briefly look at implementing systemd daemons within safer user space.

systemctl –user

The systemctl command is the main command-line tool for interacting with systemd processes. There are several good guides to using systemctl including one here: https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units.

Lesser known is that there is a flag for systemctl commands that allows everything to be set in user space within messing with root administrator settings. This is the --user flag.

Working in user space avoids having to mess with system files in the /etc and /lib directories (/etc for user added services and /lib for installed package services). Instead, user services can be added within the current user’s home directory. Here is a guide to doing that: https://computingforgeeks.com/how-to-run-systemd-service-without-root-sudo/.

In short, you can add custom service definitions to a ~/.config/systemd/user/ directory. If this folder does not exist, you can easily make it via:

mkdir -p ~/.config/systemd/user/

How Services Work

Services are another name for daemons – processes that run in the background. In Linux these are defined via a <service_name>.service file. For custom user services, these can live in the ~/.config/systemd/user/ directory.

The <service_name>.service has a number of sections defined by square brackets with the main sections being [Unit], [Service] and [Install]. In the [Service] section you can define the command or script you want to run, and various options like environment variables and working directories. If you are familiar with tools such as Docker, these service files can be thought of as similar to composition and deployment files in those tools.

A basic service file might look like:

[Unit] 
Description=My Service - this is what it does

[Service] 
ExecStart=<path_to_command_or_script>/<command> <command-line-options>

[Install] 
WantedBy=default.target

For example, the ExecStart field could be: /home/$USER/my_app/my_executable.sh or python /home/$USER/my_app/.

A warning: the service may run in a more limited environment than your standard bash shell (i.e. the terminal). I found I needed to add some paths to my PATH variable so I could get the right conda command to run.

You can either code the service file within the user folder using a text editor:

nano ~/.config/systemd/user/<service_name>.service

Or use an Interactive Development Environment such as VSCode or PyCharm to write and then save in that folder.

Configuring for Start on Reboot

Once the service is defined, you need to restart the user system manager:

systemctl --user daemon-reload

And then enable the service:

systemctl --user enable --now <service_name>.service

You can check the status using:

systemctl --user status <service_name>.service

And use normal systemctl commands with the --user flag, such as disable, restart, start and stop.

Debugging

In all honesty, it took me a while to get a simple service running. Common problems involved changes in running environment, typos in names and paths, and failures to run. I found root services were often a little trickier and riskier to configure than user services.

A good tip is to check that your executable or command runs fine if the service fails. A very useful command is:

systemctl --user show-environment

that shows the environment in which the command is run. I found that additional paths were added to my normal shell environment on startup (via ~/.bashrc), so I added some changes to the PATH environment variable in the service file (e.g. Environment="PATH=<copy_and_paste_output_of_$PATH>"). I also found the WorkingDirectory variable useful to set the working directory as my project directory (both are defined in the [Service] section).

Another useful [Service] field is:

Restart=on-failure

This automatically restarts the service if it crashes.

You can also get the systemd logs via:

journalctl --user --unit <service_name>.service

Configuring useful logging is another task, which we will address on another day.

Shell Scripts

You can also bundle a set of commands to run into a shell script. To do this, first create a shell script (.sh file):

#!/bin/bash

# Script to setup environment and run webserver on startup
<command 1>
<command 2>

Then make this file executable:

sudo chmod +x ~/my_project/api_start.sh

Then, in the [Service] section, you can set:

ExecStart=/home/$USER/my_project/api_start.sh

Activating Conda

When implementing complex machine-learning environments, I often find using conda helps a lot. However, conda can complicate running custom processes as you need to activate and configure the environment beforehand.

Now, initially I had problems running the standard conda activate <env_name> command within a script being run as a service. To get around this I found that you can use the conda run command that allows certain commands to be run in a specific environment:

conda run -n <env_name> <command>

All of this can be very useful for configuring machine-learning APIs or background processes without worrying you are going to delete all the Linux services by accident.