How to automatically restart a linux background process if it fails?

Question

I have a process which is executed by init.d script on the background. Eg:

case "$1" in 
    start)
       /bin/myprocess &
    stop)
       killall myprocess
    restart)
       killall myprocess
       /bin/myprocess &
esac

In certain conditions, myprocess can fail and return. Is there any (standard) way how to detect its fail and restart in automatically?

Answer 1

The easiest way would be to add it to /etc/inittab, which is designed to do this sort of thing:

respawn If the process does not exist, start the process. Do not wait for its termination (continue scanning the /etc/inittab file). Restart the process when it dies. If the process exists, do nothing and continue scanning the /etc/inittab file.

For example, you could do this:

# Run my stuff
myprocess:2345:respawn:/bin/myprocess

Answer 2

Buildroot has three possible init systems, so there are three ways to do this:

BusyBox init

With this, one adds an entry to /etc/inittab.

::respawn:/bin/myprocess

Note that BusyBox init has an idiosyncratic /etc/inittab format. The second field is meaningless, and the first field is not an ID but a device basename.

Linux "System V" init

Again, one adds an entry to /etc/inittab.

myprocess:2345:respawn:/bin/myprocess

systemd

One writes a unit file in, say, /etc/systemd/system/myprocess.service:

[Unit]
Description=My Process

[Service]
ExecStart=/bin/myprocess
Restart=always

[Install]
WantedBy=multi-user.target

Enable this to autostart at bootup with:

systemctl enable myprocess.service

Start it manually with:

systemctl start myprocess.service

Further reading

• "3.1.3 init system". The Buildroot user manual.

Answer 3

What about creating a subshell with a loop that calls constantly the same process?

If it ends, the next iteration of the loop goes on and starts it again.

(while true; do 
    /bin/myprocess
done) &

If the subshell dies, it's over though. The only possibility in that case would be to create another process (I'll call it necromancer) that checks whether yourprocess is alive, start it if it isn't and run this necromancer with cron, so that you can check that regularly.

Next step would be wondering what could happen if cron dies, but at some point you should feel safe and stop worrying.

Answer 4

You could make use of Monit . It's really easy to use and quite flexible. See for example this configuration for restarting the Tomcat process on failure.

check process tomcat with pidfile /var/run/tomcat.pid
   start program = "/etc/init.d/tomcat start"
   stop  program = "/etc/init.d/tomcat stop"
   if failed port 8080 type tcp then restart

It also has a lot of configuration examples for many use cases.

Answer 5

You may use restarter

start)
   restarter -c /bin/myprocess &
stop)
   pkill -f myprocess

On newer systems use systemd which solves all those trivial issues

Answer 6

If you're not a super user or root, and if your Linux system has Docker installed, then you can create a docker image of your process, using docker to restart your process if the system is rebooted.

File: docker-compose.yml

version: "3"
services:
  lserver:
    image: your_docker_image:latest
    ports:
    - 8080:8080   # just use 8080 as an example
    restart: always  # this is where your process can be guaranteed to restart

To start your docker container,

docker-compose up -d

I find it's easy to handle my-own-process with auto-restart if I am not a super user of the system.

For a sample example of how to create a docker image, here is a quick example:

File: Dockerfile

FROM alpine:3.5

RUN apk update && apk upgrade && rm -rf /var/cache/apk/*
WORKDIR /app
COPY my-process-server /app
RUN ln -s /app/my-process-server /usr/local/bin/my-process-server

EXPOSE 8080

CMD ["my-process-server"]

Answer 7

In my case, as a quick-fix, I modified and used the solution of @Trylks to wrap the program I was launching. I wanted it to end only on clean exit.

Should run in most shells:

#!/bin/sh

echo ""
echo "Use: $0 ./program"
echo ""

#eg="/usr/bin/apt update"

echo "Executing $1 ..."

EXIT_CODE=1
(while [ $EXIT_CODE -gt 0 ]; do
    $1
    # loops on error code: greater-than 0
    EXIT_CODE=$?
done) &

Answer 8

(Edit): Sometimes programs hang without quitting, for no apparent reason. (Yes, of course there's always a reason but it can take a lot of time and effort to find it, particularly if it's not your own code.)

The problem I had was that the process (a Python server) was hanging from time to time, so I needed to regularly kill and restart it. I did it with a cron task that runs every couple of hours. Here's the shell script:

#!/bin/sh

# This cron script restarts the server
# Look for a running instance of myapp.py
p=$(ps -eaf | grep "[m]yapp.py")
# Get the second item; the process number
n=$(echo $p | awk '{print $2}')
# If it's not empty, kill the process
if [ "$n" ]
then
   kill $n
fi
# Start a new instance
python3 myapp.py

Answer 9

The process can be launched with until loop as it's been answered here: How do I write a bash script to restart a process if it dies?. Until loop works with the requirement to run it the background, see sample:

#!/bin/sh
(until /bin/myprocess; do
    echo "'/bin/myprocess' crashed with exit code $?. Restarting..." >&2
    sleep 1
done) & 

Comparing to the @Trylks answer with while loop the last couldn't restart process when it's written in entrypoint.sh script of a Dockerfile. Despite it the until loop is working there too..