Why is systemd stopping service immediately after it is started?

Question

I have a C++ based application which I'm running(executable) as a daemon with systemd.

Unit File:

[Unit]
Description=Console Service
After=network.target

[Service]
Environment="USER=ubuntu" "Path=/home/ubuntu/console/bin" 
WorkingDirectory=/home/ubuntu/console/bin
ExecStart=/bin/sh -ec "exec /sbin/start-stop-daemon -S -c ${USER} -d ${Path} --pidfile=/var/run/console.pid --oknodo  --exec consoleExecutable " #2>/dev/null
ExecStop=/bin/sh -ec "exec /sbin/start-stop-daemon -K --quiet -c ${USER} -d ${Path} --pidfile=/var/run/console.pid  --retry=TERM/30/KILL/5 --oknodo --exec consoleExecutable" #2>/dev/null
Restart=on-failure
RemainAfterExit=no
TimeoutStopSec=10
SuccessExitStatus=0 1
TimeoutStartSec=360

[Install]
WantedBy=multi-user.target

When I issue start command the service is starting up, but then it immediately receives a shutdown signal and then exits. Any clue, what is happening?

sudo systemctl status console.service
● console.service - Console Service
   Loaded: loaded (/etc/systemd/system/console.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm) since Mon 2017-09-25 19:58:58 UTC; 1s ago
  Process: 8706 ExecStop=/bin/sh -ec exec /sbin/start-stop-daemon -K --quiet -c ${USER} -d ${Path} --pidfile=/var/run/console.pid  --retry=TERM/30/KILL/5 --oknodo --exec consoleExecutable #2>/dev/null (code=exited, status=0/SUCCESS)
  Process: 8701 ExecStart=/bin/sh -ec exec /sbin/start-stop-daemon -S -c ${USER} -d ${Path} --pidfile=/var/run/console.pid --oknodo  --exec consoleExecutable  #2>/dev/null (code=exited, status=0/SUCCESS)
 Main PID: 8701 (code=exited, status=0/SUCCESS)
    Tasks: 1
   Memory: 1.8M
      CPU: 53ms
   CGroup: /system.slice/console.service
           └─8705 consoleExecutable

Sep 25 19:58:58 mgmt1 systemd[1]: Started Console Service.

sudo systemctl status console.service
● console.service - Console Service
   Loaded: loaded (/etc/systemd/system/console.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2017-09-25 19:59:01 UTC; 947ms ago
  Process: 8706 ExecStop=/bin/sh -ec exec /sbin/start-stop-daemon -K --quiet -c ${USER} -d ${Path} --pidfile=/var/run/console.pid  --retry=TERM/30/KILL/5 --oknodo --exec consoleExecutable #2>/dev/null (code=exited, status=0/SUCCESS)
  Process: 8701 ExecStart=/bin/sh -ec exec /sbin/start-stop-daemon -S -c ${USER} -d ${Path} --pidfile=/var/run/console.pid --oknodo  --exec consoleExecutable  #2>/dev/null (code=exited, status=0/SUCCESS)
 Main PID: 8701 (code=exited, status=0/SUCCESS)

Sep 25 19:58:58 mgmt1 systemd[1]: Started Console Service.

Answer 1

Environment="USER=ubuntu" "Path=/home/ubuntu/console/bin" 
WorkingDirectory=/home/ubuntu/console/bin
ExecStart=/bin/sh -ec "exec /sbin/start-stop-daemon -S -c ${USER} -d ${Path} --pidfile=/var/run/console.pid --oknodo  --exec consoleExecutable " #2>/dev/null
ExecStop=/bin/sh -ec "exec /sbin/start-stop-daemon -K --quiet -c ${USER} -d ${Path} --pidfile=/var/run/console.pid  --retry=TERM/30/KILL/5 --oknodo --exec consoleExecutable" #2>/dev/null

This is almost worthy of the systemd House of Horror. Were it not the case that there's a horror story already in there that does this.

Do not use start-stop-daemon in a service unit to do all of the things that a service unit already does. With unnecessary PID files and the wrongheaded assumption that ExecStart accepts shell syntax comments, no less.

And do not do what the other answer says and try to bodge it with Type=forking. That makes things worse, not better.

The nonsense with start-stop-daemon is why things are going wrong. Because the process running start-stop-daemon does not become the service, but in fact exits pretty much imediately, systemd is thinking that your service is terminating. In your first systemctl status output, you can see that systemd is in the middle of sending SIGTERM to clean up all left-over running processes after running the ExecStop action, which is what it does when it thinks that a service has terminated.

Just do things simply:

Type=simple
WorkingDirectory=/home/ubuntu/console/bin
User=ubuntu
ExecStart=/home/ubuntu/console/bin/consoleExecutable

No ExecStop nor Environment is actually required.

Further reading

• Jonathan de Boyne Pollard (2015). You really don't need to daemonize. Really.. The systemd House of Horror.
• Jonathan de Boyne Pollard (2016). If you have two services, define two services.. The systemd House of Horror.
• Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.
• Systemd kills service immediately after start

Answer 2

systemd thinks the pid of your daemon is 8701 which is the pid of start-stop-daemon. The pid of your daemon is 8705. You need to set Type=forking (under [Service]) in your unit file. The default is Type=simple which assumes the process does not fork.

However, why are you using this /bin/sh -ec exec /sbin/start-stop-daemon monstrosity? systemd can handle everything start-stop-daemon does.