How do I figure out why my systemctl service didn't start on CentOS 7?

Question

I'm using CentOS 7. How do I figure out why a service is failing to start? I have created this service

[rails@server ~]$ sudo cat /usr/lib/systemd/system/nodejs.service
[Unit]
Description=nodejs server

[Service]
User=rails
Group=rails
ExecStart=/home/rails/NodeJSserver/start.sh
ExecStop=/home/rails/NodeJSserver/stop.sh

[Install]
WantedBy=multi-user.target

The file points to this

[rails@server ~]$ cat /home/rails/NodeJSserver/start.sh
#!/bin/bash

forever start /home/rails/NodeJSserver/server.js

I can run this file just fine by itself. But when I try and run it as part of the service, I notice my nodeJS server isn't started. Even when I check "sudo systemctl --state=failed," I don't see any errors ...

[rails@server ~]$ sudo systemctl enable NodeJSserver
[rails@server ~]$ sudo systemctl start NodeJSserver
[rails@server ~]$
[rails@server ~]$
[rails@server ~]$ forever list
info:    No forever processes running
[rails@server ~]$
[rails@server ~]$
[rails@server ~]$ sudo systemctl --state=failed
  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION
● nginx.service                  loaded failed failed The nginx HTTP and reverse proxy server
● systemd-sysctl.service         loaded failed failed Apply Kernel Variables
● systemd-vconsole-setup.service loaded failed failed Setup Virtual Console

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

3 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

How do I figure out why my service failed to start?

Answer 1

Your service has no Type= specified in the [Service] section, so systemd assumes you meant Type=simple.

That means systemd will expect the process that was started with ExecStart= to keep running as long as the service is running. But it looks like your start.sh only runs one command and then exits. That is the forever command: forever start starts the target command as a daemon, or in other words, in the background. As soon as the forever start command completes, the shell running start.sh will exit.

At that point, systemd considers this service as failed. But wait, the control group assigned for that service still has a running process in it. "So," thinks systemd, "not only did it fail, but it also left a mess after itself. Can't have that." Since there is no KillMode= nor KillSignal= specified, systemd goes on with its defaults and sends a SIGTERM for any remaining processes in that control group, and if they don't stop in a timely manner, follows up with a SIGKILL. After that, your actual NodeJS process will be dead, guaranteed.

How to fix it

Since the command you run with ExecStart= will exit as soon as the actual server is started, you cannot use the default Type=simple. You must specify another service type.

You could use the Type=forking. With this type, man systemd.service recommends using a PIDFile= option, so if your NodeJS server creates a PID file for itself (or you add options to the forever command to make it create one for it), you should let systemd know where it will be.

[Service]
Type=forking
PIDFile=/absolute/path/to/nodejs.pid
User=rails
... <the rest as before>

---

If Type=forking does not work for you, then you could specify Type=oneshot with RemainAfterExit=yes.

That makes systemd just run the ExecStart= command when starting your service and ExecStop= when stopping it, and not care about anything else.

systemd will still remember whether the service was last set in a stopped or started state, though. So if you set another service to depend on this service, and then stop your NodeJS service manually, the other service won't stop automatically and will no doubt return errors when it cannot use your NodeJS service.

---

The third option is to skip the forever command entirely and let systemd do the job of restarting the NodeJS process. In that case, your entire nodejs.service unit would be:

[Unit]
Description=nodejs server

[Service]
User=rails
Group=rails
ExecStart=/home/rails/NodeJSserver/server.js
Restart=always

[Install]
WantedBy=multi-user.target

You could add other options.

For example, you could specify RestartSec=5 to specify a 5-second sleep before attempting to restart the service if it unexpectedly dies, to avoid hogging system resources by frequent restart attempts if your service keeps dying immediately after being restarted for some reason. (The default RestartSec= value is 100 ms.)

Or if you want the service to be restarted if it returns some specific exit status values, but consider it failed on others, there are options for that too.