Starting Tomcat 8.5 using systemd on Centos 7

Question

I installed Tomcat 8.5 and I am able to start it successfully manually, for example:

su tomcat startup.sh

works properly and tomcat is able to serve on port 8080.

I need to automate the startup, so I created the file /etc/systemd/system/tomcat-prod.service:

[Unit]
Description=Tomcat 8.5 servlet container - Production
After=network.target

[Service]
Type=forking

ExecStart=/opt/appservers/production/apache-tomcat-8.5.37/bin/startup.sh
ExecStop=/opt/appservers/production/apache-tomcat-8.5.37/bin/shutdown.sh

User=tomcat
Group=tomcat   

[Install]
WantedBy=multi-user.target

The file is bare on purpose, all my env variables are in setenv.sh. I tried running it using:

# systemctl daemon-reload
# systemctl start tomcat-prod

For some reason above results in Tomcat exiting just after startup. I get no log anywhere that explains why. Tomcat creates an empty catalina.out file on startup and systemctl status tomcat-prod gives only the following:

tomcat-prod.service - Tomcat 8.5 servlet container - Production Loaded: loaded (/etc/systemd/system/tomcat-prod.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2019-01-04 08:08:27 UTC; 3s ago Process: 3583 ExecStop=/opt/appservers/production/apache-tomcat-8.5.37/shutdown.sh (code=exited, status=203/EXEC) Process: 3569 ExecStart=/opt/appservers/production/apache-tomcat-8.5.37/bin/startup.sh (code=exited, status=0/SUCCESS) Main PID: 3581 (code=exited, status=0/SUCCESS)

Jan 04 08:08:27 *.net startup.sh[3569]: Existing PID file found during start. Jan 04 08:08:27 *.net startup.sh[3569]: Removing/clearing stale PID file.

Can someone point me in the right direction?

Answer 1

Systemd keeps track of the "main process" of every service, so that it could accurately know whether the service is still alive or not. When the main process exits, the service manager assumes that the service has crashed or just stopped on its own. And when the service has stopped, all its leftover processes are automatically killed, to provide a clean state for future starts.

For Type=forking services, determining which is the "main" process can be somewhat difficult, especially when it's hidden behind layers and layers and layers of shell scripts. The default heuristics in systemd work well for regular daemons, but in your case they likely misdetect one of those shellscript layers as the main process, instead of tomcat itself. So as soon as the script ends, the service is assumed to have stopped and its leftovers (including the real daemon) are killed.

You have two options:

• Explicitly specify the main process using a "pidfile". This is the traditional method used by SysVinit. Your startup scripts already store the Tomcat daemon's process ID in some file, so your tomcat.service unit needs to reference it using the PIDFile= option. With this, the service manager will not need autodetection heuristics and will just track the provided PID as the main process.

• Get rid of the shellscripts entirely and start Tomcat directly from the ExecStart= option. This is the recommended method for systemd, and you can find example .service units already written by various distributions' packagers. For example, Arch Linux (Tomcat 8 with Jsvc).

Answer 2

No doubt grawity is already aware of the systemd horror mentioned elsewhere in which the author writes "we already have a better dæmon supervisor. It's systemd" - which would mean we don't need jsvc either. On that page, however, the author does add a "Bonus Track": "Notice that M. Benjamin invokes catalina.sh with run rather than with start. There is actually another path through catalina.sh, for systems that aim to run dæmons, not to spawn them." Adding a few tweaks to Jonathan de Boyne Pollard's script led me to the following which a) is not too hard to write, b) appears to eliminate unnecessary layers in functionality, the unnecessary use of PIDs and of additional daemonizing commands and c) is clear in its output:

# file: /usr/local/lib/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat 9
After=network.target

[Service]
User=tomcat
Group=tomcat

ExecStart=/opt/apache/tomcat/bin/catalina.sh run
Type=simple

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

For additional options refer to the top of the catalina.sh script where it reads:

#   Do not set the variables in this script. Instead put them into a script
#   setenv.sh in CATALINA_BASE/bin to keep your customizations separate.