0

I have two Spring applications that chug along nicely, nothing special happening, and then out of the blue the applications restart; journalctl shows these lines:

Jan 15 04:43:11 nyota.softworks.nl systemd[1]: Stopping calendaraggregator.service...
Jan 15 04:43:11 nyota.softworks.nl systemd[1]: calendaraggregator.service: Deactivated successfully.
Jan 15 04:43:11 nyota.softworks.nl systemd[1]: Stopped calendaraggregator.service.
Jan 15 04:43:11 nyota.softworks.nl systemd[1]: calendaraggregator.service: Consumed 3min 313ms CPU time.
Jan 15 04:43:11 nyota.softworks.nl systemd[1]: Started calendaraggregator.service.
Jan 15 10:41:43 nyota.softworks.nl systemd[1]: Stopping calendaraggregator.service...
Jan 15 10:41:43 nyota.softworks.nl systemd[1]: calendaraggregator.service: Deactivated successfully.
Jan 15 10:41:43 nyota.softworks.nl systemd[1]: Stopped calendaraggregator.service.
Jan 15 10:41:43 nyota.softworks.nl systemd[1]: calendaraggregator.service: Consumed 1min 48.505s CPU time.
Jan 15 10:41:43 nyota.softworks.nl systemd[1]: Started calendaraggregator.service.
Jan 15 18:17:17 nyota.softworks.nl systemd[1]: Stopping calendaraggregator.service...
Jan 15 18:17:17 nyota.softworks.nl systemd[1]: calendaraggregator.service: Deactivated successfully.
Jan 15 18:17:17 nyota.softworks.nl systemd[1]: Stopped calendaraggregator.service.
Jan 15 18:17:17 nyota.softworks.nl systemd[1]: calendaraggregator.service: Consumed 2min 24.748s CPU time.
Jan 15 18:17:17 nyota.softworks.nl systemd[1]: Started calendaraggregator.service.

But the systemctl configuration AFAIK does not do a restart, not even after failure (took that out to debug this).

[Service]
WorkingDirectory=/home/calendaraggregator/
User=calendaraggregator
Group=calendaraggregator
ExecStart=/bin/bash /home/calendaraggregator/run.sh

Is there anyway to see what triggered the shutdown?

The log of the application is not of much use, it only says INFO org.springframework.core.log.CompositeLog [SpringApplicationShutdownHook] Stopping. without any reason / exception. Also, the application is restarted externally, since the main is called again (I logged that) and AFAIK I do not have systemd configured to restart.

Update 2024-05-06

I've rewritten one of the applications from Spring to Quarkus, to exclude the framework being a factor. I've moved the system configuration from /etc to ~/.config. But the applications keep restarting.

Interestingly, even though the applications are now configured under separate users, each with its ~/.config/systemd/user/service file, they keep restarting all at exactly the same time! So there must be some system level factor triggering the restart.

Improve this question
9
  • What do the logs of the application say? Spring applications usually have pretty detailed and configurable logging,
    – zwets
    Commented Jan 15 at 21:46
  • See additional information in the question; the logging is not clear, all it says that it is shutting down.
    – tbeernot
    Commented Jan 17 at 16:22
  • In terms of journalctl's output ... is that all there is, or did you filter out just messages pertaining to your service?
    – tink
    Commented May 6 at 21:51
  • That is exactly the output of journalctl. Because the services run per user, that is a natural filter.
    – tbeernot
    Commented May 7 at 7:25
  • But your remark did get me started on a path involving certbot, that could very well be the connecting factor. I am (also) having issues reverse-proxying websockets and maybe something from those attempts with Apache and nginx is creating havoc
    – tbeernot
    Commented May 7 at 7:35

1 Answer 1

Reset to default
0

I've figured it out: in an attempt to include a websocket in a reverse proxy, certbot created a faulty configuration (using a not running reverse proxy server). Everytime certbot ran, this config apparently caused it to execute its post events. I have no idea why, no new certificate was fetched so why would it, but it resulted in the restart of all systemd services that uses the certificate.

So even though the certificate was not renewed (usually the hint that a restart is needed) the services were still restarted. I was so barking up the wrong tree.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .