13

My PostgreSQL 9.5 server on Debian 8 (Jessie) keeps exiting directly after being started via service postgresql start:

# service postgresql status
● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
   Active: active (exited) since Fr 2016-12-02 11:02:51 CET; 11min ago
  Process: 2360 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 2360 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service

Dez 05 16:29:24 dev systemd[1]: Starting PostgreSQL RDBMS...
Dez 05 16:29:24 dev systemd[1]: Started PostgreSQL RDBMS.

Note the active (exited) state. The server is down, e.g. I cannot connect to it via TCP or domain socket, and I can't find any associated processes.

However, when I start PostgreSQL manually it works:

# sudo -u postgres /usr/lib/postgresql/9.5/bin/postgres -d 3 -D /var/lib/postgresql/9.5/main/ -c config_file=/etc/postgresql/9.5/main/postgresql.conf 

2016-12-05 16:34:32 CET [1593-1] DEBUG:  postgres: PostmasterMain: initial environment dump:
2016-12-05 16:34:32 CET [1593-2] DEBUG:  -----------------------------------------
2016-12-05 16:34:32 CET [1593-3] DEBUG:     LC_PAPER=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-4] DEBUG:     LC_ADDRESS=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-5] DEBUG:     LC_MONETARY=C
2016-12-05 16:34:32 CET [1593-6] DEBUG:     TERM=xterm-256color
2016-12-05 16:34:32 CET [1593-7] DEBUG:     LC_NUMERIC=C
2016-12-05 16:34:32 CET [1593-8] DEBUG:     LC_TELEPHONE=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-9] DEBUG:     LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
2016-12-05 16:34:32 CET [1593-10] DEBUG:    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2016-12-05 16:34:32 CET [1593-11] DEBUG:    LC_IDENTIFICATION=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-12] DEBUG:    LANG=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-13] DEBUG:    LC_MEASUREMENT=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-14] DEBUG:    LC_TIME=C
2016-12-05 16:34:32 CET [1593-15] DEBUG:    LC_NAME=de_DE.UTF-8
2016-12-05 16:34:32 CET [1593-16] DEBUG:    SHELL=/bin/bash
2016-12-05 16:34:32 CET [1593-17] DEBUG:    MAIL=/var/mail/postgres
2016-12-05 16:34:32 CET [1593-18] DEBUG:    LOGNAME=postgres
2016-12-05 16:34:32 CET [1593-19] DEBUG:    USER=postgres
2016-12-05 16:34:32 CET [1593-20] DEBUG:    USERNAME=postgres
2016-12-05 16:34:32 CET [1593-21] DEBUG:    HOME=/var/lib/postgresql
2016-12-05 16:34:32 CET [1593-22] DEBUG:    SUDO_COMMAND=/usr/lib/postgresql/9.5/bin/postgres -d 3 -D /var/lib/postgresql/9.5/main/ -c config_file=/etc/postgresql/9.5/main/postgresql.conf
2016-12-05 16:34:32 CET [1593-23] DEBUG:    SUDO_USER=root
2016-12-05 16:34:32 CET [1593-24] DEBUG:    SUDO_UID=0
2016-12-05 16:34:32 CET [1593-25] DEBUG:    SUDO_GID=0
2016-12-05 16:34:32 CET [1593-26] DEBUG:    PGLOCALEDIR=/usr/share/locale
2016-12-05 16:34:32 CET [1593-27] DEBUG:    PGSYSCONFDIR=/etc/postgresql-common
2016-12-05 16:34:32 CET [1593-28] DEBUG:    LC_COLLATE=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-29] DEBUG:    LC_CTYPE=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-30] DEBUG:    LC_MESSAGES=en_US.UTF-8
2016-12-05 16:34:32 CET [1593-31] DEBUG:  -----------------------------------------
2016-12-05 16:34:32 CET [1593-32] DEBUG:  invoking IpcMemoryCreate(size=23887872)
2016-12-05 16:34:32 CET [1593-33] DEBUG:  mmap(25165824) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory
2016-12-05 16:34:32 CET [1593-34] DEBUG:  SlruScanDirectory invoking callback on pg_notify/0000
2016-12-05 16:34:32 CET [1593-35] DEBUG:  removing file "pg_notify/0000"
2016-12-05 16:34:32 CET [1593-36] DEBUG:  dynamic shared memory system will support 288 segments
2016-12-05 16:34:32 CET [1593-37] DEBUG:  created dynamic shared memory control segment 1173813643 (2316 bytes)
2016-12-05 16:34:32 CET [1593-38] DEBUG:  max_safe_fds = 983, usable_fds = 1000, already_open = 7
2016-12-05 16:34:32 CET [1593-39] LOG:  redirecting log output to logging collector process
2016-12-05 16:34:32 CET [1593-40] HINT:  Future log output will appear in directory "/var/log/postgresql".

(server is now up and running and can be connected to)

The logs in /var/log/postgresql don't contain anything suspicious. Here's the output for a working manual run:

2016-12-05 16:48:07 CET [1700-1] LOG:  database system was shut down at 2016-12-05 16:36:04 CET
2016-12-05 16:48:07 CET [1700-2] DEBUG:  checkpoint record is at 0/13823548
2016-12-05 16:48:07 CET [1700-3] DEBUG:  redo record is at 0/13823548; shutdown TRUE
2016-12-05 16:48:07 CET [1700-4] DEBUG:  next transaction ID: 0/14019; next OID: 35693
2016-12-05 16:48:07 CET [1700-5] DEBUG:  next MultiXactId: 1; next MultiXactOffset: 0
2016-12-05 16:48:07 CET [1700-6] DEBUG:  oldest unfrozen transaction ID: 617, in database 1
2016-12-05 16:48:07 CET [1700-7] DEBUG:  oldest MultiXactId: 1, in database 1
2016-12-05 16:48:07 CET [1700-8] DEBUG:  commit timestamp Xid oldest/newest: 0/0
2016-12-05 16:48:07 CET [1700-9] DEBUG:  transaction ID wrap limit is 2147484264, limited by database with OID 1
2016-12-05 16:48:07 CET [1700-10] DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 1
2016-12-05 16:48:07 CET [1700-11] DEBUG:  starting up replication slots
2016-12-05 16:48:07 CET [1700-12] DEBUG:  MultiXactId wrap limit is 2147483648, limited by database with OID 1
2016-12-05 16:48:07 CET [1700-13] LOG:  MultiXact member wraparound protections are now enabled
2016-12-05 16:48:07 CET [1700-14] DEBUG:  MultiXact member stop limit is now 4294914944 based on MultiXact 1
2016-12-05 16:48:07 CET [1700-15] DEBUG:  shmem_exit(0): 1 before_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-16] DEBUG:  shmem_exit(0): 3 on_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-17] DEBUG:  proc_exit(0): 2 callbacks to make
2016-12-05 16:48:07 CET [1700-18] DEBUG:  exit(0)
2016-12-05 16:48:07 CET [1700-19] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-20] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2016-12-05 16:48:07 CET [1700-21] DEBUG:  proc_exit(-1): 0 callbacks to make
2016-12-05 16:48:07 CET [1698-41] LOG:  database system is ready to accept connections
2016-12-05 16:48:07 CET [1705-1] DEBUG:  removing permanent stats file "pg_stat/db_12381.stat"
2016-12-05 16:48:07 CET [1704-1] LOG:  autovacuum launcher started
2016-12-05 16:48:07 CET [1704-2] DEBUG:  InitPostgres
2016-12-05 16:48:07 CET [1705-2] DEBUG:  removing permanent stats file "pg_stat/db_0.stat"
2016-12-05 16:48:07 CET [1704-3] DEBUG:  StartTransaction
2016-12-05 16:48:07 CET [1705-3] DEBUG:  removing permanent stats file "pg_stat/global.stat"
2016-12-05 16:48:07 CET [1704-4] DEBUG:  name: unnamed; blockState:       DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children: 
2016-12-05 16:48:07 CET [1704-5] DEBUG:  CommitTransaction
2016-12-05 16:48:07 CET [1704-6] DEBUG:  name: unnamed; blockState:       STARTED; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children: 
2016-12-05 16:48:07 CET [1705-4] DEBUG:  received inquiry for database 0
2016-12-05 16:48:07 CET [1705-5] DEBUG:  writing stats file "/var/run/postgresql/9.5-main.pg_stat_tmp/global.stat"
2016-12-05 16:48:07 CET [1705-6] DEBUG:  writing stats file "/var/run/postgresql/9.5-main.pg_stat_tmp/db_0.stat"
2016-12-05 16:48:07 CET [1701-1] DEBUG:  checkpointer updated shared memory configuration values

In case of an unsuccessful run via service postgresql start, no logs are written at all to /var/log/postgresql. Similarly, I can't find any PostgreSQL-related messages in /var/log/messages.

I've installed PostgreSQL via the official repository:

# apt-cache policy postgresql-9.5
postgresql-9.5:
  Installed: 9.5.5-1.pgdg80+1
  Candidate: 9.5.5-1.pgdg80+1
  Version table:
 *** 9.5.5-1.pgdg80+1 0
        500 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main amd64 Packages
        100 /var/lib/dpkg/status

All of this is on a VirtualBox VM.

Update 1

The systemd unit seems to belong to the postgresql-common package:

# find /lib/systemd -iname "*postgres*"
/lib/systemd/system/postgresql.service
/lib/systemd/system/[email protected]
/lib/systemd/system-generators/postgresql-generator

# dpkg -S postgresql.service
postgresql-common: /lib/systemd/system/postgresql.service

# dpkg -S [email protected]
postgresql-common: /lib/systemd/system/[email protected]

# dpkg -S postgresql-generator
postgresql-common: /lib/systemd/system-generators/postgresql-generator

Here's their content:

/lib/systemd/system/postgresql.service

# systemd service for managing all PostgreSQL clusters on the system. This
# service is actually a systemd target, but we are using a service since
# targets cannot be reloaded.

[Unit]
Description=PostgreSQL RDBMS

[Service]
Type=oneshot
ExecStart=/bin/true
ExecReload=/bin/true
RemainAfterExit=on

[Install]
WantedBy=multi-user.target

/lib/systemd/system/[email protected]

# systemd service template for PostgreSQL clusters. The actual instances will
# be called "postgresql@version-cluster", e.g. "[email protected]". The
# variable %i expands to "version-cluster", %I expands to "version/cluster".
# (%I breaks for cluster names containing dashes.)

[Unit]
Description=PostgreSQL Cluster %i
ConditionPathExists=/etc/postgresql/%I/postgresql.conf
PartOf=postgresql.service
ReloadPropagatedFrom=postgresql.service
Before=postgresql.service

[Service]
Type=forking
# @: use "postgresql@%i" as process name
ExecStart=@/usr/bin/pg_ctlcluster postgresql@%i --skip-systemctl-redirect %i start
ExecStop=/usr/bin/pg_ctlcluster --skip-systemctl-redirect -m fast %i stop
ExecReload=/usr/bin/pg_ctlcluster --skip-systemctl-redirect %i reload
PIDFile=/var/run/postgresql/%i.pid
SyslogIdentifier=postgresql@%i
# prevent OOM killer from choosing the postmaster (individual backends will
# reset the score to 0)
OOMScoreAdjust=-900
# restarting automatically will prevent "pg_ctlcluster ... stop" from working,
# so we disable it here. Also, the postmaster will restart by itself on most
# problems anyway, so it is questionable if one wants to enable external
# automatic restarts.
#Restart=on-failure
# (This should make pg_ctlcluster stop work, but doesn't:)
#RestartPreventExitStatus=SIGINT SIGTERM

[Install]
WantedBy=multi-user.target

/lib/systemd/system-generators/postgresql-generator

#!/bin/sh

# This systemd generator creates dependency symlinks that make all PostgreSQL
# clusters with "auto" in their start.conf file be started/stopped/reloaded
# when postgresql.service is started/stopped/reloaded.

set -eu

gendir="$1"
wantdir="$1/postgresql.service.wants"
pgservice="/lib/systemd/system/[email protected]"

mkdir -p "$wantdir"

for conf in /etc/postgresql/*/*/postgresql.conf; do
    test -e "$conf" || continue
    dir="${conf%/*}"

    # evaluate start.conf
    if [ -e "$dir/start.conf" ]; then
        start=$(sed 's/#.*$//; /^[[:space:]]*$/d; s/^\s*//; s/\s*$//' "$dir/start.conf")
    else
        start=auto
    fi
    [ "$start" = "auto" ] || continue

    verdir="${dir%/*}"
    version="${verdir##*/}"
    cluster="${dir##*/}"
    ln -s "$pgservice" "$wantdir/postgresql@$version-$cluster.service"
done

exit 0

postgresql-common was installed from the official PostgreSQL repository as a dependency of the postgresql-9.5 package:

# apt-cache policy postgresql-common
postgresql-common:
  Installed: 177.pgdg80+1
  Candidate: 177.pgdg80+1
  Version table:
 *** 177.pgdg80+1 0
        500 http://apt.postgresql.org/pub/repos/apt/ jessie-pgdg/main amd64 Packages
        100 /var/lib/dpkg/status
     165+deb8u1 0
        500 http://httpredir.debian.org/debian/ jessie/main amd64 Packages

Update 2

As requested, here's the output of strace -f service postgresql start on Pastebin (due to its size).

Improve this question
9
  • Do netstat -punta after starting it manually, is postgres listening on any port?
    – sysfiend
    Commented Dec 5, 2016 at 16:17
  • @Alex When I start PostgreSQL manually it listens and accepts connections via TCP (port 5432) and domain socket (/var/run/postgresql/.s.PGSQL.5432). When I start it via service none of that works.
    – Florian Brucker
    Commented Dec 5, 2016 at 16:21
  • You could give strace a try:
    – Oliver Rahner
    Commented Dec 12, 2016 at 8:51
  • strace -f service postgresql start
    – Oliver Rahner
    Commented Dec 12, 2016 at 8:58
  • @OliverRahner: Thanks for the idea, i've updated my answer with the output of strace.
    – Florian Brucker
    Commented Dec 12, 2016 at 10:02

2 Answers 2

Reset to default
8
+50

Theres a couple of things going on here. If you call a systemd service with an @ in it, its a signal to start an instance rather than main service. Putting @9.5-main is telling it the instance parameters to use.

The postgres service file you are seeing is nothing more than a placeholder. There is a directory on the system (its normally somewhere in /run) called postgresql.wants.d which the systemd will look for when postgresql.service is started.

Anything in that directory will be started along with it as postgresql.service is told it wants it.

That directory is filled through a shell script generator file as everything in /run is a tmpfs. Thats the shell script you quoted above.

If I had to hazard a guess whats going on there via the logic of the shell script, you probably have got the file /etc/postgresql/9.5/main/start.conf set to manual.

This means the generator skips over it and the postgresql.wants.d never gets added with that instance name.

According to the script logic simply removing it has the right affect of making it auto.

You can also set it to auto and do a daemon-reload to regenerate the necessary config.

Improve this answer
3
  • Thanks, this has already clarified a lot! However, my /etc/postgresql/9.5./main/start.conf already says auto. If I manually execute /lib/systemd/system-generators/postgresql-generator 9.5-main from within /run/systemd/generator/postgresql.service.wants then the necessary link gets created and everything works after I systemctl restart postgresl (although systemctl status postgresql still says active (exited)). However, after a reboot the wants link is not created automatically, and systemctl daemon-reload doesn't re-create it either. Any ideas?
    – Florian Brucker
    Commented Dec 19, 2016 at 9:38
  • You could possibly try copying the generator to /etc/systemd/system-generators/ and seeing if it works from that path on reboot. Something might be wrong with the shellscript at boot time (maybe a filesystem isn't mounted?). You'd have to alter the shellscript to get it to be more verbose and store the data somewhere to see what gets executed from within it.
    – Matthew Ife
    Commented Dec 19, 2016 at 10:09
  • systemctl daemon-reload does run the generator, I must have missed that in my previous comment. However I can't get the generator to run at boot-time, so as a hacky workaround I'm now running systemctl daemon-reload and systemctl restart postgresql after booting via a Vagrant provision script. Thanks a lot for your help!
    – Florian Brucker
    Commented Dec 19, 2016 at 13:42
2

As of 26 May 2017, the issue still exists on Debian 8 (as far as I can see): PostgreSQL fails to start on boot.

The solution for the time being is to issue:

systemctl daemon-reload
systemctl restart postgresql

manually on boot and everything should be all right. Thanks to Florian Brucker for pointing this out above.

The bug is apparently in systemd and, hopefully, should be rectified soon.

Improve this answer
3
  • Had this issue with ubuntu 20 and postgres 12.6 now. Can this still be the same bug or can this workaround address also other issues? No error log, nothing, I had no idea why the service was not starting.
    – jan
    Commented Jul 8, 2021 at 10:59
  • @jan check systemctl status [email protected]
    – Алексей Неудачин
    Commented Oct 9, 2021 at 13:30
  • This systemctl combo helped me on Ubuntu 20.04 with Postgres 14 that worked fine for a couple of restarts over a few days and then suddenly went on strike today. I had an error in postgresql.conf that wasn't reported, perhaps because I also changed the logging config, but no log appeared on custom location either (which worked when started properly). Even after reverting the config, the restart didn't help, but after both commands from above, it did. Thank you. But I'm still completely puzzled why no log appeared anywhere.
    – virgo47
    Commented Oct 18, 2021 at 10:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .