I just updated my Nextcloud instance to 19.0.5 and then to 20.0.2 (built-in updater).
After the first update the web interface and client were both experiencing connection issues (504 Gateway-Timeout) and slow responsiveness when the connection did finally go through. I managed to get back to the Settings page and update from 19.05 to 20.02, but the connection problems persist.
When in maintenance mode the page loads normally.
Entering my ip address loads the nginx page very quickly.
Entering my nextcloud domain into https://www.ssllabs.com/ssltest/ shows that I have an A+ security rating. I was sure to clear the cache so that it had to make a new connection.
I have taken some steps to diagnose the problem, but I am coming up empty:
restarted nginx with
sudo systemctl reload nginx
sudo nginx -t
shows that
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
I checked: /var/log/php7.4.log and found many lines of this
05-Dec-2020 12:06:50] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
I went to: /etc/php/fpm/pool.d/www.conf and increased the max_children to 40
and PHP seems to be running fine:
● php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-12-06 09:16:53 EST; 4h 15min ago
Docs: man:php-fpm7.4(8)
Process: 1086 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=>
Main PID: 868 (php-fpm7.4)
Status: "Processes active: 40, idle: 0, Requests: 22306, slow: 0, Traffic: 0req/sec"
Tasks: 41 (limit: 28592)
Memory: 790.9M
CGroup: /system.slice/php7.4-fpm.service
├─ 868 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
├─ 3195 php-fpm: pool www
├─ 3198 php-fpm: pool www
├─ 3202 php-fpm: pool www
├─ 3226 php-fpm: pool www
├─ 3239 php-fpm: pool www
├─ 3251 php-fpm: pool www
├─ 3258 php-fpm: pool www
├─ 3261 php-fpm: pool www
├─ 3268 php-fpm: pool www
├─ 3271 php-fpm: pool www
├─ 3281 php-fpm: pool www
├─ 3337 php-fpm: pool www
├─ 3340 php-fpm: pool www
├─ 3363 php-fpm: pool www
├─ 3369 php-fpm: pool www
├─ 3376 php-fpm: pool www
├─ 3389 php-fpm: pool www
├─ 3393 php-fpm: pool www
├─ 3399 php-fpm: pool www
├─ 3406 php-fpm: pool www
├─ 3409 php-fpm: pool www
├─ 3412 php-fpm: pool www
├─ 3415 php-fpm: pool www
├─ 3480 php-fpm: pool www
├─ 3631 php-fpm: pool www
├─14761 php-fpm: pool www
├─14763 php-fpm: pool www
├─14765 php-fpm: pool www
├─14767 php-fpm: pool www
├─14769 php-fpm: pool www
├─14822 php-fpm: pool www
├─14824 php-fpm: pool www
├─14826 php-fpm: pool www
├─14828 php-fpm: pool www
├─14830 php-fpm: pool www
├─14832 php-fpm: pool www
├─14957 php-fpm: pool www
├─14962 php-fpm: pool www
├─14964 php-fpm: pool www
└─47101 php-fpm: pool www
I set up the PHP to show info with
sudo nano /usr/share/nginx/html/info.php
<?php phpinfo(); ?>
But instead of showing the info, navigating to localhost/info.php in the browser asks me to open a file.
I am monitoring the server with NetHogs and see that when I contact the server from a remote web browser a large number of php-fpm: pool www processes are engaged, but only sending/receiving ~1KB/sec.
Here is the contents of /etc/nginx/conf.d/nextcloud.conf: (replaced my actual server with 'mynextcloud')
server {
server_name mynextcloud;
# Add headers to serve security related headers
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;
#I found this header is needed on Ubuntu, but not on Arch Linux.
add_header X-Frame-Options "SAMEORIGIN";
# Path to the root of your installation
root /usr/share/nginx/nextcloud/;
access_log /var/log/nginx/nextcloud.access;
error_log /var/log/nginx/nextcloud.error;
location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
# The following 2 rules are only needed for the user_webfinger app.
# Uncomment it if you're planning to use this app.
#rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
#rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json
# last;
location = /.well-known/carddav {
return 301 $scheme://$host/remote.php/dav;
}
location = /.well-known/caldav {
return 301 $scheme://$host/remote.php/dav;
}
location ~ /.well-known/acme-challenge {
allow all;
}
# set max upload size
client_max_body_size 16G;
fastcgi_buffers 64 4K;
# Disable gzip to avoid the removal of the ETag header
gzip off;
# Uncomment if your server is build with the ngx_pagespeed module
# This module is currently not supported.
#pagespeed off;
#error_page 403 /core/templates/403.php;
#error_page 404 /core/templates/404.php;
location / {
rewrite ^ /index.php$uri;
}
location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
deny all;
}
location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
deny all;
}
location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/>
include fastcgi_params;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
#Avoid sending the security headers twice
fastcgi_param modHeadersAvailable true;
fastcgi_param front_controller_active true;
fastcgi_pass unix:/run/php/php7.4-fpm.sock;
fastcgi_intercept_errors on;
fastcgi_request_buffering off;
}
location ~ ^/(?:updater|ocs-provider)(?:$|/) {
try_files $uri/ =404;
index index.php;
}
# Adding the cache control header for js and css files
# Make sure it is BELOW the PHP block
location ~* \.(?:css|js)$ {
try_files $uri /index.php$uri$is_args$args;
add_header Cache-Control "public, max-age=7200";
# Add headers to serve security related headers (It is intended to
# have those duplicated to the ones above)
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header X-Robots-Tag none;
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Referrer-Policy no-referrer;
# Optional: Don't log access to assets
access_log off;
}
location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {
try_files $uri /index.php$uri$is_args$args;
# Optional: Don't log access to other assets
access_log off;
}
listen 443 ssl http2; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/www.dysoncloud.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.dysoncloud.net/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
add_header Strict-Transport-Security "max-age=31536000" always;
ssl_trusted_certificate /etc/letsencrypt/live/mynextcloud/chain.pem; # managed by Certbot
ssl_stapling on; # managed by Certbot
ssl_stapling_verify on; # managed by Certbot
}
server {
if ($host = mynextcloud) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name mynextcloud;
listen 80;
return 404; # managed by Certbot
}
contents of /usr/share/nginx/nextcloud/config/config.php
<?php
$CONFIG = array (
'instanceid' => 'xxx',
'passwordsalt' => 'xxx',
'secret' => 'xxx',
'trusted_domains' =>
array (
0 => 'www.dysoncloud.net',
),
'datadirectory' => '/usr/share/nginx/nextcloud-data',
'dbtype' => 'mysql',
'version' => '19.0.5.2',
'overwrite.cli.url' => 'https://www.dysoncloud.net',
'dbname' => 'nextcloud',
'dbhost' => 'localhost:3306',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'nextclouduser',
'dbpassword' => 'xxx',
'installed' => true,
'memcache.local' => '\\OC\\Memcache\\APCu',
'mail_from_address' => 'dysonsphere',
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_domain' => 'startmail.com',
'mail_smtpauthtype' => 'LOGIN',
'mail_smtpauth' => 1,
'mail_smtphost' => 'smtp.startmail.com',
'mail_smtpport' => '465',
'mail_smtpname' => '[email protected]',
'mail_smtppassword' => 'xxxxxxxxxxxxxxxxx',
'mail_smtpsecure' => 'ssl',
'maintenance' => false,
'updater.release.channel' => 'stable',
'updater.secret' => 'xxx',
'theme' => '',
'loglevel' => 2,
);
contents of /var/log/nginx/error.log:
020/12/06 09:16:52 [warn] 1056#1056: "ssl_stapling" ignored, host no
t found in OCSP responder "ocsp.int-x3.letsencrypt.org" in the certif
icate "/etc/letsencrypt/live/collabra.ddns.net/fullchain.pem"
2020/12/06 09:16:52 [warn] 1056#1056: "ssl_stapling" ignored, host no
t found in OCSP responder "ocsp.int-x3.letsencrypt.org" in the certif
icate "/etc/letsencrypt/live/www.dysoncloud.net/fullchain.pem"
2020/12/06 10:02:36 [error] 1064#1064: *7389 open() "/usr/share/nginx
/html/robots.txt" failed (2: No such file or directory), client: 66.2
49.66.215, server: collabra.ddns.net, request: "GET /robots.txt HTTP/
1.1", host: "dysoncloud.net"
2020/12/06 10:43:21 [error] 1064#1064: *13017 open() "/usr/share/ngin
x/html/TP/public/index.php" failed (2: No such file or directory), cl
ient: 134.175.245.162, server: collabra.ddns.net, request: "GET /TP/p
ublic/index.php HTTP/1.1", host: "24.202.235.154"
2020/12/06 10:43:23 [error] 1064#1064: *13018 open() "/usr/share/ngin
x/html/TP/index.php" failed (2: No such file or directory), client: 1
34.175.245.162, server: collabra.ddns.net, request: "GET /TP/index.ph
p HTTP/1.1", host: "24.202.235.154"
2020/12/06 10:43:25 [error] 1064#1064: *13019 open() "/usr/share/ngin
x/html/thinkphp/html/public/index.php" failed (2: No such file or dir
ectory), client: 134.175.245.162, server: collabra.ddns.net, request:
"GET /thinkphp/html/public/index.php HTTP/1.1", host: "24.202.235.15
4"
2020/12/06 10:43:26 [error] 1064#1064: *13020 open() "/usr/share/ngin
x/html/html/public/index.php" failed (2: No such file or directory),
client: 134.175.245.162, server: collabra.ddns.net, request: "GET /ht
ml/public/index.php HTTP/1.1", host: "24.202.235.154"
2020/12/06 10:43:29 [error] 1064#1064: *13021 open() "/usr/share/ngin
x/html/public/index.php" failed (2: No such file or directory), clien
t: 134.175.245.162, server: collabra.ddns.net, request: "GET /public/
index.php HTTP/1.1", host: "24.202.235.154"
2020/12/06 10:43:29 [error] 1064#1064: *13022 open() "/usr/share/ngin
x/html/TP/html/public/index.php" failed (2: No such file or directory
), client: 134.175.245.162, server: collabra.ddns.net, request: "GET
/TP/html/public/index.php HTTP/1.1", host: "24.202.235.154"
2020/12/06 10:43:31 [error] 1064#1064: *13023 open() "/usr/share/ngin
x/html/elrekt.php" failed (2: No such file or directory), client: 134
.175.245.162, server: collabra.ddns.net, request: "GET /elrekt.php HT
TP/1.1", host: "24.202.235.154"
2020/12/06 10:43:32 [error] 1064#1064: *13024 open() "/usr/share/ngin
x/html/index.php" failed (2: No such file or directory), client: 134.
175.245.162, server: collabra.ddns.net, request: "GET /index.php HTTP
/1.1", host: "24.202.235.154"
2020/12/06 11:15:46 [error] 1064#1064: *17649 open() "/usr/share/ngin
x/html/wp-login.php" failed (2: No such file or directory), client: 1
61.35.36.90, server: collabra.ddns.net, request: "GET /wp-login.php H
TTP/1.1", host: "aaa.aaa.dysoncloud.net"
020/12/06 12:01:14 [error] 1064#1064: *22713 open() "/usr/share/ng
inx/html/jars" failed (2: No such file or directory), client: 91.24
1.19.84, server: collabra.ddns.net, request: "GET /jars HTTP/1.1",
host: "24.202.235.154:80"
2020/12/06 12:03:27 [error] 1064#1064: *22915 open() "/usr/share/ng
inx/html/.env" failed (2: No such file or directory), client: 104.4
1.134.160, server: collabra.ddns.net, request: "GET /.env HTTP/1.1"
, host: "24.202.235.154"
2020/12/06 12:04:07 [error] 1064#1064: *22983 open() "/usr/share/ng
inx/html/.env" failed (2: No such file or directory), client: 104.4
1.134.160, server: collabra.ddns.net, request: "GET /.env HTTP/1.1"
, host: "24.202.235.154"
2020/12/06 12:11:01 [error] 1064#1064: *23750 open() "/usr/share/ng
inx/html/.env" failed (2: No such file or directory), client: 104.4
1.134.160, server: collabra.ddns.net, request: "GET /.env HTTP/1.1"
, host: "24.202.235.154"
2020/12/06 12:11:21 [error] 1064#1064: *23754 open() "/usr/share/ng
inx/html/.env" failed (2: No such file or directory), client: 104.4
1.134.160, server: collabra.ddns.net, request: "GET /.env HTTP/1.1"
, host: "24.202.235.154"
2020/12/06 12:18:50 [crit] 1064#1064: *24549 SSL_do_handshake() fai
led (SSL: error:14201044:SSL routines:tls_choose_sigalg:internal er
ror) while SSL handshaking, client: 64.41.200.107, server: 0.0.0.0:
443
2020/12/06 12:19:07 [error] 1064#1064: *24559 open() "/usr/share/ng
inx/html/favicon.ico" failed (2: No such file or directory), client
: 192.168.1.1, server: collabra.ddns.net, request: "GET /favicon.ic
o HTTP/1.1", host: "24.202.235.154", referrer: "http://24.202.235.1
54/"
020/12/06 12:48:50 [error] 1064#1064: *28039 "/usr/share/nginx/htm
l/phpmyadmin/index.html" is not found (2: No such file or directory
), client: 113.24.145.137, server: collabra.ddns.net, request: "GET
/phpmyadmin/ HTTP/1.1", host: "24.202.235.154"
2020/12/06 12:57:14 [error] 1064#1064: *28972 open() "/usr/share/ng
inx/html/nextcloud" failed (2: No such file or directory), client:
192.168.1.1, server: collabra.ddns.net, request: "GET /nextcloud HT
TP/1.1", host: "24.202.235.154"
2020/12/06 12:57:26 [error] 1064#1064: *28972 open() "/usr/share/ng
inx/html/dysoncloud" failed (2: No such file or directory), client:
192.168.1.1, server: collabra.ddns.net, request: "GET /dysoncloud
HTTP/1.1", host: "24.202.235.154"
2020/12/06 13:04:50 [crit] 1064#1064: *29760 SSL_do_handshake() fai
led (SSL: error:14201044:SSL routines:tls_choose_sigalg:internal er
ror) while SSL handshaking, client: 64.41.200.107, server: 0.0.0.0:
443
2020/12/06 13:35:05 [error] 1064#1064: *33403 open() "/usr/share/ng
inx/html/config/getuser" failed (2: No such file or directory), cli
ent: 2.57.122.186, server: collabra.ddns.net, request: "GET /config
/getuser?index=0 HTTP/1.1", host: "24.202.235.154:80"
2020/12/06 13:45:54 [error] 1064#1064: *34617 open() "/usr/share/ng
inx/html/Licenses/jQuery FileSearch (MIT).php" failed (2: No such f
ile or directory), client: 35.189.229.112, server: collabra.ddns.ne
t, request: "POST /Licenses/jQuery%20FileSearch%20(MIT).php HTTP/1.
1", host: "24.202.235.154", referrer: "https://www.google.com/"
Recent contents of /var/log/nginx/nextcloud.error:
2020/12/06 11:17:35 [error] 1064#1064: *17387 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/files/?fileNotFound=1 HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.
4-fpm.sock", host: "www.dysoncloud.net"
2020/12/06 12:18:34 [error] 1064#1064: *24364 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/files/?fileNotFound=1 HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.
4-fpm.sock", host: "www.dysoncloud.net"
2020/12/06 12:22:33 [error] 1064#1064: *24364 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/recommendations/api/recommendations HTTP/2.0", upstream: "fastcgi://unix:
/run/php/php7.4-fpm.sock", host: "www.dysoncloud.net"
2020/12/06 12:23:54 [error] 1064#1064: *25125 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /settings/apps HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock",
host: "www.dysoncloud.net"
2020/12/06 12:27:50 [error] 1064#1064: *25732 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /settings/apps HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock",
host: "www.dysoncloud.net"
2020/12/06 12:31:14 [error] 1064#1064: *25732 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /settings/apps HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock",
host: "www.dysoncloud.net"
2020/12/06 12:33:01 [error] 1064#1064: *26227 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/dashboard/ HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock"
, host: "www.dysoncloud.net"
2020/12/06 12:36:49 [error] 1064#1064: *26725 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/dashboard/ HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock"
, host: "www.dysoncloud.net"
2020/12/06 12:39:10 [error] 1064#1064: *26725 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/dashboard/ HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock"
, host: "www.dysoncloud.net"
2020/12/06 12:42:13 [error] 1064#1064: *26725 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/dashboard/ HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock"
, host: "www.dysoncloud.net"
2020/12/06 12:44:17 [error] 1064#1064: *27415 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/files/?fileNotFound=1 HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.
4-fpm.sock", host: "www.dysoncloud.net"
2020/12/06 12:49:00 [error] 1064#1064: *27954 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/files/?fileNotFound=1 HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.
4-fpm.sock", host: "www.dysoncloud.net"
2020/12/06 12:55:24 [error] 1064#1064: *28647 upstream timed out (110: Connection timed out) while reading response header from upstream, cl
ient: 192.168.1.1, server: www.dysoncloud.net, request: "GET /apps/files/?fileNotFound=1 HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.
4-fpm.sock", host: "www.dysoncloud.net"
I am at a loss for what to do at this point. My nextcloud instance is pretty much useless right now, and I was getting so much use out of it until yesterday. I hope there is a way to fix this without having to rebuild the whole thing.
Thanks in advance to anyone with any thoughts on this.
