Since December 24 my server experiences increased resource consumption. Additionally the Nginx server seems to be unstable which causes several errors.
The CPU usage used to average at around 5%. But a few days ago it increased and is now at a constant 10% to 30%. The same behavior holds true for IPv4 traffic. Plus I frequently get errors such as Error 525: SSL handshake failed
or Error 500: Internal server error
when trying to access my Nextcloud or website. The error messages and the increased traffic arose at the same time. To exclude the possibility of recently done configuration causing the problems, I restored to a Dec 20 backup. So it must have been external influence causing the trouble.
I scanned for viruses using ClamAV on the entire system, but no infected files were found:
----------- SCAN SUMMARY -----------
Known viruses: 8844122
Engine version: 0.103.0
Scanned directories: 28082
Scanned files: 167224
Infected files: 0
Data scanned: 15009.11 MB
Data read: 23880.07 MB (ratio 0.63:1)
Time: 3684.616 sec (61 m 24 s)
Start Date: 2021:01:02 23:54:21
End Date: 2021:01:03 00:55:45
Also I checked for suspicious activity with Netstat:
$ netstat -nt | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -r
2648 104.218.232.38
2589 104.218.232.37
143 5.182.209.124
143 185.189.14.123
132 5.182.209.47
131 54.198.115.81
121 23.8.7.207
113 23.224.103.238
95 185.255.134.153
64 142.93.135.65
37 31.206.5.1
37 134.209.92.79
37 103.29.71.18
35 81.70.202.141
34 194.87.95.95
28 106.52.158.118
26 23.32.85.243
26 116.17.102.163
25 94.103.87.21
25 118.193.41.157
25 111.229.125.162
24 120.53.118.158
23 173.249.18.223
22 81.70.210.159
22 43.227.180.230
22 193.109.79.134
21 139.162.72.45
21 116.17.102.198
21 115.238.196.100
20 159.226.21.39
20 113.100.209.209
19 45.84.196.129
19 173.249.44.200
19 172.104.118.85
19 161.97.135.26
19 113.100.209.120
18 172.105.240.46
18 172.105.191.4
18 172.104.85.88
17 172.105.35.35
17 116.17.102.82
17 116.17.102.251
17 116.17.102.225
17 113.100.209.153
16 61.160.223.228
16 59.38.222.34
16 45.118.135.77
16 207.180.206.180
16 173.212.225.16
16 161.97.135.28
16 139.162.116.216
16 116.17.102.190
16 116.17.102.128
16 113.100.209.91
15 198.58.96.176
15 164.68.101.83
15 116.17.102.71
15 116.17.102.141
15 113.100.209.159
14 176.58.109.91
14 172.104.127.52
14 172.104.117.113
14 139.162.3.85
14 116.17.102.87
14 116.17.102.77
14 113.100.209.9
14 113.100.209.215
13 59.38.223.85
13 42.192.15.120
13 212.102.60.158
13 178.151.141.116
13 116.17.102.117
13 113.100.209.234
13 113.100.209.135
12 195.154.241.248
12 176.119.156.84
12 161.97.135.33
12 161.97.135.32
12 161.97.135.30
12 139.162.104.140
12 113.100.209.207
11 91.193.173.1
11 45.118.133.9
11 207.180.203.143
11 188.195.109.42
11 173.212.226.149
11 172.105.233.224
11 172.104.98.78
11 161.97.76.238
11 161.97.135.224
11 144.202.8.244
11 116.17.102.68
11 116.17.102.237
11 113.100.209.134
11 113.100.209.127
11 113.100.209.119
11 113.100.209.104
10 81.70.103.9
10 59.38.223.115
10 59.38.223.107
10 49.12.66.76
10 191.19.149.198
10 144.91.114.81
10 139.162.77.193
10 127.0.0.1
10 116.17.102.241
10 113.100.209.219
9 59.38.222.171
9 212.7.210.103
9 207.180.237.10
9 178.79.179.193
9 172.105.237.55
9 161.97.135.225
9 139.162.117.120
9 116.17.102.96
9 116.17.102.36
9 116.17.102.142
9 113.100.209.71
9 113.100.209.52
9 113.100.209.235
9 113.100.209.213
9 113.100.209.118
9 113.100.209.115
9 108.28.122.6
9 106.53.136.62
9 103.29.70.181
8 59.38.223.82
8 59.38.222.43
8 27.221.79.31
8 182.254.223.162
8 172.105.196.229
8 164.68.111.16
8 161.97.135.223
8 161.97.135.221
8 161.97.135.220
8 151.106.3.179
8 116.17.102.32
8 116.17.102.254
8 116.17.102.130
8 116.17.102.112
8 113.100.209.147
7 92.241.9.162
7 60.169.78.63
7 212.7.210.104
7 207.180.213.12
7 207.180.211.45
7 164.68.106.182
7 161.97.135.219
7 116.17.102.137
7 113.100.209.80
7 113.100.209.2
7 113.100.209.179
7 113.100.209.125
7 106.55.53.215
6 61.184.1.10
6 59.38.223.238
6 59.38.222.63
6 42.48.184.9
6 221.8.141.164
6 173.249.20.2
6 172.105.58.130
6 172.104.68.177
6 164.68.108.221
6 116.17.102.247
6 116.17.102.223
6 116.17.102.150
6 116.17.102.129
6 113.100.209.69
6 113.100.209.249
6 113.100.209.245
6 113.100.209.169
5 97.107.137.170
5 59.38.222.207
5 47.90.205.159
5 45.87.2.231
5 222.180.195.154
5 180.232.99.133
5 176.99.159.19
5 172.105.37.185
5 172.104.62.99
5 172.104.173.94
5 164.68.107.32
5 154.27.68.105
5 116.17.102.75
5 116.17.102.45
5 116.17.102.172
5 116.17.102.134
5 113.100.209.186
5 113.100.209.181
5 113.100.209.18
4 89.108.84.27
4 82.77.76.92
4 59.38.222.175
4 51.103.40.29
4 204.93.226.69
4 192.46.233.130
4 178.63.149.89
4 173.249.31.254
4 121.29.46.177
4 121.29.46.138
4 118.193.42.237
4 116.17.102.9
4 116.17.102.21
4 113.57.148.194
4 109.27.192.44
4 1.193.20.197
3 81.91.179.207
3 81.71.42.207
3 70.37.160.210
3 59.38.223.98
3 5.255.183.209
3 47.90.255.174
3 47.89.181.151
3 45.82.68.174
3 45.12.212.75
3 36.51.254.229
3 27.147.202.120
3 195.2.67.224
3 185.87.51.122
3 178.124.185.120
3 161.97.76.240
3 139.9.216.230
3 139.204.122.237
3 139.204.117.87
3 136.175.9.57
3 136.175.9.105
3 135.148.12.143
3 116.17.102.20
3 115.231.218.252
3 113.100.209.162
3 113.100.209.140
3 104.131.180.136
3 104.128.58.19
3 103.107.161.129
2 96.126.118.183
2 95.217.249.73
2 94.60.176.83
2 94.50.240.252
2 94.198.98.138
2 94.198.100.8
2 93.77.19.241
2 91.236.120.189
2 81.16.141.51
2 81.16.141.28
2 59.38.222.202
2 51.75.255.151
2 43.248.186.67
2 42.192.16.54
2 39.89.64.117
2 36.51.254.228
2 31.135.149.97
2 3.239.88.227
2 3.236.246.248
2 27.159.82.67
2 27.145.211.135
2 222.93.16.183
2 217.182.173.209
2 203.195.195.235
2 198.27.100.135
2 194.67.218.133
2 188.40.57.143
2 187.107.10.10
2 185.81.158.109
2 183.17.231.237
2 182.253.176.11
2 177.47.87.13
2 173.249.30.9
2 171.252.189.83
2 171.107.124.35
2 163.172.30.116
2 154.8.246.137
2 143.244.42.77
2 143.178.170.214
2 139.204.117.240
2 139.155.172.64
2 122.238.117.25
2 121.29.46.172
2 121.29.46.146
2 118.193.41.84
2 116.17.102.217
2 116.17.102.155
2 115.159.92.188
2 111.49.79.113
2 110.249.208.137
2 104.161.112.234
2 1.189.60.149
2 1.183.243.31
1 servers)
1 Address
1 95.216.244.56
1 95.182.120.9
1 95.168.183.69
1 95.141.46.182
1 95.106.255.97
1 95.10.232.21
1 94.249.192.218
1 94.244.50.10
1 94.103.90.30
1 93.204.184.102
1 92.53.65.210
1 91.206.15.91
1 90.225.65.71
1 88.226.100.225
1 88.218.16.105
1 84.64.221.58
1 82.223.104.78
1 82.162.58.171
1 81.69.44.108
1 8.208.82.133
1 78.47.32.154
1 75.109.4.43
1 74.208.253.135
1 69.167.7.49
1 69.164.210.76
1 66.228.34.13
1 64.64.250.83
1 61.145.49.81
1 59.80.30.164
1 59.38.222.195
1 58.58.237.82
1 51.68.120.72
1 51.210.43.24
1 51.178.240.246
1 51.103.72.158
1 5.9.215.100
1 49.232.87.68
1 47.88.170.127
1 47.75.190.154
1 46.91.22.28
1 46.4.148.26
1 46.17.43.98
1 45.91.20.228
1 45.76.161.122
1 45.236.149.152
1 44.242.167.214
1 42.192.52.67
1 42.192.138.217
1 40.120.54.92
1 39.156.65.236
Then I requested IP information using an IP Geolocation API:
$ curl "http://ip-api.com/line/example_ip_address?fields=country"
And although it should only be accessed by friends, family, teachers and myself, it's pinged from all around the world. It seems to be getting hundreds/thousands of requests from China, Singapore, Bangladesh, Vietnam, Russia, France, United States, Netherlands and more.
I also checked for invalid logins in /var/log/auth.log
. There were multiple attempts to login to my server using usernames that don't exist on the system.
# grep "Invalid user" /var/log/auth.log
Jan 1 10:09:54 server sshd[20560]: Invalid user jake from 117.247.183.216 port 59544
Jan 1 10:11:18 server sshd[20637]: Invalid user pydio from 106.12.97.115 port 36824
Jan 1 10:26:14 server sshd[21278]: Invalid user ts3 from 106.124.136.227 port 43942
Jan 1 11:03:58 server sshd[22909]: Invalid user test1 from 37.114.36.172 port 41906
Jan 1 11:04:00 server sshd[22912]: Invalid user paco from 67.205.142.48 port 40838
Jan 1 11:05:50 server sshd[22998]: Invalid user trade from 114.207.139.203 port 32833
Jan 1 11:07:43 server sshd[23084]: Invalid user teamspeak from 61.155.106.101 port 55632
Jan 1 11:11:05 server sshd[23265]: Invalid user maria from 81.68.83.82 port 49822
Jan 1 11:14:55 server sshd[23434]: Invalid user ts3user from 51.68.226.27 port 57540
Jan 1 11:22:02 server sshd[23737]: Invalid user dave from 43.226.69.100 port 45332
Jan 1 11:53:54 server sshd[25138]: Invalid user pi from 188.76.66.65 port 23060
Jan 1 11:53:54 server sshd[25139]: Invalid user pi from 188.76.66.65 port 22840
Jan 1 13:19:49 server sshd[28963]: Invalid user csgoserver from 61.93.240.18 port 1665
Jan 1 13:23:22 server sshd[29130]: Invalid user hxeadm from 178.128.80.85 port 39950
Jan 1 13:25:05 server sshd[29187]: Invalid user mcserver from 195.29.102.42 port 42286
Jan 1 13:28:52 server sshd[29354]: Invalid user felix from 37.252.190.224 port 59594
Jan 1 13:30:52 server sshd[29440]: Invalid user dinesh from 81.183.213.37 port 60185
Jan 1 13:41:13 server sshd[29920]: Invalid user testuser from 161.82.130.186 port 39300
Jan 1 13:41:48 server sshd[29957]: Invalid user ranger from 106.124.136.227 port 34749
Jan 1 13:46:34 server sshd[30171]: Invalid user vbox from 115.159.161.81 port 36826
Jan 1 13:51:11 server sshd[30352]: Invalid user admin2 from 105.73.83.18 port 36252
Jan 1 13:52:32 server sshd[30428]: Invalid user test from 51.210.5.171 port 54958
Jan 1 13:57:08 server sshd[30609]: Invalid user pmd from 185.234.219.5 port 15368
Jan 1 14:09:00 server sshd[31116]: Invalid user ftpadmin from 111.229.181.50 port 35512
Jan 1 14:13:01 server sshd[31338]: Invalid user maximo from 112.196.43.202 port 42158
Jan 1 14:20:54 server sshd[31680]: Invalid user www from 51.38.70.175 port 60434
Jan 1 15:06:16 server sshd[1391]: Invalid user rd from 49.235.11.137 port 36864
Jan 1 15:19:07 server sshd[1996]: Invalid user roberto from 45.155.205.86 port 44624
Jan 1 15:48:27 server sshd[3277]: Invalid user dennis from 123.58.109.42 port 40322
Jan 1 15:50:35 server sshd[3365]: Invalid user deploy from 106.52.22.230 port 48356
Jan 1 15:52:42 server sshd[3454]: Invalid user admin1 from 122.152.215.115 port 37214
Jan 1 16:05:15 server sshd[3976]: Invalid user user from 195.19.102.173 port 45690
Jan 1 16:12:21 server sshd[4322]: Invalid user git from 118.145.8.50 port 56276
Jan 1 16:51:57 server sshd[6066]: Invalid user ubuntu from 157.231.102.250 port 51841
Jan 1 16:54:17 server sshd[6157]: Invalid user hdfs from 51.77.230.49 port 36038
Jan 1 16:54:29 server sshd[6161]: Invalid user rabbit from 165.22.234.248 port 39244
Jan 1 17:47:33 server sshd[9479]: Invalid user pi from 182.84.124.120 port 50662
Jan 1 17:47:33 server sshd[9480]: Invalid user pi from 182.84.124.120 port 50660
Jan 1 18:09:04 server sshd[10427]: Invalid user test1 from 130.61.134.151 port 58688
Jan 1 18:24:56 server sshd[1387]: Invalid user botuser from 179.131.11.234 port 45754
Jan 1 18:53:49 server sshd[3748]: Invalid user jenkins from 157.230.97.148 port 47838
Jan 1 18:55:20 server sshd[3830]: Invalid user dlwsadmin from 157.230.97.148 port 49102
Jan 1 18:56:50 server sshd[3881]: Invalid user ascend from 157.230.97.148 port 50382
Jan 1 18:58:15 server sshd[3958]: Invalid user dlwsadmin from 157.230.97.148 port 51648
Jan 1 18:59:37 server sshd[4009]: Invalid user ascend from 157.230.97.148 port 52920
Jan 1 19:10:21 server sshd[4539]: Invalid user es from 157.230.97.148 port 34834
Jan 1 19:11:43 server sshd[4590]: Invalid user dolphinscheduler from 157.230.97.148 port 36114
Jan 1 19:57:54 server sshd[1466]: Invalid user bserver from 106.55.41.76 port 33176
Jan 1 19:58:11 server sshd[1500]: Invalid user www from 62.171.157.83 port 64476
Jan 1 19:58:41 server sshd[1507]: Invalid user tom from 86.61.70.243 port 51011
Jan 1 20:00:10 server sshd[1589]: Invalid user admin1 from 150.158.175.66 port 41138
Jan 1 20:09:33 server sshd[2039]: Invalid user guest3 from 49.234.24.246 port 39462
Jan 1 20:09:42 server sshd[2035]: Invalid user upload from 13.82.0.138 port 34294
Jan 1 20:43:07 server sshd[3522]: Invalid user pi from 212.68.244.157 port 45541
Jan 1 20:43:07 server sshd[3521]: Invalid user pi from 212.68.244.157 port 45542
Jan 1 20:54:24 server sshd[3993]: Invalid user support from 185.156.74.65 port 8975
Jan 1 20:54:24 server sshd[3995]: Invalid user support from 185.156.74.65 port 9161
Jan 1 21:04:18 server sshd[4437]: Invalid user ansible from 167.99.210.58 port 51446
Jan 1 21:04:26 server sshd[4441]: Invalid user ansible from 167.99.210.58 port 37472
Jan 1 21:04:59 server sshd[4484]: Invalid user butter from 167.99.210.58 port 37914
Jan 1 21:05:17 server sshd[4496]: Invalid user dev from 167.99.210.58 port 39260
Jan 1 21:05:26 server sshd[4498]: Invalid user user from 167.99.210.58 port 53592
Jan 1 21:12:36 server sshd[4857]: Invalid user sdtdserver from 36.250.229.84 port 50448
Jan 1 21:14:35 server sshd[4943]: Invalid user uftp from 107.175.153.27 port 36842
Jan 1 21:15:39 server sshd[4997]: Invalid user testa from 45.64.184.140 port 51020
Jan 1 21:16:47 server sshd[5042]: Invalid user teamspeak from 113.250.0.149 port 44582
Jan 1 21:21:01 server sshd[5247]: Invalid user jenkins from 167.172.195.99 port 36110
Jan 1 21:39:47 server sshd[6068]: Invalid user devel from 118.24.123.34 port 36368
Jan 1 21:49:22 server sshd[6489]: Invalid user debian from 129.226.225.117 port 33020
Jan 1 21:54:08 server sshd[6670]: Invalid user weblogic from 3.138.200.187 port 40742
Jan 1 21:54:17 server sshd[6705]: Invalid user spravce from 45.155.205.87 port 49303
Jan 1 21:56:04 server sshd[6765]: Invalid user smbuser from 167.172.185.34 port 37432
Jan 1 21:56:36 server sshd[6802]: Invalid user hadoop from 130.61.100.68 port 52070
Jan 1 21:57:38 server sshd[6846]: Invalid user devel from 212.64.71.254 port 55110
Jan 1 21:59:49 server sshd[6935]: Invalid user debian from 174.88.178.92 port 46002
Jan 1 22:07:14 server sshd[7269]: Invalid user ubuntu from 45.148.10.54 port 2536
Jan 1 22:17:13 server sshd[8069]: Invalid user samba from 45.155.205.87 port 15070
It should not be possible for anyone but me to login though, because I've hardened SSH access a long time ago, by limiting action on port 22 using ufw, by installing Fail2ban and by only allowing access using a private authentication key and a password, both of which only I have. I also don't see successful logins by anyone else but me when running the last
command. Furthermore I put my server behind Cloudflare to protect it against DDoS attacks, which didn't help to solve the issues.
I also checked the Nginx error log at /var/log/nginx/error.log
and it lists the alert 768 worker_connections are not enough
over and over, because I've only configured one worker process with 768 worker connections. That would actually be enough for my use case, if the server wasn't attacked/probed by bots. Should I try to increase the number of worker connections anyway?
Thanks in advance!
Update
I just reviewed the Nginx access log at /var/log/nginx/access.log
. This is a small sample of it's contents:
5.45.74.22 - - [04/Jan/2021:00:01:27 +0100] "POST http://5.188.211.72/check.php HTTP/1.1" 200 1161 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
172.104.98.78 - - [04/Jan/2021:00:01:27 +0100] "GET https://wesley.kunlun301.com/?u=http:// HTTP/1.1" 200 292 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)"
103.29.71.18 - - [04/Jan/2021:00:01:27 +0100] "GET https://wesley.kunlun301.com/?u=http:// HTTP/1.1" 500 588 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3599.0 Safari/537.36"
172.104.68.177 - - [04/Jan/2021:00:01:27 +0100] "GET http://console.bestacdn.com:1122/?u=http:// HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)"
45.118.135.77 - - [04/Jan/2021:00:01:27 +0100] "GET http://wesley.kunlun301.com/?u=http:// HTTP/1.1" 200 292 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.18247"
176.58.109.91 - - [04/Jan/2021:00:01:27 +0100] "GET http://console.bestacdn.com:1122/?u=http:// HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
172.105.35.35 - - [04/Jan/2021:00:01:27 +0100] "GET http://wesley.kunlun301.com/?u=http:// HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
62.113.115.240 - - [04/Jan/2021:00:01:27 +0100] "CONNECT steamcommunity.com:443 HTTP/1.1" 400 166 "-" "-"
121.57.146.76 - - [04/Jan/2021:00:01:27 +0100] "CONNECT production-game-api.sekai.colorfulpalette.org:443 HTTP/1.1" 400 166 "-" "-"
139.162.116.216 - - [04/Jan/2021:00:01:27 +0100] "GET http://wesley.kunlun301.com/?u=http:// HTTP/1.1" 499 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
172.104.173.94 - - [04/Jan/2021:00:01:27 +0100] "CONNECT m.facebook.com:443 HTTP/1.1" 400 166 "-" "-"
172.104.127.52 - - [04/Jan/2021:00:01:27 +0100] "GET https://wesley.kunlun301.com/?u=http:// HTTP/1.1" 200 292 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36"
61.136.101.153 - - [04/Jan/2021:00:01:27 +0100] "CONNECT www.alipay.com:443 HTTP/1.0" 400 166 "-" "-"
193.109.79.134 - - [04/Jan/2021:00:01:27 +0100] "GET http://api.steampowered.com/IPlayerService/GetSteamLevel/v1/?key=682AA980899BA2C3A331538849BBC8D4&steamid=76561198013106964 HTTP/1.1" 200 52 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0"
Are these requests to be expected? They seem to cause errors in /var/log/nginx/error.log
.
/var/log/nginx/error.log
and it lists the alert768 worker_connections are not enough
, meaning that bots are actually probing my server, right? Should I try to increase the number of worker connections or the number of worker processes?