Trying to understand if fail2ban is working on Debian 10 VPS

Question

I have a Debian 10 server running on a VPS. The only software I installed is: tinyproxy (http proxy) and fail2ban

I have included the results of port scan using ss

I have included my specific settings in the fail2ban jail.local file.

I have included below a sample of entires from fail2ban log and auth log.

I do not understand if fail2ban is working, i.e. causing IPs to be blocked based upon entries in the IP Tables that fail2ban has made.

As example:

• fail2ban.log shows 3 entries for IP 103.226.138.245
• The 3rd entry says that the IP has already been banned.

auth.log shows numerous entries for 103.226.138.245 and I don't understand why.

I thought that based upon the IP being blocked, that the malicious user would NOT be able to attempt a login. Yet it seems that these users are indeed being able to attempt logins.

My questions:

1. Does it appear that fail2ban is working?
2. Why are malicious users allowed to even attempt login if they are banned?

---

Here is fail2ban log, starting at 10:54:06. As example, there are 3 entries for 103.226.138.245:

2024-01-23 10:54:06,466 fail2ban.filter         [29045]: INFO    [sshd] Found 139.59.92.218 - 2024-01-23 10:54:06
2024-01-23 10:54:06,467 fail2ban.filter         [29045]: INFO    [sshd] Found 139.59.92.218 - 2024-01-23 10:54:06
2024-01-23 10:54:06,504 fail2ban.actions        [29045]: WARNING [sshd] 139.59.92.218 already banned
2024-01-23 10:54:07,171 fail2ban.filter         [29045]: INFO    [sshd] Found 103.226.138.245 - 2024-01-23 10:54:07
2024-01-23 10:54:07,172 fail2ban.filter         [29045]: INFO    [sshd] Found 103.226.138.245 - 2024-01-23 10:54:07
2024-01-23 10:54:07,907 fail2ban.actions        [29045]: WARNING [sshd] 103.226.138.245 already banned
2024-01-23 10:54:08,079 fail2ban.filter         [29045]: INFO    [sshd] Found 139.59.92.218 - 2024-01-23 10:54:08
2024-01-23 10:54:08,154 fail2ban.filter         [29045]: INFO    [sshd] Found 103.226.138.245 - 2024-01-23 10:54:08
2024-01-23 10:54:13,469 fail2ban.filter         [29045]: INFO    [sshd] Found 130.61.35.0 - 2024-01-23 10:54:13
2024-01-23 10:54:13,471 fail2ban.filter         [29045]: INFO    [sshd] Found 130.61.35.0 - 2024-01-23 10:54:13
2024-01-23 10:54:13,917 fail2ban.actions        [29045]: WARNING [sshd] 130.61.35.0 already banned
2024-01-23 10:54:15,077 fail2ban.filter         [29045]: INFO    [sshd] Found 130.61.35.0 - 2024-01-23 10:54:14
2024-01-23 10:54:15,079 fail2ban.filter         [29045]: INFO    [sshd] Found 159.89.94.43 - 2024-01-23 10:54:15
2024-01-23 10:54:16,685 fail2ban.filter         [29045]: INFO    [sshd] Found 206.189.229.70 - 2024-01-23 10:54:16
2024-01-23 10:54:16,686 fail2ban.filter         [29045]: INFO    [sshd] Found 206.189.229.70 - 2024-01-23 10:54:16
2024-01-23 10:54:16,687 fail2ban.filter         [29045]: INFO    [sshd] Found 159.89.94.43 - 2024-01-23 10:54:16
2024-01-23 10:54:17,123 fail2ban.actions        [29045]: WARNING [sshd] 206.189.229.70 already banned
2024-01-23 10:54:17,123 fail2ban.actions        [29045]: WARNING [sshd] 159.89.94.43 already banned
2024-01-23 10:54:18,764 fail2ban.filter         [29045]: INFO    [sshd] Found 206.189.229.70 - 2024-01-23 10:54:18
2024-01-23 10:54:18,765 fail2ban.filter         [29045]: INFO    [sshd] Found 103.86.180.10 - 2024-01-23 10:54:18
2024-01-23 10:54:18,766 fail2ban.filter         [29045]: INFO    [sshd] Found 103.86.180.10 - 2024-01-23 10:54:18
2024-01-23 10:54:19,127 fail2ban.actions        [29045]: WARNING [sshd] 103.86.180.10 already banned
2024-01-23 10:54:20,658 fail2ban.filter         [29045]: INFO    [sshd] Found 103.86.180.10 - 2024-01-23 10:54:20
2024-01-23 10:54:24,981 fail2ban.filter         [29045]: INFO    [sshd] Found 34.84.82.194 - 2024-01-23 10:54:24
2024-01-23 10:54:24,983 fail2ban.filter         [29045]: INFO    [sshd] Found 34.84.82.194 - 2024-01-23 10:54:24
2024-01-23 10:54:25,136 fail2ban.actions        [29045]: WARNING [sshd] 34.84.82.194 already banned

Here is Auth log starting at 10:54:06. As example, there are several entries for 103.226.138.245.

Jan 23 10:54:06 racknerd-64d010 sshd[11576]: Invalid user wangyongxin from 139.59.92.218 port 33490
Jan 23 10:54:06 racknerd-64d010 sshd[11576]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:06 racknerd-64d010 sshd[11576]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.59.92.218 
Jan 23 10:54:07 racknerd-64d010 sshd[11583]: Invalid user sunaz from 103.226.138.245 port 51052
Jan 23 10:54:07 racknerd-64d010 sshd[11583]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:07 racknerd-64d010 sshd[11583]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.226.138.245 
Jan 23 10:54:08 racknerd-64d010 sshd[11576]: Failed password for invalid user wangyongxin from 139.59.92.218 port 33490 ssh2
Jan 23 10:54:08 racknerd-64d010 sshd[11583]: Failed password for invalid user sunaz from 103.226.138.245 port 51052 ssh2
Jan 23 10:54:08 racknerd-64d010 sshd[11576]: Received disconnect from 139.59.92.218 port 33490:11: Bye Bye [preauth]
Jan 23 10:54:08 racknerd-64d010 sshd[11576]: Disconnected from invalid user wangyongxin 139.59.92.218 port 33490 [preauth]
Jan 23 10:54:08 racknerd-64d010 sshd[11583]: Received disconnect from 103.226.138.245 port 51052:11: Bye Bye [preauth]
Jan 23 10:54:08 racknerd-64d010 sshd[11583]: Disconnected from invalid user sunaz 103.226.138.245 port 51052 [preauth]
Jan 23 10:54:13 racknerd-64d010 sshd[11586]: Invalid user tosi from 130.61.35.0 port 57576
Jan 23 10:54:13 racknerd-64d010 sshd[11586]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:13 racknerd-64d010 sshd[11586]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.61.35.0 
Jan 23 10:54:14 racknerd-64d010 sshd[11586]: Failed password for invalid user tosi from 130.61.35.0 port 57576 ssh2
Jan 23 10:54:14 racknerd-64d010 sshd[11586]: Received disconnect from 130.61.35.0 port 57576:11: Bye Bye [preauth]
Jan 23 10:54:14 racknerd-64d010 sshd[11586]: Disconnected from invalid user tosi 130.61.35.0 port 57576 [preauth]
Jan 23 10:54:15 racknerd-64d010 sshd[11588]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=159.89.94.43  user=root
Jan 23 10:54:16 racknerd-64d010 sshd[11590]: Invalid user es_user from 206.189.229.70 port 37586
Jan 23 10:54:16 racknerd-64d010 sshd[11590]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:16 racknerd-64d010 sshd[11590]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=206.189.229.70 
Jan 23 10:54:16 racknerd-64d010 sshd[11588]: Failed password for root from 159.89.94.43 port 60092 ssh2
Jan 23 10:54:17 racknerd-64d010 sshd[11588]: Received disconnect from 159.89.94.43 port 60092:11: Bye Bye [preauth]
Jan 23 10:54:17 racknerd-64d010 sshd[11588]: Disconnected from authenticating user root 159.89.94.43 port 60092 [preauth]
Jan 23 10:54:18 racknerd-64d010 sshd[11590]: Failed password for invalid user es_user from 206.189.229.70 port 37586 ssh2
Jan 23 10:54:18 racknerd-64d010 sshd[11592]: Invalid user mrmomeni from 103.86.180.10 port 37374
Jan 23 10:54:18 racknerd-64d010 sshd[11592]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:18 racknerd-64d010 sshd[11592]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.86.180.10 
Jan 23 10:54:20 racknerd-64d010 sshd[11590]: Received disconnect from 206.189.229.70 port 37586:11: Bye Bye [preauth]
Jan 23 10:54:20 racknerd-64d010 sshd[11590]: Disconnected from invalid user es_user 206.189.229.70 port 37586 [preauth]
Jan 23 10:54:20 racknerd-64d010 sshd[11592]: Failed password for invalid user mrmomeni from 103.86.180.10 port 37374 ssh2
Jan 23 10:54:22 racknerd-64d010 sshd[11592]: Received disconnect from 103.86.180.10 port 37374:11: Bye Bye [preauth]
Jan 23 10:54:22 racknerd-64d010 sshd[11592]: Disconnected from invalid user mrmomeni 103.86.180.10 port 37374 [preauth]
Jan 23 10:54:24 racknerd-64d010 sshd[11594]: Invalid user fan1 from 34.84.82.194 port 53972
Jan 23 10:54:24 racknerd-64d010 sshd[11594]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:24 racknerd-64d010 sshd[11594]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=34.84.82.194 
Jan 23 10:54:27 racknerd-64d010 sshd[11594]: Failed password for invalid user fan1 from 34.84.82.194 port 53972 ssh2
Jan 23 10:54:27 racknerd-64d010 sshd[11594]: Received disconnect from 34.84.82.194 port 53972:11: Bye Bye [preauth]
Jan 23 10:54:27 racknerd-64d010 sshd[11594]: Disconnected from invalid user fan1 34.84.82.194 port 53972 [preauth]
Jan 23 10:54:36 racknerd-64d010 sshd[11597]: Invalid user ckr from 43.135.163.185 port 48842
Jan 23 10:54:36 racknerd-64d010 sshd[11597]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:36 racknerd-64d010 sshd[11597]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.135.163.185 
Jan 23 10:54:38 racknerd-64d010 sshd[11597]: Failed password for invalid user ckr from 43.135.163.185 port 48842 ssh2
Jan 23 10:54:39 racknerd-64d010 sshd[11597]: Received disconnect from 43.135.163.185 port 48842:11: Bye Bye [preauth]
Jan 23 10:54:39 racknerd-64d010 sshd[11597]: Disconnected from invalid user ckr 43.135.163.185 port 48842 [preauth]
Jan 23 10:54:44 racknerd-64d010 sshd[11599]: Invalid user scuser from 43.134.92.252 port 49834
Jan 23 10:54:44 racknerd-64d010 sshd[11599]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:44 racknerd-64d010 sshd[11599]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.134.92.252 
Jan 23 10:54:46 racknerd-64d010 sshd[11599]: Failed password for invalid user scuser from 43.134.92.252 port 49834 ssh2
Jan 23 10:54:47 racknerd-64d010 sshd[11601]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=201.184.50.251  user=root
Jan 23 10:54:48 racknerd-64d010 sshd[11599]: Received disconnect from 43.134.92.252 port 49834:11: Bye Bye [preauth]
Jan 23 10:54:48 racknerd-64d010 sshd[11599]: Disconnected from invalid user scuser 43.134.92.252 port 49834 [preauth]
Jan 23 10:54:49 racknerd-64d010 sshd[11601]: Failed password for root from 201.184.50.251 port 39546 ssh2
Jan 23 10:54:51 racknerd-64d010 sshd[11601]: Received disconnect from 201.184.50.251 port 39546:11: Bye Bye [preauth]
Jan 23 10:54:51 racknerd-64d010 sshd[11601]: Disconnected from authenticating user root 201.184.50.251 port 39546 [preauth]

Here is result of ss -lntu scan. I changed my ssh port to 63xxx (obscured):

Netid   State    Recv-Q   Send-Q     Local Address:Port      Peer Address:Port  
tcp     LISTEN   0        5              127.0.0.1:61209          0.0.0.0:*     
tcp     LISTEN   0        128              0.0.0.0:63xxx          0.0.0.0:*     
tcp     LISTEN   0        128              0.0.0.0:8888           0.0.0.0:*     
tcp     LISTEN   0        128                 [::]:63xxx             [::]:*     
tcp     LISTEN   0        128                 [::]:8888              [::]:*     

Here are my entries in jail.local:

[INCLUDES]

#before = paths-distro.conf
before = paths-debian.conf

# The DEFAULT allows a global definition of the options. They can be overridden
# in each jail afterwards.

[DEFAULT]

#
# MISCELLANEOUS OPTIONS
#

# "ignorself" specifies whether the local resp. own IP addresses should be ignored
# (default is true). Fail2ban will not ban a host which matches such addresses.
#ignorself = true

# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban
# will not ban a host which matches an address in this list. Several addresses
# can be defined using space (and/or comma) separator.
ignoreip = 127.0.0.1/8 ::1 xxx.yyy.zzz.xxx

# External command that will take an tagged arguments to ignore, e.g. <ip>,
# and return true if the IP is to be ignored. False otherwise.
#
# ignorecommand = /path/to/command <ip>
ignorecommand =

# "bantime" is the number of seconds that a host is banned.
bantime  = 9000000

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime  = 7200

# "maxretry" is the number of failures before a host get banned.
maxretry = 2

#
# JAILS
#

#
# SSH servers
#

[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
#mode   = normal
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

=== From fail2ban log LATEST INFO:
2024-01-23 15:36:40,421 fail2ban.filter         [12663]: INFO    [sshd] Found 159.75.146.136 - 2024-01-23 15:36:40
2024-01-23 15:36:40,422 fail2ban.filter         [12663]: INFO    [sshd] Found 159.75.146.136 - 2024-01-23 15:36:40
2024-01-23 15:36:40,574 fail2ban.actions        [12663]: WARNING [sshd] 159.75.146.136 already banned

=== From auth log LATEST INGO:
Jan 23 15:36:40 racknerd-64d010 sshd[27856]: Invalid user ali from 159.75.146.136 port 50302
Jan 23 15:36:40 racknerd-64d010 sshd[27856]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 15:36:40 racknerd-64d010 sshd[27856]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=159.75.146.136 
Jan 23 15:36:42 racknerd-64d010 sshd[27856]: Failed password for invalid user ali from 159.75.146.136 port 50302 ssh2
Jan 23 15:36:43 racknerd-64d010 sshd[27856]: Received disconnect from 159.75.146.136 port 50302:11: Bye Bye [preauth]

===From IP Tables LATEST INGO:
    0     0 REJECT     all  --  *      *       159.75.164.110       0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       159.75.146.136       0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       159.75.127.125       0.0.0.0/0            reject-with icmp-port-unreachable

Answer 1

Let's take a look at one example entry pair:

In the the sshd authentication log:

Jan 23 10:54:18 racknerd-64d010 sshd[11592]: Invalid user mrmomeni from 103.86.180.10 port 37374
Jan 23 10:54:18 racknerd-64d010 sshd[11592]: pam_unix(sshd:auth): check pass; user unknown
Jan 23 10:54:18 racknerd-64d010 sshd[11592]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.86.180.10 

And then in fail2ban log:

2024-01-23 10:54:18,765 fail2ban.filter [29045]: INFO [sshd] Found 103.86.180.10 - 2024-01-23 10:54:18

The good news is that fail2ban has spotted the failure, and will have recorded it in its database.

Now check the output from iptables -nvL for fail2ban's sshd chain and confirm there is an entry for the offending IP address. Here's what it might look like:

iptables -nvL f2b-sshd
Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination
   85  5312 REJECT     0    --  *      *       103.86.180.10        0.0.0.0/0            reject-with icmp-port-unreachable
 …

And indeed now you've added that to your question we can see that there are entries being added to the fail2ban chains. This is where the actual work of rejecting inbound traffic occurs, and the rest of fail2ban is all about managing these rules (a Ban results in a new rule; an Unban corresponds to its removal).

The only gotcha is the throwaway comment, "I changed my ssh port to 63xxx". You need to tell fail2ban you've done this!!

[sshd]
enabled = true
port    = ssh,63xxx
logpath = %(sshd_log)s
backend = %(sshd_backend)s

We can confirm that fail2ban is banning traffic on the correct ports with a command like this (if you see nothing where I've got multiport… then it's considering all ports):

iptables -nvL INPUT | awk '!($1+0) || /f2b-/'

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
77182   11M f2b-sshd   6    --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22,63xxx    

Now let's look at your fail2ban.conf, which can be overridden with fail2ban.local or by adding sections to fail2ban.d/:

bantime  = 9000000
findtime  = 7200
maxretry = 2

You've declared that if there are two (or more) entries for the same service and IP address within two hours (7200 seconds) then the IP address will be banned. You've asked for it to be banned for 104 days (9000000 seconds) but this cannot be honoured unless you have also increased dbpurgeage from its default of 28 days to a value of at least 104 days.

Personally, I run two levels, something like this:

• ssh with bantime=86400 (one day), findtime=3600 (one hour), maxretry=6 (six attempts)
• recidive with bantime=2419200 (four weeks), findtime=432000 (five days), maxretry=3 (three attempts)

This bans groups of six failed ssh attempts within an hour for a day, and if there are three of these bans in a five day period they get banned for a further four weeks.

To allow the bans to be managed this long, I've increased dbpurgeage=2462400 (four weeks and 12 hours).

I also have used a slightly different rule action that bans a host entirely, not just against specific ports such as 22 or 63xxx. But that's for a different day.