fail2ban has maxretry of 3 but I see authentication failures repeated 5 times

Question

I am running Ubuntu 16.04 with ssh enabled through ufw and have configured fail2ban to enable the [sshd] and [sshd-ddos] jails with a maxretry of 3 (i.e. I want to ban any ips that fail to authenticate 3 times). When I look at the auth log I see a few different ips that have failed to authenticate 5 times and so have been disconnected for too many authentication failures. Why are these ips not being banned? I'm fully expecting that I have either misconfigured fail2ban or that I misunderstand its intended behaviour, but from the tutorials that I have read (digitalocean, other search results) this configuration should do what I expect. Please also note that fail2ban is successfully banning some ips, as I can see bans (and unbans) in the fail2ban log.

Here are the relevant log lines for one of the ip addresses with the issue (ip and host redacted):

"/var/log/fail2ban.log"

2017-02-17 13:23:36,148 fail2ban.filter         [24793]: INFO    [sshd] Found [ip address]
2017-02-17 13:23:38,153 fail2ban.filter         [24793]: INFO    [sshd] Found [ip address]

"/var/log/auth.log"

Feb 17 13:23:36 [host] sshd[15498]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=[ip address]  us
er=root
Feb 17 13:23:38 [host] sshd[15498]: Failed password for root from [ip address] port 9498 ssh2
Feb 17 13:23:49 [host] sshd[15498]: message repeated 5 times: [ Failed password for root from [ip address] port 9498 ssh2]
Feb 17 13:23:49 [host] sshd[15498]: error: maximum authentication attempts exceeded for root from [ip address] port 9498 ssh2 [preauth]
Feb 17 13:23:49 [host] sshd[15498]: Disconnecting: Too many authentication failures [preauth]
Feb 17 13:23:49 [host] sshd[15498]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=[ip address]  user=root
Feb 17 13:23:49 [host] sshd[15498]: PAM service(sshd) ignoring max retries; 6 > 3

Here are the relevant (non comment) lines from my fail2ban configuration, which is mostly a straight copy from jail.conf with the bantime, findtime and maxretry defaults changed, and the [sshd], [sshd-ddos] and [recidive] jails enabled:

"/etc/fail2ban/jail.local"

[DEFAULT]

ignoreip = 127.0.0.1/8
bantime = 21600
findtime = 3600
maxretry = 3
backend = auto
usedns = warn
logencoding = auto
enabled = false
filter = %(__name__)s
destemail = root@localhost
sender = root@localhost
mta = sendmail
protocol = tcp
chain = INPUT
port = 0:65535
banaction = iptables-multiport
action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
action_mw = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
            %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
action_mwl = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]
action_xarf = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath=%(logpath)s, port="%(port)s"]
action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
                %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]
action_blocklist_de  = blocklist_de[email="%(sender)s", service=%(filter)s, apikey="%(blocklist_de_apikey)s"]
action_badips = badips.py[category="%(name)s", banaction="%(banaction)s"]

[sshd]

enabled = true
port    = ssh
logpath = %(sshd_log)s

[sshd-ddos]
# This jail corresponds to the standard configuration in Fail2ban.
# The mail-whois action send a notification e-mail with a whois request
# in the body.
enabled = true
port    = ssh
logpath = %(sshd_log)s

[recidive]

enabled  = true
logpath  = /var/log/fail2ban.log
banaction = iptables-allports
bantime  = 604800  ; 1 week
findtime = 86400   ; 1 day
maxretry = 5

Further investigation:

sudo fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d

log file is correct as /var/log/auth.log Lines: 1127 lines, 0 ignored, 125 matched, 1002 missed [processed in 1.77 sec]

sudo iptables -L -n | less

I can see some banned ips (with REJECT) under "Chain f2b-sshd (1 references)""

Answer 1

Fail2ban sees only the logs in standard format and counts how many logs appeared during some time frame. Also your syslog is merging the same logs into aggregated form, which prevents this parsing:

Feb 17 13:23:49 [host] sshd[15498]: message repeated 5 times: [ Failed password for root from [ip address] port 9498 ssh2]

Generally authentication failure is considered as a connection that failed to authenticate. But this single connection can consist of several "password authentication attempts" (how many depends on the configuration of the client NumberOfPasswordPrompts and server MaxAuthTries).

When you set up fail2ban, it is good to know how things work around and when setting a policy in there, know what does it mean. In this case, this is expected behaviour and if you want to limit the authentication attempts further, you need to adjust the configuration of sshd.