What's the best method to implement rate limiting with Apache?

Question

I'm running an Apache web server, and I'd like to add some simple rate limiting per individual IP address.

I'm currently getting what appears to be a lot of bot requests hitting the website, and it's slowing down normal requests slightly.

I'd like to be able to add rate limiting to slow down the bots, whilst also not hindering crawls from googlebot when it comes around. I'm guessing somewhere in the region of 20,000 requests/hour per IP should do it, probably less.

Anyway, what's the simplest method for adding rate limiting to Apache?

So far I've found a few options:

• mod_security
• mod_evasive
• mod_ratelimit
• mod_limitpconn

But there doesn't seem to be a clear and obvious solution.

Alternatively, it looks like using Nginx as a reverse proxy (and using the rate limiting built in to that) would be another viable option. It looks easier to implement than what I've found so far for Apache as well. Although I'd prefer to just use Apache if there is a straightforward solution that I'm missing.

Fail2Ban seems like another simple solution, but I'm not sure if I could use it to return 429 response codes when a visitor is hitting the rate limit.

What would you recommend?

Answer 1

mod_evasive

I found mod_evasive to be the most simple and effective solution for simple rate limiting.

It was easier to set up than I expected, and it's the best solution I found out of all the options for Apache (as opposted to going through the effort of setting up Nginx as a reverse proxy).

This is the repo for the mod:

https://github.com/jzdziarski/mod_evasive

Alternatively, there is a slightly newer fork that allows you to return a response code of 429 (instead of 503 in the original version). This is preferable for letting bots like Google know that they're hitting your rate limit and to adjust their crawl rate accordingly:

https://github.com/jvdmr/mod_evasive

I went for the second option.

Install

The mod can be downloaded and installed with:

wget https://github.com/jvdmr/mod_evasive/releases/download/2.2.0/libapache2-mod-evasive.deb
sudo dpkg -i libapache2-mod-evasive.deb

You can check it has been installed using:

sudo a2enmod evasive

Config

Use the following config in your virtual host config file (e.g /etc/apache2/sites-available/000-default.conf) to get started:

DOSEnabled true
DOSHashTableSize 1000
DOSPageCount 10
DOSPageInterval 1
DOSSiteCount 50
DOSSiteInterval 5
DOSBlockingPeriod 20
DOSHTTPStatus 429

This returns a 429 response code if someone requests the same page 10 times in 1 second, or makes 50 requests across the website in 5 seconds. Adjust these figures as necessary.

They will then be blocked for 20 seconds. You can set a fairly small blocking period, as this amount of time will continue extend if they make further rate-limiting requests whilst they are still blocked.

This link explains the config settings: https://www.linode.com/docs/guides/modevasive-on-apache/

Running

After you've saved your config, you just need to restart Apache:

sudo service apache2 restart

You can then check the most recently banned IPs via syslog (I don't believe the banned IPs from mod_evasive are written to their own file):

cat /var/log/syslog | grep mod_evasive

So far it's working well, and from what I've found it's the best mod out of all the available options for simple rate limiting per IP in Apache.