Strategic IT Consulting and Implementation Services

Using fail2ban To Mitigate Excessive Apache 403, 404, 500, and 503 Attacks

By Jonathan Mergy
In Cybersecurity, InfoSec, Linux
August 8, 2020
4 comments

I finally spent some time last weekend to address the botnets attacking my site and specifically looking for known exploits, bad WordPress plugins, and just general random stuff. I should disclaimer that messing around and systematically blocking hosts viewing your website generating 404s may or may not make sense for you. If you create a fail2ban filter, jail, and start picking off hosts that generate 404 errors or even 403, 500, and 503 error codes, that may not be the right thing to do for you because you could be blocking legitimate webcrawlers and human visitors in the process. That is not an issue in my case. So, here is how you can do it. I’m running on Ubuntu/Debian and I guess still old-school with Apache2.

Filter Configuration

First, create a fail2ban filter or two for what you want to target. I did two – one for forbidden codes and one of unavailable Apache error codes. These are typically located in

/etc/fail2ban/filter.d/

My new simple filters files are:

/etc/fail2ban/filter.d/apache-forbidden.conf
/etc/fail2ban/filter.d/apache-unavailable.conf

apache-forbidden.conf is just:

[Definition]
failregex = <HOST> - - .*HTTP/[0-9]+(.[0-9]+)?" 403 *

apache-unavailable.conf is just:

[Definition]
failregex = <HOST> - - .*HTTP/[0-9]+(.[0-9]+)?" 503 *
<HOST> - - .*HTTP/[0-9]+(.[0-9]+)?" 500 *
<HOST> - - .*HTTP/[0-9]+(.[0-9]+)?" 404 451 *

Jail Config

After creating the filters for fail2ban, adding jail configs for each of them are trivial. I use jail.local to hold all the jails info. Here is my snippet for the two new filters to pair them to new jails in fail2ban.

[apache-forbidden]
enabled = true
port = http,https
filter = apache-forbidden
logpath = /var/log/apache2/*access.log
maxretry = 2

[apache-unavailable]
enabled = true
filter = apache-unavailable
logpath = /var/log/apache2/*access.log
maxretry = 3
port = http,https

The maxretry is pretty harsh. You are going to have to see what works for you and your sites but in testing now for the past few weeks the hits I received and addresses placed in jails were totally justifiable from hostile attackers. Your situation and results will vary.

Reloading after you make those adds along with checking your /var/log/fail2ban.log to see hits and verify you are seeing baddies get blocked is mandatory since these types of draconian filters can go bad and start blocking a lot of legit visitors or webcrawlers if this isn’t the right type of mechanism for you.

It’s always necessary to check your jail status and whois some samples just to confirm you get the typical Russian, Chinese, OVH, Digital Ocean regulars and not real users just trying to see site content.

# fail2ban-client status apache-unavailable

Status for the jail: apache-unavailable |- Filter | |- Currently failed: 298 | |- Total failed: 878 | - File list: /var/log/apache2/access.log /var/log/apache2/other_vhosts_access.log /var/log/apache2/ssl_access.log - Actions |- Currently banned: 161 |- Total banned: 161 `- Banned IP list: 195.201.251.88 51.38.115.166 164.132.44.97 106.3.38.73 5.135.138.188 5.79.105.33 51.89.201.9 52.186.165.217 46.36.39.239 136.243.2.135 141.98.189.50 159.69.109.50 49.7.20.28 93.158.161.14 77.75.76.161 62.210.91.20 79.112.123.126 207.241.231.37 46.229.161.131 49.7.21.12 95.142.196.32 51.38.92.2 116.203.105.92 95.81.207.245 40.115.24.141 77.75.79.17 126.103.172.79 185.222.57.183 77.75.78.161 111.231.205.120 116.203.72.161 77.75.78.166 31.170.123.253 58.61.249.15 159.203.81.46 159.203.182.52 177.75.21.199 185.25.35.15 117.81.195.128 198.8.85.230 82.79.236.59 159.69.109.52 207.241.231.188 82.117.194.229 95.163.255.68 37.187.132.5 95.163.255.69 95.163.255.62 77.75.77.17 95.163.255.63 95.163.255.66 77.75.79.11 95.163.255.64 95.163.255.67 185.25.35.12 13.68.153.44 95.163.255.65 5.62.20.30 77.75.78.163 35.231.111.37 77.75.77.11 31.14.73.64 77.75.79.119 34.74.55.123 189.60.92.193 159.69.183.149 185.25.35.9 95.163.255.120 35.196.38.103 77.75.79.36 23.102.182.204 185.25.35.11 77.75.76.166 202.171.75.133 35.190.218.27 5.62.60.54 77.75.77.101 114.79.7.241 77.75.76.160 185.25.35.8 5.62.62.54 77.75.76.171 77.75.79.62 77.75.77.32 35.228.46.165 77.247.127.98 185.25.35.13 95.163.255.111 34.204.198.218 185.25.35.14 77.75.76.165 93.158.161.3 94.130.9.166 35.229.118.118 95.163.255.121 95.163.255.131 95.163.255.129 95.163.255.198 95.163.255.130 95.163.255.141 13.92.27.130 34.75.207.58 77.75.76.164 137.74.193.70 35.229.91.121 69.55.62.22 104.131.107.89 104.236.75.170 95.163.255.138 106.38.241.186 77.75.76.163 106.38.241.181 77.75.76.167 51.255.43.81 34.207.121.57 142.234.200.79 176.31.244.49 5.62.20.31 77.75.77.62 69.39.239.21 5.62.20.23 195.211.23.206 95.142.197.2 195.211.23.207 95.142.195.129 195.211.23.208 95.142.196.17 95.142.197.130 95.142.195.134 77.75.79.95 34.84.233.164 185.25.35.10 77.75.79.109 192.200.215.91 82.76.234.254 5.15.118.244 50.63.196.123 178.62.103.239 77.75.78.167 77.75.76.170 195.211.23.209 195.211.23.210 77.75.78.165 77.75.76.162 77.75.77.36 94.249.167.182 77.75.78.168 104.196.70.128 77.75.78.162 93.158.161.53 52.250.120.105 192.243.53.51 95.142.195.143 95.142.196.27 195.211.23.214 95.142.196.133 195.211.23.215 195.211.23.216 162.255.84.188 93.158.161.54 37.187.93.22

# tail -20 /var/log/fail2ban.log
2020-08-08 07:46:28,487 fail2ban.filter [2249]: INFO [killswitch] Found 64.227.61.176 - 2020-08-08 07:46:25 2020-08-08 07:46:29,044 fail2ban.actions [2249]: NOTICE [killswitch] Ban 64.227.61.176 2020-08-08 08:08:36,125 fail2ban.filter [2249]: INFO [killswitch] Found 141.98.81.138 - 2020-08-08 08:08:36 2020-08-08 08:08:36,157 fail2ban.actions [2249]: WARNING [killswitch] 141.98.81.138 already banned 2020-08-08 08:10:21,548 fail2ban.filter [2249]: INFO [killswitch] Found 91.241.19.15 - 2020-08-08 08:10:21 2020-08-08 08:10:21,679 fail2ban.actions [2249]: NOTICE [killswitch] Ban 91.241.19.15 2020-08-08 08:11:11,040 fail2ban.filter [2249]: INFO [killswitch] Found 161.97.94.155 - 2020-08-08 08:11:11 2020-08-08 08:11:11,775 fail2ban.actions [2249]: NOTICE [killswitch] Ban 161.97.94.155 2020-08-08 08:37:56,628 fail2ban.filter [2249]: INFO [apache-forbidden] Found 88.218.17.117 - 2020-08-08 08:37:56 2020-08-08 08:37:56,808 fail2ban.actions [2249]: WARNING [apache-forbidden] 88.218.17.117 already banned 2020-08-08 08:50:04,118 fail2ban.filter [2249]: INFO [apache-unavailable] Found 37.187.93.22 - 2020-08-08 08:50:04 2020-08-08 08:50:06,894 fail2ban.filter [2249]: INFO [apache-unavailable] Found 37.187.93.22 - 2020-08-08 08:50:06 2020-08-08 08:50:14,789 fail2ban.filter [2249]: INFO [apache-unavailable] Found 37.187.93.22 - 2020-08-08 08:50:14 2020-08-08 08:50:14,864 fail2ban.actions [2249]: NOTICE [apache-unavailable] Ban 37.187.93.22 2020-08-08 08:50:18,681 fail2ban.filter [2249]: INFO [killswitch] Found 45.227.255.204 - 2020-08-08 08:50:18 2020-08-08 08:50:18,724 fail2ban.actions [2249]: WARNING [killswitch] 45.227.255.204 already banned 2020-08-08 08:53:15,927 fail2ban.filter [2249]: INFO [killswitch] Found 71.6.167.142 - 2020-08-08 08:53:15 2020-08-08 08:53:16,128 fail2ban.actions [2249]: WARNING [killswitch] 71.6.167.142 already banned 2020-08-08 09:36:10,550 fail2ban.filter [2249]: INFO [killswitch] Found 79.107.73.42 - 2020-08-08 09:36:10 2020-08-08 09:36:10,555 fail2ban.actions [2249]: NOTICE [killswitch] Ban 79.107.73.42

Doing a quick tail against the /var/log/fail2ban.log you can see progress. In the above snippet of the logs you see a few of the new filters/jails taking hits along with the super-handy killswitch filter I built last year that has been amazing!

Hope this helps. I wanted to add these filters for a long time and finally got around to it a few weeks ago and wanted to share.

About the author

Jonathan Mergy
View all posts

4 comments

Leave a Reply to Jonathan Mergy Cancel reply

  • Hi Jonathan,

    First, I truly appreciate that you took the time to put out the 403,404,500, 503 filter instructions for Fail2Ban.
    I’ve been searching for days for just this sort of “How-to” as the filter expressions are so poorly documented.
    That being said, the filter doesn’t appear work in my .11 version of fail2ban. I double checked the filter.d and jails.local entries and they look identical to yours however, I get the following error when restarting.
    Failed to restart server : 2021-07-23 08:54:19,608 fail2ban [201537]: ERROR Failed during configuration: Source contains parsing errors: ‘/etc/fail2ban/filter.d/apache-unavailable.conf’ [line 3]: ‘ – – .*HTTP/[0-9]+(.[0-9]+)?” 500 *\n’ [line 4]: ‘ – – .*HTTP/[0-9]+(.[0-9]+)?” 404 451 *’

    Do you have any ideas on what needs to be changed for the latest fail2ban version?

    Reply


    • failregex = - - .*HTTP/[0-9]+(.[0-9]+)?" 503 *
       - - .*HTTP/[0-9]+(.[0-9]+)?" 500 *
       - - .*HTTP/[0-9]+(.[0-9]+)?" 404 451 *

      Ahh - but I am running v0.10.2

      Fail2Ban v0.10.2

      Copyright (c) 2004-2008 Cyril Jaquier, 2008- Fail2Ban Contributors
      Copyright of modifications held by their respective authors.
      Licensed under the GNU General Public License v2 (GPL).

      Reply

      • I am getting enough hits though for sure.


        # fail2ban-client status apache-unavailable
        Status for the jail: apache-unavailable
        |- Filter
        | |- Currently failed: 84
        | |- Total failed: 1004
        | `- File list: /var/log/apache2/other_vhosts_access.log /var/log/apache2/access.log
        `- Actions
        |- Currently banned: 149
        |- Total banned: 264
        `- Banned IP list: 37.48.64.101 35.238.221.48 95.213.141.126 3.143.215.171 104.197.165.66 35.239.158.35 157.55.39.58 34.152.4.122 3.97.237.63 77.75.76.166 77.75.77.119 77.75.76.169 77.88.5.102 77.88.5.237 77.88.5.165 93.158.161.72 123.58.210.133 77.88.5.27 77.88.5.62 85.208.98.51 77.88.5.168 77.88.5.242 77.88.5.139 77.88.5.246 77.75.76.162 40.87.57.169 77.88.5.83 77.88.5.47 77.88.5.221 77.88.5.108 77.88.5.235 93.158.161.40 77.88.5.116 77.88.5.51 216.18.204.214 93.158.161.36 34.244.222.85 85.208.98.25 93.158.161.49 176.98.29.127 157.90.209.81 23.95.191.195 74.207.228.142 199.21.113.251 34.74.95.130 198.23.172.233 80.82.77.192 35.231.124.233 34.78.249.119 20.150.209.28 91.242.162.2 50.116.22.34 85.208.98.24 51.89.213.55 34.254.199.194 23.251.102.74 172.105.172.151 107.174.37.156 95.24.6.218 3.84.102.169 45.33.103.57 39.96.140.104 104.154.31.26 85.208.98.21 172.105.254.10 167.71.107.82 5.255.231.102 5.255.231.46 13.82.120.55 5.45.207.163 213.180.203.55 5.255.231.55 5.255.231.244 213.180.203.103 5.255.253.159 5.255.231.151 5.45.207.86 5.255.231.129 5.45.207.181 5.255.231.194 5.45.207.110 5.45.207.105 85.208.98.29 85.208.98.20 85.208.98.31 85.208.98.23 141.8.142.82 35.196.247.104 5.255.231.196 207.46.13.128 213.180.203.111 5.255.231.77 91.236.136.143 5.255.253.168 213.180.203.89 176.9.10.174 5.255.231.66 5.45.207.144 193.169.254.115 5.255.231.88 5.255.231.163 5.255.253.81 5.45.207.109 213.180.203.117 35.195.57.216 93.124.81.60 199.247.22.42 213.180.203.114 123.58.212.83 213.180.203.105 213.180.203.29 141.8.142.79 77.247.108.77 94.19.80.19 45.76.176.73 213.180.203.175 45.33.96.205 18.117.249.88 23.251.102.90 139.162.215.70 54.87.31.96 141.8.142.94 52.221.194.26 67.207.84.68 213.180.203.83 213.180.203.101 34.71.20.225 37.252.92.13 45.12.221.18 51.15.164.30 213.180.203.107 139.162.231.247 213.180.203.23 34.79.107.251 54.85.86.52 213.180.203.25 83.41.123.192 157.55.39.69 157.55.39.163 213.180.203.17 194.113.106.127 194.49.69.242 213.180.203.36 213.180.203.54 213.180.203.80 157.55.39.161 23.90.160.122 141.8.142.93 213.180.203.33

        Reply

Read more

Strategic IT Consulting and Implementation Services