Ban-Aid

The few sys admins who haven't heard of fail2ban are likely to be just starting out or focus on areas other than server administration. In my experience, it's quite rare that really small utilities can affect the way you run your servers to the extent that fail2ban does, which certainly explains its popularity.

The fail2ban feather-weight set of scripts easily integrate with popular firewalls and, among other things, catches any failed logins for services that you're running and then bans the IP address of the offender after a certain number of failed attempts. Admittedly, that sounds like quite simple functionality, but when you get down to the innards of the software, it's a truly powerful tool.

I had been using fail2ban on SSH login failures, probably it's most common usage, before I became increasingly annoyed with web server logs filling up with nefarious probes attempting to compromise PHP with remote exploits (and a myriad of other HTTP attacks). It got to the point at where a large proportion of the Apache logs were failed attempts to find hidden directories or non-existent Joomla installations among the legitimate hits on the websites.

Also, I ran a few mail servers that allowed mail relaying via SASL password authentication [1], which (and there are other ways of running the authentication side) had system user accounts with PAM [2] checking for correct passwords. I had set the SASL user accounts so that a shell login couldn't be used to access the server, but I was still more than aware that having a piece of software so readily open to abuse by brute force was far from ideal. So, fail2ban stepped forward yet again; I could simply ban any IP that entered the wrong password three times for as long as I wanted.

From the scenarios above, I hope you will agree that fail2ban can be applied in all sorts of ways. In this article, I'll offer some examples To give you a head start, ranging from those straight out of the documentation to those that were hard won. (Those of you who speak regular expressions, or regex, as your second language would no doubt have found them easy, but I prefer a cogent language that doesn't involve an aching head coupled with eye strain!)

It Must Be Magic

Rather than the smoke and mirrors that some pieces of software employ to keep their workings secret, fail2ban is transparent in the work it does behind the scenes.

In simple terms, fail2ban [3] keeps a close eye on your logfiles, and when a specific pattern is matched within those logs, it triggers an action that you've predefined within its "action" config files. The distinction between its filters and actions in the directory structure is clear once you've installed it.

Script Kiddies and Port 22

As I mentioned, the most popular usage of fail2ban is stopping probes that try every username ever conceived by mankind on your poor SSH server on port 22. Not only is this SSH vulnerability a logging irritation, it is highly insecure, allowing people limitless tries at guessing your username and password combinations.

Now is not the time to detail how to secure your SSH server, but for goodness sake, move it to a port other than port 22, and unless you've got a really, really good reason, permit access by IP address with TCP Wrappers [4]. Either way, you should consider banning failed logins for a short time in case a poor configuration mistake gives miscreants access inadvertently.

By my reckoning, for user-accessible services on a server, any user who gets a password wrong more than three times is going to need a new password (from a Support Department most likely), so I'd prefer not to allow nefarious types to keep trying repeatedly until they give up. Bear in mind that some scripts attack a port several times a second, so in a minute, they could have just about exhausted all popular logins beginning with the first letter of the alphabet!

Figure 1 shows an SSH logfile example of failed logins. All you have to do (and please bear in mind my comments about eye-strain and regex earlier) is create a regular expression to catch those failed login attempts so you can trigger a response when they're spotted in the logs.

Thankfully, a few useful examples are already available post-installation, with a couple of handy tags to save you from figuring out other ways of spotting patterns – one such tag being <HOST>. If you're sitting comfortably, the pattern below matches an SSH login attempt failure which, on a Debian box, lives in the /etc/fail2ban/filter.d/sshd.conf file. Don't worry: this first example isn't too trying. The two salient lines in the config file are as follows:

failregex = ^%(__prefix_line)sFailed (?:password|publickey) for .* from <HOST>(?: port \d*)?(?: ssh\d*)?$
ignoreregex =

By comparing that regex with the logfile in Figure 1, I hope you get the gist of how it pieces together. As I said, don't be too perturbed by the regex complexity; thankfully, several solid examples are included at installation time.

Fortunately, fail2ban is flexible – not just in the services it can monitor but also across different operating systems, all of which have many versions themselves. So, understanding your potential regex pain, the developers, even in the standard SSH config file, have included a number of failregex examples that should match your system's logfile format. The examples shown in Listing 1 are originally commented out in the config file, ready for you to choose an entry to copy and paste in place of the uncommented line.

01 failregex = ^%(__prefix_line)s(?:error: PAM: )?Authentication failure for .* from <HOST>\s*$
02             ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from<HOST>\s*$
03             ^%(__prefix_line)sFailed (?:password|publickey) for .* from <HOST>(?: port \d*)?(?: ssh\d*)?$
04             ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
05             ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$
06             ^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers$
07             ^%(__prefix_line)sauthentication failure; logname=\S* uid=\S* euid=\S* tty=\S* ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$
08             ^%(__prefix_line)srefused connect from \S+ \(<HOST>\)\s*$
09             ^%(__prefix_line)sAddress <HOST> .* POSSIBLE BREAK-IN ATTEMPT!*\s*$
10             ^%(__prefix_line)sUser .+ from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*$

I still need to explain the ignoreregex line in the first example. I hope it's clear, though, what the setting in that line, again written in hieroglyphics (sorry regex), would produce: an IP address or hostname match that should NOT trigger an action if it's spotted within that failregex pattern in the logfiles. I'll give you some ignoreregex examples a little later.

SASL

I briefly touched on SASL and mail server password authentication earlier, as well as my concerns about opening up a username and password combination from /etc/shadow to the Internet as a serious hole for abuse. To allay my fears, a simple fail2ban instance is now configured to catch bad logins in a highly efficient manner.

Here is the failregex line for SASL login attempts from the config file /etc/fail2ban/filter.d/sasl.conf:

failregex = (?i): warning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed: authentication failure

A quick hunt through my /var/log/mail.log files reveals:

Dec 20 22:59:16 Ganymede postfix/smtpd[26681]: warning: a-not-so-innocent.ip-has-been-disguised.com[20.666.92.40]: SASL LOGIN authentication failed: authentication failure

If you squint long enough at the failregex and the mail server log format (Postfix, in this case), you can see that they should probably tie in with each other. After some testing, you should be confident that they do.

Lay of the Land

Now I'll look a bit closer at how the config files are laid out in fail2ban's directory structure. The preceding configuration files live in the filter.d/ directory. A quick ls shows a lot of out-of-the-box fail2ban filter examples (Table 1). A quick warning: if they don't fit your operating system perfectly, you might be using the search engines for a while, unless you have a diploma in regex.

For many scenarios, though, the main site [3] has how-tos, as well as the bundled examples. As you can see, a multitude of examples are ready to go, including a few Apache config files.

The main configuration file is called jail.conf, and it's commented with lots of helpful instructions. The ignoreip setting can take CIDR IP address formats or just single IP addresses, so the Admin doesn't get locked out accidentally; also, that's really handy for testing. Additionally, a default setting that applies to all jails (a brief set of rules about a particular service) that don't have the ignoreip setting explicitly configured is:

bantime  = 1
maxretry = 3

Here, it's referring to how many opportunities a visitor has to trigger the filter; in this case, it's set really tight and is probably impractical for most services for which users occasionally make mistakes.

The next setting is which back-end daemon to use for polling the logfiles. In Debian's case, I set it to gamin, which is great at checking files like mailboxes and logfiles really frequently without putting system load up.

Finally, the actions.d directory contains what actions to take and with which firewall. Aside from some information on the level of action required, for all intents and purposes, the remainder of the jail.conf file is filled with a short config for each service (a jail), switched on or off with true or false. Here's the entry for SASL:

[sasl]
enabled  = true
port        = smtp
filter       = sasl
logpath   = /var/log/mail.log

Aside from inheriting the bantime and maxretry default setting from above, all you have to think about now is the action setting in jail.conf, which is relatively straightforward. Also, you can calm fail2ban's reaction to just writing a logfile entry or triggering an email alert; however, I prefer all guns blazing, with all the offending log entries and a who is lookup of the banned IP address mailed to me (Listing 2).

01 # ban the IP & send an e-mail with whois report and include relevant log lines to the destemail
02 action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s]
03              %(mta)s-whois-lines[name=%(__name__)s, dest="%(destemail)s", logpath=%(logpath)s]

Lights, Camera, Action

Now that you know what to send and when to send it, you simply need to know how to ban the offending IP address. I'll use the most popular firewall, iptables, the kernel's Netfilter firewall.

The file /etc/fail2ban/action.d/iptables-multiport.conf has these key config lines present:

Actionban: actionban = iptables -I fail2ban-<name> 1 -s <ip> -j DROP
Actionunban: actionunban = iptables -D fail2ban-<name> -s <ip> -j DROP

To anyone familiar with iptables, the commonplace switches -I for INSERT and -D for DELETE for the ban and unban settings should be pretty clear. These settings are editable, as you'd expect if you want to log further or manipulate iptables in some other way.

Apache

Several configurations are already in place for Apache, in-hand with the mail POP3/IMAP and FTP services, but I needed a mishmash of the examples, and admittedly it took me ages to get the regex correct.

I had a very specific requirement because I absolutely knew that no pages were missing on the websites on the web server (which might generate HTTP status response 404s), but I also needed to ignore explicitly some files and directories.

Any other traffic to the web server that I did not specifically allow could therefore be banned if it attempted to access files that didn't exist or probe other parts of the server looking for PHP exploits.

Inside my filter.d/ directory, I created a filter called apache-misc. I've shortened the example a little so it's readable (Listing 3), but the first line of the failregex after the [A-Z] part effectively drops any request to Apache that doesn't include a forward slash. (Pretty much everything legitimate should include a forward slash.)

01 [Definition]
02 failregex = <HOST>.*"[A-Z]* /(cms|user|muieblackcat|db|cpcommerce|wp-login|joomla|awstatstotals|wp-content|wp-includes|pma|phpmyadmin|myadmin|mysql|mysqladmin|sqladmin|mypma|admin|xampp|mysqldb|pmadb|phpmyadmin1|phpmyadmin2).*"
03             <HOST>.*\" (502|500|417|416|415|414|413|412|404|405|403|401|400)
04 ignoreregex = .*\"GET \/(press|mailto|domestic|word).*

After the slash and inside the brackets are entries for the requests I want banned. The next line down (which is still part of the same failregex) beginning with <HOST> looks for matches, including the ever-common 404 errors. Finally the ignoreregex works beautifully, giving some leeway when it's needed.

Whereas I might have had a hundred attempts per IP probing the server, the result of applying this filter to Apache is that now I only have three entries in the logs (which all happened in the first second before gamin read the logfile and added the iptables rule).

I'm delighted with the lack of unwanted logfile noise, and my web servers are safer! If the config is too strict for you, try and remove the 404 entry from the second failregex line and increase the bantime and maxretry numbers to see how you get on.

I've barely scratched the surface of this topic, but I hope this article will inspire you to look further into how fail2ban can help you secure your servers. After you've installed it, have a look at how many IP addresses are "already banned" in the fail2ban logfile, /var/log/fail2ban.log, and see how many attempts you've successfully blocked in a day.

With its level of reporting and inherently efficient and controllable configuration, fail2ban is a knockout piece of software that should help any sys admin search for intruders and protect the network from future attacks.