Knocked Down

A few years ago I was spending a great deal of time on call – day and night. It rapidly became obvious that we needed to implement company-wide server security that would be effective when on-call personnel were away from the office.

In this article, I describe a solution the team developed that combines TCP Wrappers with port knocking, allowing engineers to work remotely, without compromising the integrity of the company security policy.

Almost every computer application demands a level of security to enforce privacy – from software houses developing the latest ground-breaking game to time-sensitive academic discoveries. Across the many facets of computing, security is one of the most fascinating, especially online security, because security of the Internet must contend with many unseen hurdles thanks to its bare-all public-facing nature.

Hit the Road

One of areas we looked at included using one mobile cell phone carrier universally within the business for engineering staff and tying SSH access specifically only to their IP address blocks.

Sadly, two show-stopping issues quickly became apparent. First, because large carriers (Verizon in this case) allocated hundreds, if not thousands, of IP addresses to its GPRS and 3G customers, this approach opened up our servers to many more IP addresses than in our old system, with its 10 or 20 authorized IP addresses at the company offices and at the homes of those on call. Granted, the rest of the Internet couldn't gain access, but the difference between a handful of authorized IP addresses and thousands is huge.

Second, this approach added to administration because the IP address blocks changed every month or two; as the carrier's IP address pool grew larger, sometimes despite our best efforts, the on-call engineer was inevitably locked out during emergency callouts.

Next, we tried dial-up access via back doors and other out-of-band access (which I will keep to myself for security reasons because it's still in use). Unfortunately, it was all too easy to become overly reliant on the providers offering this access, and when they had connectivity outages, disaster struck at our end.

Being as diligent as possible, we all carried SSH keys around on phones, memory sticks, and laptops, but as hard as we tried, at some point, we didn't have access to those SSH keys in the middle of night, which meant an emergency couldn't be dealt with without harassing other members of staff at home who weren't on call.

Finally – and, as you can tell, we really searched for a solution – we paid for a couple of remote shell accounts with static IP addresses and allowed them access in addition to the office IP addresses.

Yet again, the reliance on these providers caused headaches at one point or another. Additionally, we were conscious that we were advertising our ever-so-precious login information to third parties (which could have been compromised).

Enough Already

Eventually, after too much coffee and a little testing, we were certain that we had found a solution. What the team liked about this solution in particular was that for some systems we could still arbitrarily meld it with our other preferred security methods, such as SSH keys and firewalling, if required.

As with anything relating to computing, there's always one hurdle or another. In this case, we only began implementing our security solution after some unexpected head scratching, because our preferred route used iptables (or more accurately Netfilter, the built-in Linux firewall) by default.

The iptables scripts were getting lengthier, and as powerful as iptables is, we didn't want to let junior members of staff break access for everyone by inadvertently making a config mistake while setting changes live under pressure.

Deviating a little from the norm, therefore, our solution incorporated what I like to think of as a natty fix using my favorite security tool for SSH: TCP Wrappers. By adding TCP Wrappers and a highly effective security methodology called port knocking, you can switch SSH access on and off securely and arbitrarily from any IP address on the Internet.

Knock on Wood

If you haven't heard of port knocking, it allows a sys admin to configure a sequence of connection attempts of varying complexity that acts as the first step in gaining remote access to servers.

The next step would be to provide a valid SSH key or password (e.g., if SSH was used). One such package available in the Debian and Ubuntu software repositories is the well-thought-out and intelligently built knockd.

The sequence I mentioned involves using a client program to knock on what appear to be otherwise closed ports on a server. Because knockd listens at the network link layer, it's suitably sophisticated to allow monitoring of both closed port traffic as well as open ports.

This means you can check without discrimination what goes into your network interface, making no effort to read that information and otherwise add load to your server, but instead just read where the traffic is destined.

So clever is port knocking that if you use iptables to hide your port completely, then you can also disguise the SSH server port entirely, pretending that it doesn't even exist to attackers (so there's nothing to attack).

Until the predetermined sequence of ports or secret knocks has been accepted, allowing the server port to then become visible, these ports are entirely invisible. This approach is also very slick because it means you can catch nefarious port scanners probing for services running on your servers and banish them outright without fear of them guessing your magic port sequence [1].

At this juncture, it's also worth noting that the long-standing armed guard of port monitoring, PortSentry (which has been around since calculators were oversized and on your wrist), also captures – or more accurately monitors – connections to closed ports in the same way. To my mind, the "Customizing PortSentry" article  [1] complements the topic of this article very nicely and might help you build a super-simple but highly effective security solution if you're looking to do so.

Rat-a-tat-tat

In this article, I'll run through the installation of the aforementioned knockd on Debian and Ubuntu. Barring the procedure behind the package install, I would hope that the config would also play nicely with other members of the Linux family, too.

On Debian, use:

apt-get install knockd

Once the small, circa 200KB package is downloaded and installed, the first thing you might notice is the response from the installation script as it comes to an end:

* knockd disabled: not starting. To enable it edit /etc/default/knockd

You'll be glad to know that this is not an error, but a precaution. Because you don't want to be locked out of your remote servers before you're even given a chance to configure the package, knockd thoughtfully installs and doesn't fire itself up until you are ready to proceed carefully. For example, you might want to run a second SSH instance on a different port while you test, make sure that an engineer is handy onsite within the data center, or spawn a failsafe Telnet server just in case you're locked out during testing.

The useful README file, which can be found in /usr/share/doc/knockd/README, runs through a brief description of how knockd works with a firewall, looks at the simplicity of the client-side knock sequence, and explains how popular packages such as hping, sendip, and packit are all capable of initiating a connection sequence.

In the README, you will see added information about at least two other implementations of port knocking software called Pasmal [2] and Doorman [3]. The subtitle on the Doorman website – Silent Running – is apt, in that a server can run completely silently, in the sense that all TCP ports are invisible and closed.

Harking back to ye olde days of secret societies, which required a secret knock before the door would be opened, your server suddenly appears like an ephemeral apparition, intriguingly, only to you.

Who's That Knocking?

As you might expect, I next inspect the file mentioned at the end of the install, /etc/default/knockd. Figure 1 demonstrates the simplicity of the file. As you can see, /etc/default/knockd only enables and disables the daemon and changes which interfaces the daemon will listen on; the highest level of config for knockd, you might say.

Following the sensible Linux standard, the global configuration file is in the root of the /etc directory; as the output in Figure 1 shows, you're now pointed toward a file called /etc/knockd.conf (Figure 2). From this file, you'll set about putting this sophisticated package to good use.

What immediately strikes me when looking at this file is that, as packages go, it's definitely very light on the config and entirely unintimidating. That suits me just fine: In my book, simplicity is by far the most intelligent approach to dealing with complexity.

Not Today, Thanks

The next task is to incorporate the trusty TCP Wrappers into the [openSSH] and [closeSSH] sections. This can be achieved in a number of ways once TCP Wrappers is verified as being installed.

Back in the day, it was a case of downloading a tarball and building the file from source with a complex, multiply wrapping command line. Thankfully, however, tcpd, the daemon responsible for wrapping inbound traffic (with useful logging and access control granularity) installs from package managers with ease these days. If for some reason you're not seeing the man page, you can install it with:

apt-get install tcpd

Incidentally, if you're thinking of using TCP Wrappers with other programs, you can easily check whether they have been compiled with support by running:

ldd /usr/local/filename | grep libwrap

In theory, if your software can indeed speak to TCP Wrappers without further ado, you should see some output.

Knock It Off

At this point, I'll mention that we deployed port knocking across three bastion hosts. In other words, we used servers, sitting mainly on the network perimeter (but not always), which were hardened to the extent that they could withstand most types of attacks and still hold the fort. These servers each had access to a particular segment of the network that could reach the rest of the network at large, but in between segments were several addition layers of security, in case one were to suffer a compromise.

Therefore even if one server, and therefore a network segment, was breached, in theory we would know about it before other parts of the network could be affected. We used three servers in this case for redundancy. If any of the servers were to fail for any of a number of reasons (e.g., hardware failure, connectivity, or attacks), we still hoped to have remote access.

Knocking On

TCP Wrappers mainly pays attention to the /etc/hosts.allow file these days because its counterpart, /etc/hosts.deny, essentially just says, "no one is allowed in"; that is, it uses a sensible and pragmatic default deny policy. You can either let IP addresses or DNS names gain access (please see the article that discusses TCP Wrappers [1] in greater detail).

In this case, I explicitly allow access by adding IP addresses in the /etc/hosts.allow file, prepended by the service I want to allow them to use. Conversely, I simply remove them from that file to ban access.

If the key tcpd files don't exist, then you can create them with the permissions shown in Listing 1. By way of comparison, the file /etc/hosts.deny might look like Listing 2 by default, depending on the version (just copy these lines in the ALL:ALL section if the file doesn't exist), and its close relation, /etc/hosts.allow, looks something like Listing 3 by default (simply copy in the SSHD: line if your file doesn't exist).

-rw-r--r-- 1 root root 351 Nov  3  2012 /etc/hosts.deny
-rw-r--r-- 1 root root 348 Jun 17 08:23 /etc/hosts.allow

# hosts.deny    This file describes the names of the
#       hosts which are *not* allowed to use the local
#       INET services, as decided by the '/usr/sbin/
#       tcpd' server.
#
# The portmap line is redundant, but it is left to
# remind you that the new secure portmap uses hosts.deny
# and hosts.allow.  In particular you should know that
# NFS uses portmap!
ALL:ALL

# hosts.allow   This file describes the names of the hosts
#       which are allowed to use the local INET services,
#       as decided by the '/usr/sbin/tcpd' server.
#
SSHD: 12.34.56.78, 78.56.43.12

I should mention that I've been caught out in the past when adding TCP Wrappers to a server already running other services (back when inetd controlled many of the available services on a server) without realizing that ALL:ALL within /etc/hosts.deny had locked a service down. If my memory serves me, it was access to a POP3 mail server. If that trips you up, then change ALL:ALL to something like SSHD:ALL so it only affects SSH – at least temporarily while you're finding your way.

Someone's at the Door

Now you can consider how to trigger SSH access by opening its port, once port knocking has approved the secret door-knocking sequence. Ideally, it's as simple as adding a line to /etc/hosts.allow with the approved IP address to allow an IP address through, and then reversing the process by deleting that IP address when the magic closing sequence has been received, in much the same way as the default iptables setup shown in Figure 2.

Another consideration is to avoid triggering additional scripts that need their permissions set correctly on each server on which the script is deployed. Perl, the undisputed heavyweight champion of one-liners, will be useful here, so make sure it is installed on your system.

The default config file (Listing 4) begins with UseSyslog, which implies that the file /var/log/syslog (or /var/log/messages on some Linux flavors) receives logging information. On Debian I'm becoming more accustomed to looking inside /var/log/auth.log for all things relating to logins, but I don't like to mess with that file because it's too important.

01 [options]
02
03    UseSyslog
04
05 [openSSH]
06         sequence    = 7000,8000,9000
07         seq_timeout = 10
08         tcpflags    = syn
09         command     = /usr/sbin/iptables -A INPUT -s \
           %IP% -p tcp --dport 22 -j ACCEPT
10
11 [closeSSH]
12         sequence    = 9000,8000,7000
13         seq_timeout = 10
14         tcpflags    = syn
15         command     = /usr/sbin/iptables -D INPUT -s \
           %IP% -p tcp --dport 22 -j ACCEPT

In testing, I would put a hash symbol in front of the UseSyslog entry to disable it and use a knockd-specific logfile, such as:

#  UseSyslog
logfile = /var/log/knockd.log

The /etc/knockd.conf file in Figure  3 shows this change. Looking at this file closely, you need to pay most attention to the %IP% variable, a built-in variable used by knockd that is populated with the connecting IP address, which you can manipulate within your open and close access commands.

Clearly it would be madness to leave the default port-knocking sequence in place to secure a production server, almost rendering the use of port knocking useless and practically leaving your front door keys on the doormat.

However, you can configure that later; now is the time to make use of the %IP% variable in a magical Perl one-liner:

/usr/bin/perl -ni -e 'print unless /SSHD: %IP%/' /etc/hosts.allow

The -i says run the file in place (i.e., Perl opens the file, makes a quick temporary file as a backup, and uses that to replace the original file). The -e simply means to execute this command. Perl loops through your file with the -n switch and asks Perl to print what's in the file unless it matches the string of characters that follow. Because there's no substitution of characters, the line is deleted cleanly rather than adding whitespace. In other words, you don't want to use something like this,

/usr/bin/perl -pi -e 's/SSHD: %IP%/ /' /etc/hosts.allow

which replaces the matched pattern with a single space. Although this will work, it would not work as well as the print alternative, because it adds extra lines to the /etc/hosts.allow file.

In the past, I have seen /etc/hosts.allow break so that no one can log in if unusual characters are added the file. Once I had to dial in to the back of a server to get remote access again using mgetty. There's a very good reason I don't intend to add and remove commas and extra CIDR (network notations) in the /etc/hosts.allow file. All I want is a straightforward IP address appended to the bottom of that file once and then cleanly (and I mean really cleanly) removed just once from the end of the file afterward.

With the ever so slightly tweaked line

/usr/bin/perl -ni.bak -e 'print unless /SSHD: %IP%/' /etc/hosts.allow

Perl can create a potentially life-saving backup file. However, enough Perl for now. If you've altered the /etc/knockd.conf file to use TCP Wrappers, as shown in Figure 3, then you can begin to test your config.

Knocking at Heaven's Door

At this stage, you can test your installation. Yet again, I ask you to please make sure that you're not going to lock yourself out of your remote server. With regard to creating a knocking sequence from the client side, in addition to a tiny binary called knock, I've also seen a handy, lightweight Android app that conveniently sends the knocking sequence from your smartphone.

Other than adjusting the secret knock sequences, one final piece of config needs to change, which means going back to the higher level file /etc/default/knockd to set the daemon live. To begin, change the interface setting to the correct network card on which you want knockd to listen:

KNOCKD_OPTS="-i eth0"

To set the service live, enable the daemon:

START_KNOCKD=1

This next part is optional; but, before restarting your daemon, I suggest adding the -D to the init.d startup script and watching your logfile closely. If you get stuck, it's probably thanks to a superfluous character in your /etc/knockd.conf. If push comes to shove, you can packet sniff your connection to make sure your knocking is getting through.

To add debugging information to your logfile, use -D on the OPTIONS line in /etc/init.d/knockd, paying careful attention to the spacing inside the double quotation marks on that line (be sure to make a copy before editing):

OPTIONS=" -d -D"

After a daemon config refresh, then,

service knockd restart

you're ready to go.

Come On In

To see if you can get access, run through the following tests:

1. Check that you have don't have access to the SSH server first (i.e., the IP address from which you'll be connecting to the server isn't already in the /etc/hosts.allow file in one incarnation or another, such as a wide IP range or DNS name).

2. Send your knocking sequence to append your IP address to the end of /etc/hosts.allow and check your /var/log/knockd.log file for an OPEN SESAME command to denote success. You can use this to watch that file in real time (or at least updates every second) in another terminal with:

watch -n1 cat /etc/hosts.allow

3. Assuming you have access, try closing access with the other knocking sequence, such as in this example of the tiny knock client program running on ports 1022, 3022, and 3204:

knock -v 12.34.56.78 1022 3022 3024

To knock solely on one of the 65,535 UDP ports, rather than TCP, you can universally switch to UDP with the -u parameter:

knock -vu 203.1.2.3 2022 3022 3022

However, as one of the examples in the knockd documentation shows, it's also perfectly possible to mix up both TCP and UDP ports at the client side,

knock myserver.example.com 123:tcp 456:udp 789:tcp

thus making the sequence safer, akin to using upper- and lowercase letters in your passwords.

If all is going well, you have now seen a line appended to your /etc/hosts.allow file and then seen that exact same line deleted after locking up access again with your second knocking sequence.

More, More, More

Critics of port knocking claim that it's nothing more than security through obscurity, a term used to describe the act of disguising any valuable resources that might be there for the taking, as opposed to actually securing them with locks and keys and suchlike.

Indeed, they have a point, but alongside your other security measures, port knocking is a helpful extra layer of protection for an attacker to penetrate. You have to admit that using iptables and so forth to hide your precious ports from the world at large is a really nice, if not somewhat clandestine, touch.

Although TCP Wrappers won't be for everyone, I wanted to demonstrate the customizable knockd. For those who want to use firewalling instead of tcpd, here are two examples of start and stop commands for the uncomplicated firewall (UFW; Listing 5) and for iptables (Listing 6). Your mileage may vary.

start_command  = /sbin/iptables -I INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
stop_command  = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT

start_command = ufw allow from %IP% to any port 22
stop_command = ufw delete allow from %IP% to any port 22

The End Is Nigh

The author of knockd has clearly thought further about the most relevant criticism about port knocking, which is the man-in-the-middle attack. If hackers spot any such traffic and pick up which IP address is running port knocking, then they potentially have more information than they should, which can help them further guess about ways to gain access.

The additional feature to knockd, which I have to admit really intrigues me, uses a sophisticated series of one-off, unique sequences. In other words, although your destination IP address might be revealed, even if attackers manage to uncover your port knocking sequence (i.e., secret knock) and repeat that sequence, they will fail because each sequence simply expires after use.

To enable such an excellent feature, all you do is change the sequence config lines with the path to your sequences file (preferably auto-generated by a script):

one_time_sequences = /etc/knockd/one_time_sequences

This file could be brimming with easily remembered one-time sequences, such as obfuscated phone numbers, dates of birth, or other numbers.

Using this method in combination with closing the firewall within 10 seconds (having successfully spawned an SSH session yourself),

cmd_timeout = 10

makes for a truly powerful addition to your security arsenal.

Conclusion

In this article, I've covered a few very relevant but ultimately different events. I started with the problematic scenario of ever-changing remote IP addresses and the lack of control over those IP addresses when working away from the office. I hope I've suitably extolled the virtues of knockd, to the point that you might try to install it yourself.

Finally, with some considered customization, my task would be complete if you felt comfortable enough to conceive of a creative security solution using this package along with PortSentry and deploy it along with other powerful tools to improve your server security and make the Internet a safer place.