Against the Wall

As soon as a machine becomes accessible over the Internet, attacks start to rain down on it. Tools such as Fail2ban help against brute force attacks but are not a panacea. A DIY script offers flexible and fault-tolerant protection.

Maintaining a Linux server's packet filtering rules is one of the routine tasks of any administrator. Often, carefully maintained scripts, white lists, and blacklists are used to protect the server from unauthorized access. Public blacklists (e.g., OpenBL.org [1]) distribute the IP addresses of honeypot systems that help document attacks and distribute the IP addresses of the attackers. The Fail2ban script [2] supports local defenses with early blocking of hosts that have made several unsuccessful access attempts, and although this might lock out the boss when they try a dozen passwords from their cheat sheet, it keeps the number of false positive lock-outs manageable. Besides SSH, Fail2ban supports all services that document failed login attempts in the Syslog. After a configurable time interval, the script removes the locks again – until the next wave of attacks.

Scripts such as Fail2ban and the use of blacklists are useful and initially provide effective protection against casual attackers. Many attackers are better organized, have access to entire subnets with many IP addresses, or control a botnet with innumerable zombies in the dynamic address ranges of Asian DSL providers. Changing the IP address then immediately opens up the option of more free tests for the attacker. Attempts made by the abusers' providers either come to nothing or fail for other reasons. This means that unmanageably long lists of individual IP addresses accumulate for the duration of the different attacks, many from the same subnet or different subnets belonging to the same service provider.

To restore peace in the longer term, it can be useful to configure the packet filter far more strictly and generously block not only conspicuous subnets, but whole autonomous systems. Before creating the matching Bash script for this, quickly review the structures of the Internet and the ways to detect these structures and use them in the "Autonomous Systems on the Internet" box.

Three Steps to Success

The goal of the planned script has three stages. Like Fail2ban, you will first block individual IP addresses. If a certain number of unsuccessful access attempts originate from the same subnet, then all addresses of this prefix are blocked. The script thus prevents the attacker the opportunity for more free attempts through the use of additional hosts or simply by changing the IP addresses at the DSL connection. If access attempts come from multiple prefixes of the same AS, the script blocks all prefixes of this AS as a third stage. The required information cannot readily be discovered on a Linux server. Instead of the full routing table, you typically configure a default gateway on the server that takes care of forwarding the packets. The corresponding prefix of an IP address can only be determined with additional information.

Routing information, assuming you have appropriate access, can be obtained directly by querying the boundary router of your own AS. It has a full routing table with information about the available prefixes. Routing archives, such as the RIPE routing information service (RIS) [5] collect routing tables and prefix announcements in selected ASs. Groups of the received announcements can be download at intervals of about five minutes.

Additional Services Can Help

It is not necessary to take care of processing and analyzing these archives yourself, because services on the Internet provide easily searchable lists of the required results. You can thus automatically receive the required information via simple interfaces, such as the IP-to-ASN mapping service by Team Cymru [6], by regularly querying the assignment of an IP address to a prefix. To do so, you can use bulk queries for a large number of IP addresses to receive the corresponding ASs and the corresponding prefixes. The cleartext response (in the form of an ASCII table) can be very easily parsed with a script.

However, the information necessary for the second stage of the script cannot be provided by the Team Cymru service. Fortunately, a project at the University of Bonn provides these data but through a web service [7]. As opposed to the Team Cymru service, you need to register first, although this is free of charge. After receiving an API key, you can then query all the AS prefixes with the supplied Python client. The text output (one row per prefix) is also suitable for automatic processing within the script. The third stage is optional and depends on whether the Python client exists in the current directory. If so, it provides the information you need. If it does not exist, the script ignores this step without an error message.

iptables and Rescue Rules

The Linux iptables packet filter is used here, because it is still found on many production systems; however, an adaption to the newer nftables is quite easy with some minor manual interventions. To handle the automatically generated rules without any trouble, you create separate chains for the individual stages. In this way, you everything is clear-cut and keeps the other chains tidy. To enforce the locks, iptables must go through the chain at an early stage of processing. Make sure it is as near as possible to the top of the INPUT chain, which might require minor adjustments to existing scripts.

Automatic fire wall rule changes (e.g., packet filters or manual routing entries) can lead to unwanted side effects. The most dreaded side effect is completely locking yourself out of the server. For remote servers, especially, where you don't have console access, this is especially annoying and usually leads to additional costs for manual access by maintenance personnel. To prevent this, running small scripts as cron jobs should ensure that a rescue rule is always available.

A static route entry to another server or the standard prefix of the Internet service provider also reduces your worries, as well as the size of the white list when you first introduce the packet filtering rules. The script will allow IP addresses or prefixes to be set for which it always grants access to the server. The more precisely you define these permissions the better; otherwise, they could give potential attackers on your own network unlimited access attempts.

comment Module

To cancel locks after a certain time, the iptables comment module must also be in place, either integrated directly into the kernel or downloaded as a kernel module. Without this feature, the script works, but it releases locked areas quite quickly once the logfile no longer documents the access attempts. The duration of the lock is thus automatically based on the interval of any logrotate scripts. In the comments, note the time of the last lock. The script then automatically removes obsolete records after a configurable wait.

Script Structure

The script [8] has three functions: (1) setting up the system to use the rule chains; (2) removing the rules and setup without leaving any leftovers; and (3) executing the filter functions to detect IP addresses, prefixes, and AS numbers. Before the individual functions, the script defines the helper functions that assist iptables in creating and deleting rules (Listing 1).

001 #!/bin/bash
002
003 # Configuration
004 BLOCK_IP_THRESHOLD=1
005 BLOCK_PREFIX_THRESHOLD=3
006 BLOCK_ASN_THRESHOLD=3
007
008 # Automatically remove old rules after X seconds
009 UNBLOCK_TIME=$((60 * 60 * 24 * 30)) # Unblock after 30 days
010
011 # IPs that should never be locked (can also be a prefix)
012 LAST_RESORT_IPS='
013 127.0.0.1
014 '
015
016 FILTER='
017 cat /var/log/secure | awk '\''$5~/^ssh/ && $6~/^Failed/ && $7~/^keyboard-interactive/ && $9~/[^invalid]/ {print $11}'\''
018 cat /var/log/secure | awk '\''$5~/^ssh/ && $6~/^Failed/ && $7~/^keyboard-interactive/ && $9~/[invalid]/ {print $13}'\''
019 '
020
021 # Path to the client script at the University of Bonn
022 BART_CLIENT='./request.py'
023
024 ##########################################################
025 ##### Nothing has to be changed after this line #####
026 ##########################################################
027
028 IPTABLES=`which iptables`
029
030 # Test to see whether iptables supports comments; otherwise, no auto-unblock
031 UNBLOCK_AUTO=0
032
033 OUT=`cat /proc/net/ip_tables_matches | grep 'comment'`
034 if [ "$OUT" = "comment" ]; then
035   UNBLOCK_AUTO=1
036 fi
037
038 function chain_exists
039 {
040   TABLENAME="-t filter"
041   CHAINNAME=$1
042   if [ "$2" != "" ]; then
043     TABLENAME="-t $1"
044     CHAINNAME=$2
045   fi
046   if [ "$CHAINNAME" != "" ]; then
047     ${IPTABLES} ${TABLENAME} -nL ${CHAINNAME} 2> /dev/null 1> /dev/null
048     RET=$?
049     return ${RET}
050   fi
051 }
052
053 function delete_rule
054 {
055   # Find correct rule parameters
056   RULE=`echo ${@} | awk 'BEGIN{FS=" -j"}; {print $1}'`
057   TARGET=`echo ${@} | awk 'BEGIN{FS=" -j"}; {print "-j"$2}'`
058
059   ${IPTABLES} -S | grep --regexp="$RULE" | grep --regexp="$TARGET" | cut -d " " -f 2- | xargs -L1 ${IPTABLES} -D 2> /dev/null
060 }
061
062 function insert_rule
063 {
064   # echo "insert: $@"
065   delete_rule $@
066
067   if [ "${UNBLOCK_AUTO}" -ne 1 ]; then
068     ${IPTABLES} -I $@
069   else
070     TS=$(date +%s)
071     ${IPTABLES} -I $@ -m comment --comment "Created: $TS"
072   fi
073 }
074
075 function add_rule
076 {
077   # echo "add: $@"
078   delete_rule $@
079
080   if [ "${UNBLOCK_AUTO}" -ne 1 ]; then
081     ${IPTABLES} -A $@
082   else
083     TS=$(date +%s)
084     ${IPTABLES} -A $@ -m comment --comment "Created: $TS"
085   fi
086 }
087
088 if [ "$1" = "setup" ]; then
089   # Deactivate comments to keep these rules permanently
090   UNBLOCK_AUTO=0
091
092   echo "Initialize iptables"
093   # Create rule chains
094   chain_exists LAST_RESORT || ${IPTABLES} -N LAST_RESORT
095   ${IPTABLES} -F LAST_RESORT
096   add_rule LAST_RESORT -j RETURN
097   for IP in "${LAST_RESORT_IPS}"; do
098     insert_rule LAST_RESORT -s ${IP} -j ACCEPT
099   done
100
101   chain_exists BLOCK_IP || ${IPTABLES} -N BLOCK_IP
102   ${IPTABLES} -F BLOCK_IP
103   add_rule BLOCK_IP -j RETURN
104
105   chain_exists BLOCK_PREFIX || ${IPTABLES} -N BLOCK_PREFIX
106   ${IPTABLES} -F BLOCK_PREFIX
107   add_rule BLOCK_PREFIX -j RETURN
108
109   chain_exists BLOCK_ASN || ${IPTABLES} -N BLOCK_ASN
110   ${IPTABLES} -F BLOCK_ASN
111   add_rule BLOCK_ASN -j RETURN
112
113
114   # Create rules in the INPUT chain
115   insert_rule INPUT -j BLOCK_IP
116   insert_rule INPUT -j BLOCK_PREFIX
117   insert_rule INPUT -j BLOCK_ASN
118   insert_rule INPUT -j LAST_RESORT
119   echo "Done"
120   exit
121 elif [ "$1" = "teardown" ]; then
122   echo "Remove all rules and chains"
123   delete_rule INPUT -j BLOCK_IP
124   delete_rule INPUT -j BLOCK_PREFIX
125   delete_rule INPUT -j BLOCK_ASN
126   delete_rule INPUT -j LAST_RESORT
127
128   ${IPTABLES} -F LAST_RESORT 2> /dev/null
129   ${IPTABLES} -X LAST_RESORT 2> /dev/null
130
131   ${IPTABLES} -F BLOCK_IP 2> /dev/null
132   ${IPTABLES} -X BLOCK_IP 2> /dev/null
133
134   ${IPTABLES} -F BLOCK_PREFIX 2> /dev/null
135   ${IPTABLES} -X BLOCK_PREFIX 2> /dev/null
136
137   ${IPTABLES} -F BLOCK_ASN 2> /dev/null
138   ${IPTABLES} -X BLOCK_ASN 2> /dev/null
139   echo "Done"
140   exit
141 fi
142
143 # Test whether setup is already performed
144 chain_exists BLOCK_IP || echo "Please start '$0 setup'"; exit
145
146 IPS=""
147 OLD_IFS=${iFS}
148 IFS="
149 "
150 for RULE in ${FILTER}; do
151   # Apply the filter rule
152   eval FOUND_IPS=\`${RULE}\`
153   IPS="${IPS}
154 ${FOUND_IPS}"
155 done
156
157 # Filter by IP address, incidence count, and use of the IP threshold
158 IPS=`echo "${IPS}" | grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -n | uniq -c | awk '$1 >= '${BLOCK_IP_THRESHOLD}' {print $2}'`
159
160 # echo "${IPS}"
161 for IP in ${IPS}; do
162   insert_rule BLOCK_IP -s ${IP} -j DROP
163 done
164
165
166
167 # Finding and blocking the prefix
168 QUERY_RESPONSE=`echo "begin
169 prefix
170 $IPS
171 end" | netcat whois.cymru.com 43`
172
173 PREFIXES=`echo "${QUERY_RESPONSE}" | awk 'BEGIN{FS="|"}; {print $1" "$3}' | grep -v '^$' | sort -n | uniq -c | awk '$1 >= '${BLOCK_PREFIX_THRESHOLD}' {print $2" "$3}'`
174
175 # echo "${PREFIXES}"
176 for P in $( echo "${PREFIXES}" | awk '{print $2}' ); do
177   DEL_IPS=`echo "${QUERY_RESPONSE}" | grep "${P}" | awk 'BEGIN{FS="|"}; {print $2}' | sort -n | uniq | awk '{print $1}'`
178   for IP in ${DEL_IPS}; do
179     # echo "Delete single check for IP: ${IP}"
180     delete_rule BLOCK_IP -s ${IP} -j DROP
181   done
182   insert_rule BLOCK_PREFIX -s ${P} -j DROP
183 done
184
185
186
187 # Filter all prefixes of an AS
188 ASNLIST=`echo "${PREFIXES}" | awk '{print $1}' | sort -n | uniq -c | awk '$1 >= '${BLOCK_ASN_THRESHOLD}' {print $2}'`
189
190 if [ -x "${BART_CLIENT}" ]; then
191   for AS in ${ASNLIST}; do
192     PREFIX_LIST=`${BART_CLIENT} announced_prefixes $AS`
193     for P in ${PREFIX_LIST}; do
194       delete_rule BLOCK_PREFIX -s ${P} -j DROP
195       insert_rule BLOCK_ASN -s ${P} -j DROP
196     done
197   done
198 fi
199
200
201
202 # Discard old entries
203 if [ "${UNBLOCK_AUTO}" -eq 1 ]; then
204   NOW=$(date +%s)
205   RULES=`${IPTABLES} -S | cut -d " " -f 2-`
206   for R in $RULES; do
207     TIMESTAMP=`echo "${R}" | grep -o '"Created: .*"' | tr -d '"' | awk '{print $2}'`
208     if [ "$TIMESTAMP" != "" ] && [ "$TIMESTAMP" -lt $(($NOW - $UNBLOCK_TIME)) ]; then
209       delete_rule $R
210     fi
211   done
212 fi
213 IFS=${OLD_IFS}

Rules are deleted by specifying the filter conditions, or by specifying the line number (call iptables with --line-numbers). Using line numbers entails the risk that changes have been made displaying the current rules and deletion through other scripts and that the script thus changes the wrong lines. To prevent this, you use the filter condition itself to remove the entries. Lines 53 to 60 is the delete_rule function. To account for the automatically added timestamp in the optional comments, the function first takes the rules apart and then puts them back together with a slightly modified parameter order.

Processing Chain-by-Chain

The setup and teardown functions can be executed by simply appending the terms as parameters of the script. During setup (lines 88 to 120), the script first creates the three chains if they do not exist. It then removes all the rules from the chains so there is always an empty initial status at setup time. The end of each chain requires a rule to return the package to the calling chain. Without this rule, iptables does not examine the package. The required steps are shown in lines 109 to 111, which use the BLOCK_ASN chain as an example. The helper function chain_exists checks upfront to see whether a chain already exists; add_rule then adds the rule. At the end of the function, the script adds the branch to the new chains in the INPUT chain (lines 115 to 118).

When removing rules with teardown (lines 121 to 141), the functions work in the opposite direction. First, the script removes the rules from the INPUT chain; then, after the mandatory clear-up, it removes the chains themselves (e.g., ${IPTABLES} -X BLOCK_ASN only deletes the BLOCK_ASN chain if it no longer contains any rules). After running teardown, no traces of the script exist any longer in the packet filter.

The core functionality, besides multilevel capability, lies in the filter rules for discovering the potential attackers' IP addresses. To make the script as flexible as possible, the filter rules are arbitrary calls for displaying and filtering logfiles, as well as regular expressions for digging the IP addresses out of the log entries.

Basically, you could call other DIY scripts that output the appropriate IP addresses line by line. The filters then run within a subshell later. For the call, it is necessary to mask these accordingly. Using an example of two simple log entries from the /var/log/secure file on a Gentoo system, the next section explains the approach outlined in Listing 2, in which you create one or more rules of this type for each logfile with relevant information.

# Feb 22 08:00:00 hostname sshd[1234]: Failed keyboard-interactive/pam for root from 1.0.0.1 port 4321 ssh2
# Feb 22 8:00:10 AM hostname sshd[1234]: Failed keyboard-interactive/pam for invalid user mysql from 1.0.0.1 port 4322 ssh2
FILTER='
cat /var/log/secure | awk '\''$5~/^ssh/ && $6~/^Failed/ && $7~/^keyboard-interactive/ && $9~/[^invalid]/ {print $11}'\''
cat /var/log/secure | awk '\''$5~/^ssh/ && $6~/^Failed/ && $7~/^keyboard-interactive/ && $9~/[invalid]/ {print $13}'\''
'

The two entries in the logfile are based on the format of the software deployed. They can vary from system to system; therefore, it is important to test the filter expression thoroughly in advance. The example shows that the SSH daemon can generate different log entries. In the first case, the login for an existing user was entered for root on the system. The second entry also notes that the user ID mysql does not exist on the system. To prevent the filter from extracting incorrect values, the filter rule should be as precise as possible.

The cat command writes the contents of the logfile to the standard output channel. With the help of awk, you can check the individual fields (the default delimiter is the space character) in each line. The terms ssh, Failed, and keyboard-interactive clearly identify a failed login attempt using SSH. However, the IP address occurs in the first entry in field $11, and in the second entry in field $13; thus, you will want to check field $9, too. If this starts with a string that is not equal to invalid, the first filter delivers a result; otherwise, the IP is found two fields to the right and extracted by the second filter.

Filter Options

The number of filters is limited only by the resources available in Bash. Multiple line-wise searching through logfiles can easily try your patience given large files. You can optimize your search with more complex expressions and by regularly calling a logrotate script. At the end of iterating through all the filter rules, all IP addresses that were found are now stored in the ${IPS} variable. Each result that does not look like an IPv4 address is again subjected to targeted filtering (Listing 1, line 158). Sorting and counting multiple identical entries produces the data basis for further steps. Entries that occur less frequently than the number value in ${BLOCK_IP_THRESHOLD} will not be blocked and can be removed directly from the collection.

The first stage blocks all IP addresses that still exist in this list (lines 161 to 163). The insert_rule helper function adds the BLOCK_IP rule to the chain. In the second stage (lines 168 to 173), the script now sends all IP addresses as a bulk request to the IP-to-ASN service offered by Team Cymru. The prefixes listed in the reply are sorted and counted in line 173. If a prefix occurs less frequently than the value in the variable ${BLOCK_PREFIX_THRESHOLD}, it is removed. All remaining prefixes are then added to the BLOCK_PREFIX chain, and because the entries in the BLOCK_IP chain are no longer necessary, they are removed (lines 175 to 183).

The third optional stage of the script now checks for the occurrence of different prefixes of a single AS. For this, the script again uses the results from querying the IP-to-ASN service and this time filters and counts the ASNs from the return (line 188). If an AS occurs at least as often as configured in the ${BLOCK_ASN_THRESHOLD} variable, it then searches for all available prefixes belonging to this AS using the service provided by the University of Bonn and adds them one by one to the BLOCK_ASN chain (lines 190 to 198). Similarly, it removes the corresponding rules from the BLOCK_PREFIX chain, so they are not duplicated.

The end of the script removes the outdated entries from the chain if the iptables comment module is available (lines 203 to 212). The threshold value for removing the old entries is configured in the variable ${UNBLOCK_TIME}, which indicates the delta to the last insertion in seconds. Without the comment module, the entries disappear as soon as the IP addresses no longer appear in the logfile itself.

To update the packet filter regularly, run the script periodically as a cron job. Depending on the attack frequency, intervals of once a minute to every quarter of an hour makes sense. Regularly running logrotate prevents large logfiles and obsolete entries in the packet filter.

Conclusions

In this article, I described the structure of a script for restrictive automatic updates of the Linux packet filter based on free definable filter rules. When using tools for automatic adjustments, it is important to maintain your white lists or emergency rules that still allow the administrator access to the remote server system.