It’s been quite a while since I wrote or updated DFW, the I)ruidic FireWall. Included with that utility is a default iptables firewall policy which the user can use directly, tweak to their liking, or completely throw away and start over from scratch. NetFilter (iptables) has come a long way since I was actively working in the firewall space and regularly maintaining the DFW utility, so I thought it high time that I update the firewall policies on my servers to take advantage of some of it’s newer features, and in doing so update DFW’s default policy with some extra bells and whistles. The primary goal I wanted to accomplish was to significantly clean up my firewall logs, as the Internet is an extremely dirty and hostile place to connect a computer to. Regularly my logs would be full of default drop log entries for entire port-scans, the same worm-infected hosts connecting to the same closed ports over and over and over again, and other general random connection attempts.
The first thing I decided to do was add some rate-limiting to the allowed services. In the older policy, I did have rate-limiting, but it was policy-wide, and was intended to only be turned on while under active attack such as a DDoS. It essentially had the effect of rate-limiting all inbound traffic from a single source to one connection per second. While this is useful for it’s intended purpose, taking a service-centric approach allows for much more reasonable approaches to rate-limiting various types of traffic. Because DFW’s default policy is fairly strict by default, about all that’s allowed inbound is ICMP and SSH. Attacks against SSH rose sharply recently when SSH credential brute-forcing tools started showing up on the scene. One very effective way to curtail this attack is to severely rate-limit new SSH connections from a single source, and blacklist that source if they connect too many times beyond the allowed amount in a short time period, which brings me to the second feature I wanted to implement, a dynamic blacklist for bad sources. A number of the policy items I’ll be discussing here make use of this blacklist, so near the top of the INPUT policy table we need to both implement and handle some management of the blacklist:
echo "dropping blacklisted for 1 day"
iptables -A INPUT -m recent --name blacklist --rcheck --seconds 86400 -j DROP
iptables -A INPUT -m recent --name blacklist --remove
iptables -A INPUT -m recent --name badconns --rcheck --seconds 90 --hitcount 3 \
-m recent --name blacklist --set -j LOG --log-prefix "input blacklisting "
iptables -A INPUT -m recent --name blacklist --rcheck --seconds 86400 -j DROP
What these rules accomplish is to first check the source address of the packet against a list called “blacklist” to see if it has been added to that list within the last 24 hours. If it has, the packet is immediately dropped silently. The second rule handles the removal of source addresses from the blacklist after 24 hours, since any packets hitting this rule have obviously made it past the drop by the previous rule. The third rule checks the source address against a list of addresses that have been initiating bad connections called “badconns” to see if that source has initiated 3 or more bad connections in the last 90 seconds. If it has, the source is added to the blacklist and a note is sent to the log to indicate that the source has been blacklisted. This rule handles bulk blacklisting of sources that have been sending bad connections to otherwise allowed services, such as sources that have been added to the bad connections list by any per-service rate-limiting rules. The final rule then handles dropping any packets that just got blacklisted by the third rule. This rule is needed the second time to prevent duplication of log entries by the other blacklisting rules. All of this is done near the beginning of the INPUT policy because we want to immediately discard traffic from hosts that have been blacklisted and not waste any more processing time on them than is necessary.
Now that the blacklist handling rules are in place, we can add policy rules for controls such as SYN-flood detection, rate-limiting for various allowed services, and the default cleanup drop rule that collects sources of bad connections in a bad connection list.
First up, rate-limiting of various types of TCP packets:
echo "SYN: 60 packets per second"
iptables -A INPUT -m state --state NEW -p tcp -m tcp --syn \
-m recent --name synflood --set
iptables -A INPUT -m state --state NEW -p tcp -m tcp --syn \
-m recent --name synflood --rcheck --seconds 1 --hitcount 60 -j DROP
echo "RST: max 2 packets per second"
iptables -A INPUT -p tcp -m tcp --tcp-flags RST RST \
-m limit --limit 2/second --limit-burst 2 -j ACCEPT
These rules add SYN packet sources to a list called “synflood” and then checks to see if more than sixty have been sent in under one second. While sixty new connections per second from a single source is probably still rather extrenuous for most servers or devices, there are services with rather high connection expectancy so this limit makes a reasonable approach for rate-limiting ALL inbound connections from a single source, regardless of what port those connections are destined to. The last rule limits RST packets to two per second to help prevent TCP RST attacks against established connections.
We also want to rate-limit ICMP to prevent ICMP floods:
echo "ICMP: max 6 packets per second" iptables -A INPUT -p icmp -m icmp -m limit --limit 6/second --limit-burst 2 -j ACCEPT
The most common use for ICMP is Echo Request and Echo Reply, which generally send 1 packet per second by default. There are also the occasional ICMP messages related to TCP connections and UDP datagrams, but they are generally very low-rate. Six ICMP packets per second system-wide is likely more than reasonable unless you have an extremely high bandwidth system.
Next, rate-limiting of the SSH service:
echo "ssh: max 3 connections in 2 minute window"
iptables -A INPUT -p tcp --syn --dport 22 -m state --state NEW \
-m recent --name ssh --set
iptables -A INPUT -p tcp --syn --dport 22 -m state --state NEW \
-m recent --name ssh --rcheck --seconds 120 --hitcount 3 \
-j LOG --log-prefix "input drop ssh-ratelimit "
iptables -A INPUT -p tcp --syn --dport 22 -m state --state NEW \
-m recent --name ssh --rcheck --seconds 120 --hitcount 3 \
-m recent --name badconns --set -j DROP
These rules first add any source connecting to SSH (port 22) to a list of addresses called “ssh”, then checks to see if the source has connected 3 times in the past 120 seconds. If it has, it logs the rate limiting of the source and adds the source to the list of bad connections (badconns), which are used to trigger the blacklisting rules. The logic here is that any human shouldn’t need more than two connections every couple of minutes, even if they mistyped their username or passwords during the first connection, and then once rate-limited, the soruce won’t actually get blacklisted unless it has already been making other bad connections, or it continues to try and connect to SSH after being denied via rate-limiting.
Finally, the policy’s cleanup DROP rule will add anything hitting these rule to the bad connections list, and blacklist sources that have too many bad connections:
echo "default cleanup drop, blacklists on 3 bad connections in 90 seconds"
iptables -A INPUT -m state --state NEW -m recent --name badconns --set
iptables -A INPUT -m recent --name badconns --rcheck --seconds 90 --hitcount 3 \
-m recent --name blacklist --set -j LOG --log-prefix "input blacklisting "
iptables -A INPUT -m recent --name blacklist --rcheck --seconds 86400 -j DROP
iptables -A INPUT -j LOG --log-prefix "input drop cleanup "
iptables -A INPUT -j DROP
These are the final set of rules in the INPUT policy, which handles any and all packets not previously accepted or dropped elsewhere. As you can see, the source of any packet that hits the first rule in this set of rules gets added to the bad connections list. The second rule checks to see if the source has been added to this list 3 times in the last 90 seconds, and if so it adds the source to the actual blacklist itself. The third rule is a duplicate of the one we also have a the top of the policy table that silently drops packets sourcing from addresses in the blacklist, and the final two rules log and drop packets that were not dropped silently.
Among some other minor changes to the DFW tool’s default policies, those are the most significant. Let’s now look at some (trimmed) results of these changes that have shown up in the logs. Overall, the logs are much, much cleaner. Where I used to have pages and pages of default cleanup drop log entries for things like port scans, I now have log entries for the initial three packets being dropped with a log entry indicating that the source has been blacklisted. Here’s what it looks like when a randomized port-scan from somewhere in London gets blacklisted:
15:44:38 input drop cleanup SRC=85.234.149.27 DST=X.X.X.X PROTO=TCP SPT=2102 DPT=7756 SYN 15:44:38 input drop cleanup SRC=85.234.149.27 DST=X.X.X.X PROTO=TCP SPT=2104 DPT=62000 SYN 15:44:38 input blacklisting SRC=85.234.149.27 DST=X.X.X.X PROTO=TCP SPT=2106 DPT=1669 SYN
And here’s what an SSH brute-force attempt from China looks like getting rate-limited and then blacklisted:
17:09:13 input accept ssh SRC=60.172.219.2 DST=X.X.X.X PROTO=TCP SPT=57772 DPT=22 SYN 17:09:23 input accept ssh SRC=60.172.219.2 DST=X.X.X.X PROTO=TCP SPT=57773 DPT=22 SYN 17:09:28 input drop ssh-rate SRC=60.172.219.2 DST=X.X.X.X PROTO=TCP SPT=57774 DPT=22 SYN 17:09:31 input drop ssh-rate SRC=60.172.219.2 DST=X.X.X.X PROTO=TCP SPT=57774 DPT=22 SYN 17:09:33 input drop ssh-rate SRC=60.172.219.2 DST=X.X.X.X PROTO=TCP SPT=57775 DPT=22 SYN 17:09:36 input blacklisting SRC=60.172.219.2 DST=X.X.X.X PROTO=TCP SPT=57777 DPT=22 SYN
Once I’ve let these rules bake on my servers a bit I’ll be including them in an updated policy for the DFW utility.