Scalability Blog

Scaling tips, insights, updates, culture, and more from our Server Experts.
 

Protecting Your Managed Server From Packet Flood

packet-flood-header

There are instances when your DNS server, such as BIND or PowerDNS, comes under a heavy packet flood.  Here is a network activity on two nameservers undergoing UDP flooding to port 53:

network-utilization-1

network-utilization-2

To mitigate this issue, we need to do a little investigation as to where the packets are coming from.  Tcpdump is an excellent tool for this and combined with iptables can be used as quick measure against the UDP flood.  We can use the following command to check for connections to port 53:

# tcpdump -i eth0 -nnn port 53 and not port 22

tcpdump

Now it is important to establish which subnets are friendly and which ports can be ignored.  In our case, we’ll ignore 69.55.x.x/16 subnet and port 22 (SSH).

Although we have established that the traffic is UDP, technically the DNS server also listens on TCP port, so we will include both of these protocols when banning the flooder’s IP address later.  By filtering out this output we drop the last part of the tcpdump output, which is the source port of the IP address that was used to connect to this server.  It is usually a value between 1024 and 65535.  The following tcpdump command would let us see all the IP addresses that are connecting to port 53 on interface eth0 :

tcpdump -i eth0 -nnn port 53 and not port 22 and not src net 69.55 | awk '{print $3}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}'

This would generate a list of IPs that are currently connecting to port 53.  Once we have verified that the IPs seem foreign and repetitive, we can begin logging them into a file that will be used later:

tcpdump -i eth0 -nnn port 53 and not port 22 and not src net 69.55 | awk '{print $3}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}' >> /root/ips.txt

To parse through this file, we will use sort and uniq commands:

cat /root/ips.txt | sort | uniq -c | sort -n

A quick parsing through the log file reveals IP addresses where the spam originated from:

quick-parsing

The first column is number of instances this IP appears in log file, and second column is the IP address itself.  We can now ban these repetitive IPs with iptables :

iptables -I INPUT -s 85.236.105.27 -j DROP -m comment --comment "UDP Spam"
iptables -I INPUT -s 71.174.225.224 -j DROP -m comment --comment "UDP Spam"
...

Now as you can see, there is a stark contrast between legitimate DNS requests which were around 100-180 in our log window, and spamming requests.  The amount of spam that these IPs generated can be seen with iptables using iptables -nL -v:

# iptables -nL –v

iptables

Once we are done with this set of IPs, we can clear out the log file:

# :> /root/ips.txt

Since spammers often use scripts that modify their source IP address, this process can be repeated every few minutes to capture new IP addresses:

# tcpdump -i eth0 -nnn port 53 and not port 22 and not src net 69.55 | awk '{print $3}' | sed 's/\./ /g' | awk '{print $1"."$2"."$3"."$4}' >> /root/ips.txt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
2008557 packets captured
2008643 packets received by filter
0 packets dropped by kernel
# cat /root/ips.txt | sort | uniq -c | sort –n

cat-root

 

We can verify that the network activity due to spam has decreased on both servers:

network-activity-1

network-activity-2
It is a good idea to occasionally review the output of “iptables -nL -v” and to remove entries that are no longer spamming.  Keeping your list of rules small helps minimize CPU loads due to iptables.

This entire process could be automated to automatically scan for spamming IPs and place them in iptables rules, re-scan the iptables output every minute and drop rules that are stale.  Unfortunately this will not work if the spammer is constantly changing his IP address and only using 1-2 IPs at a time for requests, at a very high rate.  In this case, we need to use a more restrictive iptables rule that would flag packets for DNS inquires of ANY type and drop them if more than 4 were requested per minute :

# iptables -I INPUT 1 -i eth0 -d MY_IP -p udp --dport 53 -m string --from 50 --algo bm --hex-string '|0000FF0001|' -m recent --set --name dnsanyquery
# iptables -I INPUT 2 -i eth0 -d MY_IP -p udp --dport 53 -m string --from 50 --algo bm --hex-string '|0000FF0001|' -m recent --name dnsanyquery --rcheck --seconds 60 --hitcount 5 -j DROP

With this rule, there is still a heavy influx of UDP packets, but 99% of them are dropped, and there is very little outgoing traffic:

UDP
A similar approach can be taken with webservers and high connectivity rate to port 80 using netstat.  However, generally you don’t want to have a large set of iptables rules on production servers because it will increase CPU and memory usage.  An Nginx proxy in front of your webserver can be used to mitigate spam, and also add caching if you so desire.