A few days ago, I lived through a DDOS attack at one of the sites I run. It started with the general unavialability of Apache, although the system was idle and the ssh access functioned perfectly. As I already had some experience with DDOS attacks, I immediately looked at the network connections using netstat and saw over 300 IPs sending SYN_REC packets in mass. The attack was much more powerful than I ever experienced.
~# netstat -n -p | grep SYN_REC | awk '{print $5}' | awk -F: '{print $1}'| sort |uniq |wc -l
310
I immediately started the APF firewall with RAB enabled and a script to block the offending IPs in iptables, as I had these at hand from the previous DDOS attack:
BEGIN {
pipe = "netstat -n|gawk '{print $5}'|gawk 'BEGIN{FS=\":\"}{print $1}'|sort|uniq -c |sort -n"
while (( pipe| getline) > 0 ) {
if ($1 > 29 && $2 != "") {
command="/sbin/iptables -I INPUT -s " $2 " -j DROP"
date="date"
command | getline result
date | getline dateresult
print dateresult " - command: " command ", result: " result >>"/var/log/firewall.log"
}
}
close(pipe)
}
After a few hours it became clear that the IPs are rotating too often and the size of the botnet is well over 10.000, so I was left with a choice of either dying under DDOS or dying due to the performance hit of the large pool of IP addresses in iptables. The SYN_REC part was being filtered easily by syncookies, APF activated them on launch. The real problem was the HTTP flood.
I found out about ipset quite soon, but could not use it because my hosting plan did not allow me to recompile the kernel to enable ipset support. Fortunately, 50% of my traffic is very limited geographically, so I enabled the Apache geoip module to filter all but one country's IPs. After that, I just had to whitelist the main search engines
# google
allow from 66.249.64.0/19
allow from 72.14.192.0/18
allow from 74.125.0.0/16
# Rambler
allow from 81.19.64.0/19
# Yandex
allow from 213.180.192.0/19
allow from 87.250.224.0/19
allow from 77.88.0.0/18
Deny from 81.82.151.173
and to blacklist the botnet members originating from that country. This way, I could survive the DDOS with some Apache tweaking that consisted in allowing more Apache instances live shorter periods of time in order to handle bots efficiently and decline to serve them while allowing legitimate users to connect. Here's how my Apache config looked like:
ServerLimit 1024 Timeout 5 MaxKeepAliveRequests 0 KeepAliveTimeout 15 StartServers 512 MaxClients 1024
I survived this way for 36 hours until I joined an experimental project of some good guys doing DDOS mitigation professionnally, but this is another story.
P.S. The attack peaked at 4Mbps of incoming traffic with a steady 1Mbps for 5 days. Some would probably classify this attack as light, but my overloaded server and myself had great pains in dealing with it.