A first-hand account of a DDOS attack mitigation

A few days ago, I lived through a DDOS attack at one of the sites I run. It started with the general unavialability of Apache, although the system was idle and the ssh access functioned perfectly. As I already had some experience with DDOS attacks, I immediately looked at the network connections using netstat and saw over 300 IPs sending SYN_REC packets in mass. The attack was much more powerful than I ever experienced.

~# netstat -n -p | grep SYN_REC | awk '{print $5}' | awk -F: '{print $1}'| sort |uniq |wc -l

I immediately started the APF firewall with RAB enabled and a script to block the offending IPs in iptables, as I had these at hand from the previous DDOS attack:

 pipe = "netstat -n|gawk '{print $5}'|gawk 'BEGIN{FS=\":\"}{print $1}'|sort|uniq -c |sort -n"
 while (( pipe| getline) > 0 )  {
   if ($1 > 29 && $2 != "") {
     command="/sbin/iptables -I INPUT -s " $2 " -j DROP"
     command | getline result
     date | getline dateresult
     print dateresult " - command: " command ", result: " result >>"/var/log/firewall.log"

After a few hours it became clear that the IPs are rotating too often and the size of the botnet is well over 10.000, so I was left with a choice of either dying under DDOS or dying due to the performance hit of the large pool of IP addresses in iptables. The SYN_REC part was being filtered easily by syncookies, APF activated them on launch. The real problem was the HTTP flood.