Make sure NEW incoming tcp connections are SYN packets; otherwise we need to drop them:

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

Force Fragments packets check

iptables -A INPUT -f -j DROP

drop incoming malformed XMAS packets:

iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP

Drop NULLED packets:

iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP

Add the limitations to /etc/sysctl.conf

net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_synack_retries = 2

You can also use CSF firewall. Just set next parameter:

CT_LIMIT = 5

Testing

To test your server for vulnerability, you can use Backbox Linux, it has a various testing tools preinstalled. For example, to test your domain for resistance to DDoS attacks you can use slowhttptest which is in Backbox installation. Example command: