This leads to delays in logging in via ssh and other problems such as not being able to connect to the local database server. My initial thoughts were to look at the mbuf usage but I couldn't see any problems here;

OK, I figured out why the problem was occurring; port 22 was getting hammered with over 1000 connections a second (i forgot to check pfstat, auth.log and tcpdump which gave me glaring indications of a flood). I've decided to change the sshd listen port to something other than 22 and wait til my attacker finds it out.