Have you tryed the server without addons? Just to check if it's the plugin that bugs?
Check if your server is overloaded when there are so many people on the server. Disable Hyper threading, if you haven't done that already.