Having bandwidth doesn't necessarily translate to low latency -- it could be simple bad luck, the routing between you and the server hits a high-traffic area perhaps.

Of course, lacking any information at all on you ISP, location, server location, etc, it's impossible for us to diagnose. Try running a trace route to the server, see if that produces any useful information.

I personally run the Leatrix latency fix. While it does put a greater load on the processor, it keeps my latency in double-digits.

wow, im using the leatrix and that brought me down to like 100ms, im not really noticing the stress on processor though, sweet tthanks tho ;D