My VMWare ESXi 4 server appears to be under a Denial of Service attack. I am getting massive packet loss to the server (60+%) and am barely able to load any services on the VMs running on the host.

I have Cacti installed but cannot load it due to the attack. I can SSH in to the VMware host. Are there any commends I can run to either determine where the attack is coming from, or block all IP addresses except mine so that I can load Cacti again to troubleshoot?

Yes, thanks @MichaelHampton. They told me it was a DoS / DDoS but were unable to assist beyond that. If I can't get this resolved soon I am calling them again. I was hoping maybe SF would be more helpful, because they were not!
–
JoshJul 27 '12 at 19:36

3 Answers
3

I would say first and foremost would be to call your datacenter and see if they can block the offending IP with their equipment. Hopefully their hardware has the bandwidth to handle something like that, which will then at least allow yours to start functioning like normal.

Yeah, it's because they want to sell me a $500/mo DDoS prevention plan rather than helping a long time customer. It's time for a new ISP.
–
JoshJul 27 '12 at 19:48

@Josh, do you have console access to this machine? If not, you should make plans to do so going forward.
–
Mike PenningtonJul 27 '12 at 19:54

Thanks Mike. I did when I performed the initial setup through a IP KVM, but I stopped that service after the install because I was able to do all I needed to through SSH / vSphere center...
–
JoshJul 27 '12 at 19:55

The ISP was unable to determine the cause of the traffic, but what they were able to do was null-route all the IP addresses assigned to this server at the network switch. Then, one-by-one we removed the null routes, until we determined which IP addresses were being attacked. Once the target IPs were null-routed, the problem went away and I am able to access the server again.

I am now going to console in to the affected VMs and start tcpdump, and then remove the null-routes to those VMs. This will allow me to find the source IPs of the attack, which can be blocked by my ISP before traffic from them enters the core network.

There is no tcpdump on vmware ESXi, and I am getting so much packet loss that when I try to run it on the virtual machines, it just locks up.
–
JoshJul 27 '12 at 19:23

1

you misunderstand. Configure a span port on your switch so you can monitor the traffic with a laptop. Span whatever Vlan your server is in.
–
Paul AckermanJul 27 '12 at 19:24

Sorry for not being clear, I'm in the middle of trying to fix this ASAP as all my customers' sites are down. This is a managed server, I have no access to the switch. I will ask the datacenter if they can do that, however, thanks
–
JoshJul 27 '12 at 19:28

ahh. I see. In that case, can you run vm-support and get VMware to help you diagnose?
–
Paul AckermanJul 28 '12 at 1:09

I don't know the specifics but the vm-support utility can diagnose various performance issues so it may provide insight into the network traffic.
–
Paul AckermanJul 28 '12 at 1:36