There was a sudden unusual peak of outbound traffic from my hosted dedicated server (OS - Windows Server 2008 R2). The hoster threatened to block the server if the problem is not solved.

There are an ASP.NET MVC application and a SQL Server 2008 Express database installed on the server.

After that I've installed all missing security patches, changed and hardened passwords, scanned the server with Microsoft Malicious Software Removal Tool (nothing found). But I don't feel the problem is solved.

Could you provide some details on the attack which was made? Logs, etc...
–
Dog eat cat worldAug 24 '11 at 13:00

@Dog eat cat world, In fact this server is not in production mode and it was for sure not my app's activity. The day before this peak I've downloaded and installed on the server lots of updates - maybe it is somehow connected to the subsequent outgoing peak. Unfortunately, so far I don't have any logs regarding that traffic
–
remAug 24 '11 at 13:54

Have you used any tools to determine what the traffic is and it's destination?
–
NotMeAug 24 '11 at 18:27

@Chris Lively, unfortunately I have not used any. Just because of the lack of the knowledge in this field. What tools would you recommend?
–
remAug 24 '11 at 19:00

It is important to use a bootable scanner. Some modern malware can completely make itself invisible during the boot process and thereby hide from scanners, even in safe mode. By booting from a CD, you don't give any malware a chance.

The first thing to do is use your firewall (prefer hardware firewall over software on the machine). Configure it to detect malicious outbound traffic and block it. With this setting you shall avoid those unwanted peaks of bandwidth usage, or at least minimise its occurrence. Indeed this may impedes your activity, but not as much as being blocked by your ISP.

Furthermore, the firewall shall alert you and help you analyse the situation to identify what is causing the problem.

Second thing to do is to scan the machine for viruses and possible malware. I know you have done this, but let me reminds you that you need to do it in an offline mode, not from the potentially infected system because the tools could be compromised by virus already in place.

Maybe you will find more convenient/economic to backup data and set up a new system from scratch.

A sudden peak in network traffic does not indicate anything other than some processes are taking place.

I assume when you say "sudden unusual peak", you are saying that there was a high amount of traffic (that usually do not take place), and it only lasted a brief time.

Could it be a backup routine? That would also generate a high amount of network traffic for a small time.

Perhaps the problem lies at your service provider? If a sudden network peak is enough for them to close you down, I would look elsewhere for another provider. A funny thought is that your provider is effectively threatening to perform a Denial of Service attack on you.