There’s a lot of focus on proactive security testing and rightly so. It’s the best way to stay out of hot water. But what happens when the going gets tough and you end up missing a vulnerability that leads to a web security breach? There’s not a lot you can do other than plan in advance to minimize the incident’s impact to your business.

I recently worked on such a project that was chock full of lessons for all of us involved and wanted to share it here in case it can help others.

It started out when my client started receiving password reset notifications from several of their email and development collaboration accounts. Everyone involved immediately changed the appropriate account passwords and it was assumed that everything was resolved. However, within a few days, a developer noticed that a suspicious update was made to their source code within their collaboration tool.

The plot thickened.

It was then discovered that an old account used to access the collaboration tool had somehow been compromised – likely through the original email breach that was generating the password reset notifications. This account belonged to a former development team member but was no longer in use.

My client dug in even further by analyzing their server logs and firewall rulebase to look for additional signs of compromise. Nothing else was uncovered. To ensure no stone was left unturned, they hired me to perform a security assessment of their web environment to look for any additional low-hanging fruit or signs of compromise on their web servers. Again, no smoking gun but it was a beneficial experience.

Some of the lessons learned were:

Ensure you have a full account of user IDs that can access your production and development environments including accounts for hosting/colo systems, content management systems, and source code repositories.

Test your source code for vulnerabilities, especially if you’re using a development collaboration tool that allows outside sources to commit updates to code repositories.

No matter how far you go in securing your network, your applications, your databases, and your source code, all it takes is one email account compromise to provide criminal hackers with everything they need to eventually insert malicious code or otherwise break into your web environment. Use two-factor authentication for online accounts where it’s available.

Scan the IP-based hosts that make up your web environment with a traditional OS vulnerability scanner.

Have an incident response plan in place to help guide you along in a methodical fashion when breaches do occur.

Document a core set of security policies and standards that include web systems in their scope outlining how things work in your environment. If a breach turns into a lawsuit, you’re going to be asked for this documentation.

The limited IT and development staff working for at my client’s business dropped everything else after they discovered the initial breach. Although it’s not possible for all businesses to do that in every situation, I think it was a wise approach given what there was to lose had that malicious code made its way into production. Given what they had to work with, I don’t think my client could have responded any better.