Archive

This past week, as most security professionals know by now, a severe bug was discovered in the Cloudflare content delivery network’s service by noted researcher Tavis Ormandy. Organizations should pay attention when Tavis reaches out, just like they should when Brian Krebs reaches out – there’s a damn good reason, and it’s probably important. I’d like to publicly commend the team at Cloudflare for handling this as well as anyone could in that situation. They took him seriously, responded quickly, and worked their butts off to get the problem handled. From everything I’ve seen, a model vendor response to a serious issue. If you’re just learning about this, here are some links to get the background:

Rather than just be another blog talking about this issue (I think it’s been covered well enough elsewhere), I’d rather focus on the bigger picture for a minute. As someone who works with many organizations on their virtualization and cloud architecture, strategy, and more, I believe this incident is one we should really take to heart for a few reasons.

The nature of security architecture has been changing for a few years now. CDN services like Akamai and Cloudflare are almost mandatory for many organizations who need security and availability controls applied to their internet traffic. The Cloud Access Security Broker (CASB) market is also growing rapidly, and processes organizations’ cloud data.

The entire nature of trust is changing with these trends. We’re relying on SSAE 16 SOC 2 reports and other *extremely* superfluous documentation offered by the service providers to guarantee that security best practices are being followed. What we really don’t know, however, is the TRUE nature of the software and architecture in place within these environments, because the providers never offer this. Ever.

We’re exposed using these services, of course. I’m as bullish on cloud as anyone. But we are not really modeling our threat surface around these services, and occasionally things will go dramatically wrong. I believe this is an opportunity for those in the bug bounty industry to shine – where we have the least visibility, and the most trust assumed. Not to knock Cloudflare, but Tavis called out their bug bounty – a T-shirt. That’s not a bug bounty, that’s just a token to say you have a bounty program. If you want the best hackers to REALLY find your issues for you, ethically and professionally, you need to step up. More than that, WE (the community using cloud providers and the brokering services that transit our data to and fro) need the best hackers in the world looking at these technologies with a much more scrutinizing eye than a CPA firm with a checklist.

I think this will hit a tipping point sooner rather than later, sadly. Cloudflare handled the problem admirably, and we really don’t know how exposed people’s data was (although everyone and their mothers are speculating wildly, of course, this being the infosec community). That may not be the case forever – sooner or later, someone is going to turn one of these CASBs or CDNs into the world’s biggest Man-in-the-Middle tool, and things are really going to get ugly.

We’ve never really been adept at dealing with insider threats. Some organizations have internal detection and monitoring programs, usually aligned with anti-fraud efforts, and some also include more robust forensics programs to look for evidence after-the-fact, but we still have a problem with insiders. With the proliferation of virtualization and cloud computing, we have more trouble than ever. There are two trends I see that explain this.

First, let’s talk virtual environments. A number of things tend to happen in virtual infrastructure that can lead to poor privileged user management and monitoring practices. First, many shops hand virtualization over to an existing admin group, like say…the Windows team. Not a great move, for a lot of reasons. This team still has to manage their existing systems and infrastructure, like Active Directory, DNS, and other platforms and applications. This means they’re part-time virtualization admins, at least for a while. A lot of folks think virtualization is easy, and it is…to a point. But virt technologies can suffer from neglect just like any other systems and apps can, and missing patches and failing to implement configuration controls can have a devastating effect. But relative to the point of insider control and monitoring, this arrangement usually leads to shortcuts in the way that admins log in and manage the environment Many use generic administrator logins, including the local Admin account on Windows systems running vCenter. AD integration is easy, and highly recommended, and this can help with audit trails, but the practices are still poor – often the full Admin role is assigned within management platforms, with little to no role assignment or separation of duties. Coupled with the minimal logging often done in these environments and potentially generic admin account IDs…a recipe for disaster. One disgruntled admin could take out the entire environment, at least for a while.

What to do about this? Well, the most effective way to approach this issue is to follow a simple regimen, none of which is really new at all:

Before deploying virtualization, or even once you have it up and running, set time aside to carefully plan and assign roles for VM admins, cloud admins, network teams, dev and DBA teams, etc. The major vendors, certainly Microsoft and VMware (XenServer role granularity is a bit meh), offer plenty of features to properly create and manage roles.

Ensure your management interfaces to all components, including integrated and 3rd-party pieces in vBlock (Cisco UCS, EMC Ionix and Symmetrix, etc) and private cloud (vCloud, System Center varieties, etc.) are on a separate segment that you control very tightly. Ensure you have monitoring in place for this segment (behavioral and traditional signature-based) and also logging on each management platform.

Have all administrators manage systems via a bastion host or “jump box”, which can be anything ranging from a Windows server you RDP into to vSphere Management Appliance or commercial options like the HyTrust appliance. Better management control, better audit trails, more of a pain in the ass for admins, but…something you should do.

I see a lot of organizations where security teams aren’t really monitoring the virtualization and cloud admins. This should change, quickly. Speaking of monitoring virtualization and cloud admins, let’s talk about the second trend, which is moving resource to public/hybrid/community clouds. There’s really two ways to look at the insider scenario here. The first way, while pretty defeatist in nature, could certainly resonate with some folks – you’re f**ked. You are pretty much going to have to rely on the cloud provider to do internal monitoring and privileged user management. Well, THAT’s depressing. The other way to look at this is via the standard argument for auditing and assessing providers – via SSAE 16, ISO 27001, and CSA STAR or other questionnaires and responses like those in the CSA CCM and CAIQ. At the moment, there’s really no way to monitor cloud admins actively yourself, so you’re at their mercy technically. You’ll have to rely on what the provider tells you, and continually check to make sure they’re doing what they say they’re doing. A great guide to insider threats in cloud environments has recently been published by the folks at CERT, titled “Insider Threats to Cloud Computing“. It breaks down the different types of cloud admins, what data and systems/apps they have access to (typically), and what you should be looking for when talking to providers about this. I highly recommend reading it.

Hopefully, the insider threat in both virtual and cloud environments is on your radar. If it’s not, it definitely should be.

I’m a big fan of the work of Tim Ferriss. While I haven’t quite managed the 4-hour work week yet (more like the 84), the dude is smart and has no fear of saying what many of us just think. In Outside magazine’s July 2011 issue, while promoting his new book “The 4-Hour Body,” Ferriss describes his opinion on human motivations:

It pays not to be puritanical with incentives. Just look at what’s effective. We like to talk about reward, positive thinking, positive reinforcement. But the sad or useful fact of the matter is that shame, humiliation, peer pressure, financial loss – those things are all more effective.

There are so many corollaries to infosec in this statement it’s hard to know where to begin – the flaccid ineffectiveness of security awareness, repeated insane attempts to buy our way out of proper security process and tactics, and on and on. Here, though, I want to focus on the new and exciting realm of CLOUD SECURITY. There are numerous projects underway out there that are seeking to provide some degree of provider transparency. The most well-known include the following:

There’s lots of discussion in the security community around cloud standards and “best practices” related to cloud provider practices, architecture models, and such. This will continue for some time, surely, but one of the most pressing issues has been getting CSPs to disclose how well they’re safeguarding assets and operating a security-savvy environment. To this effect, STAR is probably the most high-profile effort to date, where shiny, happy CSPs can proudly proclaim that they are awesome. I think this has some merit, but I think we need a different model. Coming back around to Ferriss’ quote, this doesn’t really address the most successful motivations we have as humans (and as organizations, by extension). I think it’s time for a “Wall of Shame” for CSPs who blatantly disregard security. How many CSPs would take security more seriously if they knew there was a provision in every contract stating that customers could publicly describe security failings at the CSP, and immediately move their data and systems elsewhere with no questions asked. I’m sure you’re saying “Yeah, right, Shack – on a cold day in hell”. OK, we’re not there, but I think we need to get away from the “chosen few” mentality of STAR, which to date, has very limited participation, and on to a more realistic model, especially for SMBs and specialized companies who need very vertical-specific SaaS offerings, for example. Do you think a small healthcare billing SaaS is going to offer themselves up for STAR? Uh, no.

While some efforts along these lines have started (the one that still have hopes for is Cloutage, although it needs a lot more community involvement), we need to thinking about this problem a little differently. No STAR listing, SSAE 16, SOC2 or 3 report, etc. will get us to a point where people know what to do and where to do business. Or in this case, where NOT to do business.