Archive

The premise of my Commode Computing presentation was to reinforce that we desperately require automation in all aspects of “security” and should work toward leveraging APIs in stacks and products to enable not only control but also audit and compliance across physical and virtualized solutions.

There are numerous efforts underway that underscore both this need and the industry’s response to such. Platform providers (virtualization and cloud) are leading this charge given that much of their stacks rely upon automation to function and the ecosystem of third party solutions which provide value are following suit, also.

Most of the work exists around ensuring that the latest virtualized versions of products/solutions are API-enabled while the CLI/GUI-focused configuration of older products rely in many cases still on legacy management consoles or intermediary automation and orchestration “middlemen” to automate.

Here’s a great example of how one might utilize (Perl) scripting and RESTful APIs against VMware’s vShield Edge solution to provision, orchestrate and even audit firewall policies using their API. It’s a fantastic write-up from Richard Park of SourceFire (h/t to Davi Ottenheimer for the pointer):

Here is an overview of how to use perl code to work with VMware’s vShield API.

vShield App and Edge are two security products offered by VMware. vShield Edge has a broad range of functionality such as firewall, VPN, load balancing, NAT, and DHCP. vShield App is a NIC-level firewall for virtual machines.

We’ll focus today on how to use the API to programatically make firewall rule changes. Here are some of the things you can do with the API:

Whilst at CloudConnect I filmed some comments with Intel, RSA, Terremark and HyTrust on Intel’s Trusted Execution Technology (TXT) and its implications in the Cloud Computing space specific to “trusted cloud” and using the underlying TPM present in many of today’s compute platforms.

The 30 minute session got cut down into more consumable sound bites, but combined with the other speakers, it does a good job setting the stage for more discussions regarding this important technology.

* Update: A few hours after writing this last night, AWS announced they had achieved Level 1 PCI DSS Compliance.* If you pay attention to how the announcement is worded, you’ll find a reasonable treatment of what PCI compliance means to an IaaS cloud provider – it’s actually the first time I’ve seen this honestly described:

Merchants and other service providers can now run their applications on AWS PCI-compliant technology infrastructure to store, process and transmit credit card information in the cloud. Customers can use AWS cloud infrastructure, which has been validated at the highest level (Level 1) of PCI compliance, to build their cardholder environment and achieve PCI certification for their applications.

Note how they phrased this, then read my original post below.

However, pay no attention to the fact that they chose to make this announcement on Pearl Harbor Day 😉

—

Here’s the thing…

A cloud provider can achieve compliance (such as PCI — yes v2.0 even) such that the in-scope elements of that provider which are audited and assessed can ultimately contribute to the compliance of a customer operating atop that environment. We’ve seen a number of providers assert compliance across many fronts, but they marketed their way into a yellow card by over-reaching…

It should be clear already, but for a service to be considered compliant, it clearly means that the customer’s in-scope elements running atop a cloud provider must also undergo and achieve compliance.

That means compliance is elementally additive the same way “security” is when someone else has direct operational control over elements in the stack you don’t.

In the case of an IaaS cloud provider who may achieve compliance from the “concrete to the hypervisor,” (let’s use PCI again,) the customer in turn must have the contents of the virtual machine (OS, Applications, operations, controls, etc.) independently assessed and meet PCI compliance in order that the entire stack of in-scope elements can be described as compliant.

Thus security — and more specifically compliance — in IaaS (and PaaS) is a shared responsibility.

I’ve spent many a blog battling marketing dragons from cloud providers that assert or imply that by only using said provider’s network which has undergone and passed one or more audits against a compliance framework, that any of its customers magically inherit certification by default. I trust this is recognized as completely false.

As compliance frameworks catch up to the unique use-cases that multi-tenancy and technologies such as virtualization bring, we’ll see more “compliant cloud” offerings spring up, easing customer pain related to the underlying moving parts. This is, for example, what FedRAMP is aiming to provide with “pre-approved” cloud offerings. We’ve got visibility and transparency issues to solve , as well as temporal issues such as the frequency and period of compliance audits, but there’s progress.

We’re going to see more and more of this as infrastructure- and platform-as-a-service vendors look to mutually accelerate compliance to achieve that which software-as-a-service can more organically deliver as a function of stack control.

/Hoff

* Note: It’s still a little unclear to me how some of the PCI requirements are met in an environment like an IaaS Cloud provider where “applications” that we typically think of that traffic in PCI in-scope data don’t exist (but the infrastructure does,) but I would assume that AWS leverages other certifications such as SAS and ISO as a cumulative to petition the QSA for consideration during certification. I’ll ask this question of AWS and see what I get back.

I’m grumpy, confused and scared. Classic signs of shock. I can only describe what I’m feeling by virtue of an analog…

There’s a scene in the movie Jaws where Chief Brody, chumming with fish guts to attract and kill the giant shark from the back of the boat called “The Orca,” meets said fish for the first time. Terrified by it’s menacing size, he informs [Captain] Quint “You’re gonna need a bigger boat.”

I felt like that today as I read through the recently released draft of the long-anticipated FedRAMP documents. I saw the menace briefly surface, grin at me, and silently slip back into the deep. Sadly, channeling Brody, I whispered to myself “…we’re gonna need something much sturdier to land this fish we call cloud.”

I’m not going to make any friends with this blog.

I can barely get my arms around all of the issues I have. There will be sequels, just like with Jaws, though unlike Roy Schneider, I will continue to be as handsome as ever.

Here’s what I do know…it’s 81 pages long and despite my unhappiness with the content and organization, per Vivek Kundra’s introduction, I can say that it will certainly “encourage robust debate on the best path forward.” Be careful what you ask for, you might just get it…

What I expected isn’t what was delivered in this document. Perhaps in the back of my mind it’s exactly what I expected, it’s just not what I wanted.

This is clearly a workstream product crafted by committee and watered down in the process. Unlike the shark in Jaws, it’s missing it’s teeth, but it’s just as frightening because its heft is scary enough. Even though all I can see is the dorsal fin cresting the water’s surface, it’s enough to make me run for the shore.

As I read though the draft, I was struck by a wave of overwhelming disappointment. This reads like nothing more than a document which scrapes together other existing legacy risk assessment, vulnerability management, monitoring and reporting frameworks and loosely defines interactions between various parties to arrive at a certification which I find hard to believe isn’t simply a way for audit companies to make more money and service providers to get rubber-stamped service ATO’s without much in the way of improved security or compliance.

This isn’t bettering security, compliance, governance or being innovative. It’s not solving problems at a mass scale through automation or using new and better-suited mousetraps to do it. It’s gluing stuff we already have together in an attempt to make people feel better about a hugely disruptive technical, cultural, economic and organizational shift. This isn’t Gov2.0 at all. It’s Gov1.0 with a patch. It’s certainly not Cloud.

Besides the Center for Internet Security reference, there’s no mention of frameworks, tools, or organizations outside of government at all…that explains the myopic focus of “what we have” versus “what we need.”

The document is organized into three chapters:

Chapter 1: Cloud Computing Security Requirement Baseline
This chapter presents a list of baseline security controls for Low and Moderate
impact Cloud systems. NIST Special Publication 800-53R3 provided the foundation
for the development of these security controls.

Chapter 2: Continuous Monitoring
This chapter describes the process under which authorized cloud computing systems
will be monitored. This section defines continuous monitoring deliverables,
reporting frequency and responsibility for cloud service provider compliance with
FISMA.

Chapter 3: Potential Assessment & Authorization Approach
This chapter describes the proposed operational approach for A&A’s for cloud
computing systems. This reflects upon all aspects of an authorization (including
sponsorship, leveraging, maintenance and continuous monitoring), a joint
authorization process, and roles and responsibilities for Federal agencies and Cloud
Service Providers in accordance with the Risk Management Framework detailed in
NIST Special Publication 800-37R1.

It’s clear that the document was written almost exclusively from the perspective of farming out services to Public cloud providers capable of meeting FIPS 199 Low/Moderate requirements. It appears to be written in the beginning from the perspective of SaaS services and the scoping and definition of cloud isn’t framed — so it’s really difficult to understand what sort of ‘cloud’ services are in scope. NIST’s own cloud models aren’t presented. Beyond Public SaaS services, it’s hard to understand whether Private, Hybrid, and Community clouds — PaaS or IaaS — were considered.

It’s like reading an article in Wired about the Administration’s love affair with Google while the realities of security and compliance are cloudwashed over.

I found the additional requirements and guidance related to the NIST 800-53-aligned control objectives to be hit or miss and some of them utterly laughable (such as SC-7 – Boundary Protection: “Requirement: The service provider and service consumer ensure that federal information (other than unrestricted information) being transmitted from federal government entities to external entities using information systems providing cloud services is inspected by TIC processes.” Good luck with that. Sections on backup are equally funny.

The “Continuous Monitoring” section requirements wherein the deliverable frequency and responsibile party is laid out engenders a response from “The Princess Bride:”

You keep using that word (continuous)…I do not think it means what you think it means…

Only 2 of the 14 categories are those which FedRAMP is required to provide (pentesting and IV&V of controls.) All others are the responsibility of the provider.

Sigh.

There’s also not a clear distinction that in a service deployed on IaaS (as an example) where anything in the workload’s VM fits into this scheme (you know…all the really important stuff like information and applications) and how agency processes intersect with the CSP, FedRAMP and the JAB.

The very dynamism and agility of cloud are swept under the rug, especially in sections discussing change control. It’s almost laughable…code changes in some “cloud” SaaS vendors every few hours. The rigid and obtuse classification of the severity of changes is absolutely ludicrous.

I’m unclear if the folks responsible for some of this document have ever used cloud based services, frankly.

“Is there anything good in the document,” you might ask? Yes, yes there is. Firstly, it exists and frames the topic for discussion. We’ll go from there.

However, I’m at a loss as how to deliver useful and meaningful commentary back to this team using the methodology they’ve constructed…there’s just so much wrong here.

I’ll do my best to hook up with folks at the NIST Cloud Workshop tomorrow and try, however if I smell anything remotely like seafood, I’m outa there.

Here’s the biggest challenge I see in Cloud deployment as the topic of security inevitably occurs in conversation:

There’s too much of it.

Huh?

More specifically, much like my points regarding networking in highly-virtualized multi-tenant environments — it’s everywhere — we’ve got the same problem with security. Security is shot-gunned across the cloud landscape in a haphazard fashion…and the buck (pun intended) most definitely does not stop here.

The reality is that if you’re using IaaS, the lines of demarcation for the responsibility surrounding security may in one take seemed blurred but are in fact extremely well-delineated, and that’s the problem. I’ve seen quite a few validated design documents outlining how to deploy “secure multi-tentant virtualized environments.” One of them is 800 pages long.

Check out the diagram below.

I quickly mocked up an IaaS stack wherein you have the Cloud provider supplying, operating, managing and securing the underlying cloud hardware and software layers whilst the applications and information (contained within VM boundaries) are maintained by the consumer of these services. The list of controls isn’t complete, but it gives you a rough idea of what gets focused on. Do you see some interesting overlaps? How about gaps?

This is the issue; each one of those layers has security controls in it. There is lots of duplication and there is lots of opportunity for things to be obscured or simply not accounted for at each layer.

Each of these layers and functional solutions is generally managed by different groups of people. Each of them is generally managed by different methods and mechanisms. In the case of IaaS, none of the controls at the hardware and software layers generally intercommunicate and given the abstraction provided as part of the service offering, all those security functions are made invisible to the things running in the VMs.

A practical issue is that the FW, VPN, IPS and LB functions at the hardware layer are completely separate from the FW, VPN, IPS and LB functions at the software layer which are in turn completely separate from the FW, VPN, IPS and LB functions which might be built into the VM’s (or virtual appliances) which sit stop them.

The security in the hardware is isolated from the security in the software which is isolated from the security in the workload. You can, today, quite literally install the same capabilities up and down the stack without ever meeting in the middle.

That’s not only wasteful in terms of resources but incredibly prone to error in both construction, management and implementation (since at the core it’s all software, and software has defects.)

Keep in mind that at the provider level the majority of these security controls are focused on protecting the infrastructure, NOT the stuff atop it. By design, these systems are blind to the workloads running atop them (which are often encrypted both at rest and in transit.) In many cases this is why a provider may not be able to detect an “attack” beyond data such as flows/traffic.

To make things more interesting, in some cases the layer responsible for all that abstraction is now the most significant layer involved in securing the system as a whole and the fundamental security elements associated with the trust model we rely upon.

The hypervisor is an enormous liability; there’s no defense in depth when your primary security controls are provided by the (*ahem*) operating system provider. How does one provide a compensating control when visibility/transparency [detective] are limited by design and there’s no easy way to provide preventative controls aside from the hooks the thing you’re trying to secure grants access to?

“Trust me” ain’t an appropriate answer. We need better visibility and capabilities to robustly address this issue. Unfortunately, there’s no standard for security ecosystem interoperability from a management, provisioning, orchestration or monitoring perspective even within a single stack layer. There certainly isn’t across them.

In the case of Cloud providers who use commodity hardware with big, flat networks with little or no context for anything other than the flows/IP mappings running over them (thus the hardware layer is portrayed as truly commoditized,) how much better/worse do you think the overall security posture is of a consumer’s workload running atop this stack. No, that’s not a rhetorical question. I think the case could be argued either side of the line in the sand given the points I’ve made above.

This is the big suck. Cloud security suffers from the exact same siloed security telemetry problems as legacy operational models…except now it does it at scale. This is why I’ve always made the case that one can’t “secure the Cloud” — at least not holistically — given this lego brick problem. Everyone wants to make the claim that they’re technology is that which will be the first to solve this problem. It ain’t going to happen. Not with the IaaS (or even PaaS) model, it won’t.

However, there is a big opportunity to move forward here. How? I’ll give you a hint. It exists toward the left side of the diagram.

There are a ton of venues I haven’t added here because they are directly related to customer visits that may not wish to be disclosed. You can see the prior list of speaking engagements listed here.

[I often get a bunch of guff as to why I make these lists: ego, horn-tooting, self-aggrandizement. I wish I thought I were that important. The real reason is that it helps me keep track of useful stuff focused not only on my participation, but that of the rest of the blogosphere. It also allows folks to plan meet-ups]

You’ll forgive my impertinence, but the last time I saw a similar claim of a PCI compliant Cloud offering, it turned out rather anti-climatically for RackSpace/Mosso, so I just want to make sure I understand what is really being said. I may be mixing things up in asking my questions, so hopefully someone can shed some light.

“…Verizon’s On-Demand Cloud Computing Solution First to Achieve PCI Compliance” and the company’s cloud computing solution called Computing as a Service (CaaS) which is “…delivered from Verizon cloud centers in the U.S. and Europe, is the first cloud-based solution to successfully complete the Payment Card Industry Data Security Standard (PCI DSS) audit for storing, processing and transmitting credit card information.”

It’s unclear to me (at least) what’s considered in scope and what level/type of PCI certification we’re talking about here since it doesn’t appear that the underlying offering itself is merchant or transactional in nature, but rather Verizon is operating as a service provider that stores, processes, and transmits cardholder data on behalf of another entity.

Here’s what the article says about what Verizon undertook for DSS validation:

To become PCI DSS-validated, Verizon CaaS underwent a comprehensive third-party examination of its policies, procedures and technical systems, as well as an on-site assessment and systemwide vulnerability scan.

I’m interested in the underlying mechanicals of the CaaS offering. Specifically, it would appear that the platform – compute, network, and storage — are virtualized. What is unclear is if the [physical] resources allocated to a customer are dedicated or shared (multi-tenant,) regardless of virtualization.

According to this article in The Register (dated 2009,) the infrastructure is composed like this:

The CaaS offering from Verizon takes x64 server from Hewlett-Packard and slaps VMware’s ESX Server hypervisor and Red Hat Enterprise Linux instances atop it, allowing customers to set up and manage virtualized RHEL partitions and their applications. Based on the customer portal screen shots, the CaaS service also supports Microsoft’s Windows Server 2003 operating system.

Some details emerge from the Verizon website that describes the environment more:

Every virtual farm comes securely bundled with a virtual load balancer, a virtual firewall, and defined network space. Once the farm is designed, built, and named – all in a matter of minutes through the CaaS Customer Management Portal – you can then choose whether you want to manage the servers in-house or have us manage them for you.

If the customer chooses to manage the “servers…in-house (sic)” is the customer’s network, staff and practices now in-scope as part of Verizon’s CaaS validation? Where does the line start/stop?

I’m very interested in the virtual load balancer (Zeus ZXTM perhaps?) and the virtual firewall (vShield? Altor? Reflex? VMsafe-API enabled Virtual Appliance?) What about other controls (preventitive or detective such as IDS, IPS, AV, etc.)

The reason for my interest is how, if these resources are indeed shared, they are partitioned/configured and kept isolated especially in light of the fact that:

Customers have the flexibility to connect to their CaaS environment through our global IP backbone or by leveraging the Verizon Private IP network (our Layer 3 MPLS VPN) for secure communication with mission critical and back office systems.

It’s clear that Verizon has no dominion over what’s contained in the VM’s atop the hypervisor, but what about the network to which these virtualized compute resources are connected?

So for me, all this all comes down to scope. I’m trying to figure out what is actually included in this certification, what components in the stack were audited and how. It’s not clear I’m going to get answers, but I thought I’d ask any way.

Oh, by the way, transparency and auditability would be swell for an environment such as this. How about CloudAudit? We even have a PCI DSS CompliancePack 😉

Question for my QSA peeps: Are service providers required to also adhere to sections like 6.6 (WAF/Binary analysis) of their offerings even if they are not acting as a merchant?

I took this as a challenge in 5 minutes or less to articulate this in succinct, bulleted form. I timed it. 4 minutes & 48 seconds. Loaded with snark and Hoffacino-fueled dogma.

Here goes:

Get an Amazon Web Services [or Rackspace or Terremark vCloud Express, etc.] account, instantiate a couple of instances as though you were deploying a web-based application with sensitive information that requires resilience, security, survivability and monitoring. If you have never done this and you’re in security spouting off about the insecurities of Cloud, STFU and don’t proceed to step 2 until you do. These offerings put much of the burden on you to understand what needs to be done to secure Cloud-based services (OS, Apps, Data) which is why I focus on it. It’s also accessible and available to everyone.
–

Take some time to be able to intelligently understand that as abstracted as much of Cloud is in terms of the lack of exposed operational moving parts, you still need to grok architecture holistically in order to be able to secure it — and the things that matter most within it. Building survivable systems, deploying securable (and as secure as you can make it) code, focusing on protecting information and ensuring you understand system design and The Three R’s (Resistance, Recognition, Recovery) is pretty darned important. That means you have to understand how the Cloud provider actually works so when they don’t you’ll already have planned around that…
–

Employ a well-developed risk assessment/management framework and perform threat modeling. See OCTAVE, STRIDE/DREAD, FAIR. Understanding whether an application or datum is OK to move to “the Cloud” isn’t nuanced. It’s a simple application of basic, straightforward and prudent risk management. If you’re not doing that now, Cloud is the least of your problems. As I’ve said in the past “if your security sucks now, you’ll be pleasantly surprised by the lack of change when you move to Cloud.”
–

Proceed to the Cloud Security Alliance website and download the guidance. Read it. Join one or more of the working groups and participate to make Cloud Security better in any way you believe you have the capacity to do so. If you just crow about how “more secure” the Cloud is or how “horribly insecure by definition” it is, it’s clear you’ve not done steps 1-3. Skip 1-3, go to #5 and then return to #1.
–

Use common sense. There ain’t no patch for stupid. Most of us inherently understand that this is a marathon and not a sprint. If you take steps 1-4 seriously you’re going to be able to logically have discussions and make decisions about what deployment models and providers suit your needs. Not everything will move to the Cloud (public, private or otherwise) but a lot of it can and should. Being able to layout a reasonable timeline is what moves the needle. Being an idealog on either side of the tarpit does nobody any good. Arguing is for Twitter, doing is for people who matter.

Cloud is only rocket science if you’re NASA and using the Cloud for rocket science. Else, for the rest of us, it’s an awesome platform upon which we leverage various opportunities to improve the way in which we think about and implement the practices and technology needed to secure the things that matter most to us.