The pace of innovation and new product development at Amazon Web Services (AWS) is often cited by analysts as being one of the cloud giant’s biggest sources of competitive difference.

According to data from 451 Research, published ahead of this year’s AWS Re:Invent partner and customer conference in Las Vegas, the firm’s cloud portfolio of products and services contains more than 320,000 SKUs, and 16% of these were created within the first two weeks of November 2017 alone.

With such a fast-paced product release rate, how does AWS ensure the security concerns of cloud users remain front of mind and are accounted for at every step of development for every product it brings to market?

AWS chief information security officer (CISO), Stephen Schmidt, says the cloud giant achieves this feat by, essentially, reversing the product development process so that the first thing its teams work on is a one-page press release that clearly and concisely explains what this hypothetical service will do.

The process has nothing to do with getting a head-start on the marketing push for whatever this embryonic service or technology might be, and is designed instead to focus the minds of product development team on the job at hand, Schmidt adds.

“The point is, if you cannot – as a service owner – distil down the essence of what you’re going to build down to one page in English comprehensible to the average person, you’re not thinking clearly enough about what you’re building,” he says.

This one-page document also needs to be created in tandem with an accompanying set of security-focused frequently asked questions (FAQs) to pre-empt any queries users may have about how their data will be secured and their privacy protected when using the service.

This, in turn, is to ensure the security considerations of AWS customers are factored into the design of the product from the “very first document”, says Schmidt.

Rationalising release dates

Paul Misener, vice-president of innovation at AWS’s parent company, Amazon.com, says creating a press release for a non-existent product holds the team accountable for ensuring it eventually does make it to market.

“It [also] acts as a backward-looking roadmap, because we know what we want to do for our customers, and we work backwards from that point in the future to today [as] we figure out how that is done,” Misener told attendees at the AWS Transformation Day in London at the end of October 2017.

“You’re writing it in customer language, not in technical jargon or internal speak, [and] you’re telling your customer what’s in it for them, so they can understand it. It’s [then] up to you to figure out all the things in between that [you need to do] to get there.”

At the time of its creation, it is impossible for this press release to be distributed, Misener adds, because the technology featured in it does not exist, which puts the onus on the team behind it to get inventive.

“This [is an] acknowledgement that to get from here to the point when the press release goes out, you have to invent stuff and it can’t be done today. It’s a very humbling thing, but also very empowering because you recognise that you cannot do this. Your organisation cannot do this today, but we need to be able to do sometime in the future,” he says.

Two pizza teams

The size and make-up of the team initially tasked with shaping the press release is guided by the two pizza team rule, which is a concept dreamed up by Amazon CEO Jeff Bezos.

“The idea is they are a dedicated team, not necessarily from the same organisational silo, who are brought together from throughout the company to address an opportunity or challenge, but the number of individuals involved should be a number comfortably fed by two, extra-large, American-style pizzas,” says Misener.

The idea prevents organisations from getting too prescriptive about how big a product development team needs to, says Misener. This, in turn, can lead to attitudes developing that are at complete odds with innovation, as members begin to fixate on having a set number of people in the team at all times.

“[Say] eight people serve on the team to address opportunities and challenges, and – after a couple of months – they discover they need an expert in artificial intelligence, and there is a woman down the hall who is available and has the expertise who is happy to join the team? That would make nine people, we can’t invite her,” he says, as an example.

Following the two-pizza team rule also prevents teams from getting too large and unwieldy, he adds. When this happens, there is a risk of sub-teams developing, and complicating communications between the individual members.

Securing the product development checkpoints

Once the product design team is assembled and the press release done, the product development part of the process can begin in earnest. At various junctures, there will be checkpoints to test the service’s resiliency, durability and its security posture too, says Schmidt.

The security checkpoint sees a set of engineers brought into to test and try out various characteristics, pertaining to the service’s access management and encryption levels, while ensuring the data passing through the service is protected as every stage.

Once that is taken care of, Schmidt says the “fun part” of stress-testing the product’s security posture can begin: the application security review.

“This is a formal application security review process, focused on making sure all the common errors that people make in software development have been addressed, and also all the testing that we require has been accomplished,” he says.

This process involves testing for both positive and negative functionality to ensure the product does all the things it is supposed to do, with Amazon drawing on both black and white box methodologies to conduct these reviews.

“White box is where the test team has access to the source code and design documents for the service, so they have the best shot at breaking something possible,” says Schmidt.

“The black box [method] is testing how someone on the outside would see the service and would try to break it. I would say, if there is a single best job in the company, it is that one because you get paid to try to break things for a living.

“It’s a huge amount of fun because we have to have the same mentality and behaviour profiles that attackers do, and we have to think very critically about our services, and how are we going to cause mayhem if there is an opportunity for a bad guy out there,” says Schmidt.

Testing out its services in this way is something that is constantly repeated during the build process for quality assurance purposes, he says.

“It’s something we do in the beginning of the development process, when we have the first several lines of code, all the way down to when it’s shipped and we re-test our services over time,” he adds.

From the inside out

Many of the cloud services and products AWS brings to market come equipped with security tools and functionality that were originally created for in-house use only, before being unleashed on the wider world, says Schmidt.

“When we have discussions with our customers about how we approach certain kinds of security problems, they’ll often ask how do we [AWS] do this internally,” he says.

While many of its customers are responsible for running “hundreds to thousands” of machines, AWS has an estate that has a “lot more zeros than that”, says Schmidt, and standard, off-the-shelf security tooling does not always cut the mustard there.

“You can’t use common security tools in that environment, so we have to build security tools that will scale very broadly and horizontally. As a result, we often get pushed to take those tools we build for internal use and externalise them,” he says.

As an example of such systems, Schmidt points to the work the company has done in-house around machine learning product development, and how this has informed the creation of its recently announced service, AWS Macie.

The service uses machine learning to uncover, sort and classify data stored in AWS Simple Storage Service (S3), and is trained to find personally identifiable information or corporate intellectual property that may need additional protections afforded to it, for example.

Users are also able to trace how their data is accessed or moved through a dashboard interface, while Macie monitors the information usage patterns continually for evidence of authorised usage or data processing.

To work, users have to provide Macie with access to the contents of their S3 buckets and usage logs, and – over the course of two weeks – will use this data to begin establishing a baseline for normal usage.

“It categorises the contents, and understands what is there, and starts to say, ‘I know what’s here, I know what is sensitive, and I know how people are using that’, and when it gets two weeks of, ‘Here’s how people are using that’, it can start to make judgements and [indicate] things such as, ‘This use is not like all of the others’,” says Schmidt.

Where the system comes into its own is by removing a lot of the “heavy lifting” and repetitive work users typically have to do when sifting through information pertaining to how corporate data stored in the cloud is accessed and used.

Value of human judgement

But, Schmidt says, the intention is not to eradicate the role of humans from this process entirely, because “the human being is the single most valuable asset any security team has”.

After all, humans are the only ones equipped to make “reasoned judgements” about whether or not something that looks suspicious poses a credible risk to the company at large.

“What you want to do is build tools and deploy it so the humans are doing the judgement work, instead of having to do the same heavy lifting every day. Where machine learning comes into play is that it focuses those humans on the things that really matter and discards those things that don’t,” he says.

Macie does this by flagging to users the presence of datasets that may contain personally identifiable information, and providing lists of all the people who can access it and how they may have previously used it in the past, for example.

“[Macie] can say this one set of accesses is anomalous and looks different to all of the others, so then the human can say, ‘Why does this look different? Is it because this person is in a different job family, or doing something unexpected?’,” says Schmidt.

“It could be a sales person downloading a lot of source code, or a software developer downloading a lot of financial data about the company. That is the sort of thing Macie will alert on.”

As well as allowing customers to make use of its own in-house tools and technologies, Schmidt says, its never-ending quest to keep its cloud secure also benefits the internet as a whole.

“The security improvements we put in place at the behest of our largest customers, [also] benefits the student who is doing their university homework using our free tier because they get exactly the same security capabilities. And that democratisation of security is a really powerful thing for the internet as a whole,” he says.

“It’s the ‘a rising tide lifts all boats higher’ kind of thing, but [what] a lot of SMEs struggle with is they don’t have the investment to make in security that large companies do. By automating and mechanising security to meet the most stringent barriers, it means smaller organisations can take advantage of the same things, which makes the internet a better place for all of us.”