It seems to me that the AWS API *is* a good (enough) abstraction for now - it's so much richer than the competition and the experience of using it still too limited to establish a more abstract version. Provided your following a decent CD process, you'll have the tests in place to allow you to evolve the abstraction as you learn.

In my experience, abstraction shims can be useful, but they can also cost more than they're worth unless the problem domain is very well understood.

That's an interesting assumption. As matter of fact, I do which is why I chose AWS as my IaaS provider.

AWS has 3 independent Regions in the US (risk management), and it's cheaper to deploy multi-regions using the same toolsets that works for all, then spending time and efforts building your toolsets to work with other IaaS providers (cost management). In addition, combining both, you get stability and ease of management by reducing complexity of your Infrastructure as a whole.

With that said, I'm eagerly waiting for Google Compute Engine to grow into a "tree", as I've read favorable review on its limited preview. Google is probably the only powerhouse out there who can compete with Amazon head-on in terms of scalability and technologies. Thoughts?

I would put it this way (and I plan on writing in more detail on this): I think from an IaaS provider, you need the equivalent of EC2 (at least a few machine types), S3, EBS (snapshot/restore/detach/attach), and Security Groups from AWS. Then you should interact with that IaaS through an abstraction layer, not directly with their API, so you can switch to other, similar providers. (Netflix does argue that since they're at least--in some cases--using boto/similar libraries, it will be easier to add support for other clouds, but I am skeptical).

So, on that abstraction layer. Ultimately, from a software architecture standpoint, you can either start from a "I'm going to build a library that interfaces with X API", or you can start from, "I'm going to build a generalized set of interfaces to a certain type of service, and I'm going to translate from my generalized interfaces to specific implementations". I believe--and I think the evidence shows--if you start with the former, it's very hard to connect to other APIs that do similar things, just not in the exact same way. However, if you do the latter, and you are explicit from the beginning that you'll be translating from your outward-facing interfaces to other APIs, you'll have set the project up properly. These are really quite different software engineering projects, and I think that's why you hear the Netflix engineers really focus on the idea that other platforms will have to adopt the AWS API for things to work properly with their tools.

I view the multi-cloud issue as (a) critical, and (b) very hard, for the reasons you mentioned above. So much software uses RDBMS, and multi-master replication is generally speaking not the most reliable thing. Netflix has a good article about how they treat different types of data to handle their multi-region deployment, and I think we're going to have to move to architectures like that to take full advantage of the cloud.

In the end, I think the debate in these comments has more to do with us talking past each other than anything else (as Charles Babcock points out here somewhere). I think we all agree that Netflix has solved their particular use case in a great way, and that open sourcing their tools and giving speeches on their architecture is very useful. Netflix appears to view their AWS API lock-in as a necessary thing to keep for the future (and I don't see why they shouldn't work toward abstracting all of it) and Netflix appears to think that everyone approaching their tools will be capable of making an informed decision about the potential issues of using them (and I think that's a really bad assumption).

JoeHave you got any specific best practices in mind, beyond the obvious one of 'assume cheap, ephemeral hardware'? Which as significant changes on enterprise software. Followed by 'Continuous Delivery' (or deployment for some), so that you've got mechanisms for testing and replacing the components at low risk?

It seems to me that Netflix has done some sterling work in these dimensions, but that it's very unobvious to most enterprises that they are necessary underpinnings of moving to pay as you go infrastructure.

Can you be more specific in the abstractions that you have in mind?

I'd also add that Amazon does seem to be reacting to competition as price drops feel like they are accelerating, even if they are still nowhere near the cost advantage that AWS has according to James Hamilton.

AWS got an early head start in IaaS offering, and built a massive eco-system with all the building blocks that no others can follow even to this day. I just don't see why anyone would want invest the time and money on cross cloud mobility and multi-cloud tools at this stage.

To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.

Is DevOps helping organizations reduce costs and time-to-market for software releases? What's getting in the way of DevOps adoption? Find out in this InformationWeek and Interop ITX infographic on the state of DevOps in 2017.

Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.