API as a foundation of Datacenter operations

As a member of vExpert community – I had a very unique opportunity and honor to present at vBrownBags spot during Re:Invent 2018 – the video of my presentation is available at YouTube

Below is the content of my thoughts around my quest into the new reality of Software Defined everything.

10-15 years ago System and Network admins were operating in Click-click-click mode. If the person could use CLI – it was an advanced skillset that required “Senior” added to the title.

Some of us are still leaving in that paradigm, however the world around us has changed drastically and the scale has also multiplied, the click-click or CLI solution doesn’t scale with the industry.

SDN

The first eye opener for me was SDN (software defined networking) when suddenly we didn’t have network boxes that have management and data plane in them, but instead management is done on the controller, data is transferred by highly specialized software-based appliances that can easily adopt new hardware when it becomes available. And only changes from the controller are applied to the data plane engine. Meaning everyone does it’s own job and if control plane components go down the data plane is still up and pushing the packets over the wire. Amazingly simple why we needed to wait until 2012 when Nicira become famous?

NSX movement was equally run by Network engineers that are tired of the boxes, but also by automation teams that wanted to configure the networks as applications grow. And NSX offered a complete API interface that would allow to do such programming on the fly without UI or CLI.

SOA

It’s not clear who invented the term of Services Oriented Architecture (SOA). AWS is certainly heavy on promoting it. Learning more around it and accepting it turned my world around. More on SOA can be found on Wiki and also there is an amazing talk around the topic.

The main idea is to break monolithic application into components. It’s longer to develop but easier to update individual components that are not dependent on other components. All components “talk” to each other using a universal language called API Calls.

SOA and SDN

Next evolution step that has already happened – is the perfect (or close to it) marriage between decoupled software architecture and the new age of the infrastructure deployments. So network components not only separated by function but they also talk to each other using universal language of API – not only to each other but with other vendor products functioning on the same level of the stack. Then we get something along the lines below – if one component breaks the all system is still functioning (hopefully) and the system configured to use the best source of data to keep it functioning. In the diagram here vCenter knows the best what’s happening with VMs of terms of placement and required network configuration and the network components provide the most up-to-date view of the physical network layer in terms of the Configuration and Performance.

REST API

Previously when someone around said “API calls”– the wave of protest raised in me – I’m a system admin, not a developer! Than came up to realization that even the developers are hugely benefited from API calls and SOA, it doesn’t prevent system admins, field engineers and everyone else to widely adopt API calls without too much learning.

Most APIs today support REST API which means you can use URI (meaningful part of URL) to execute the call

Rest API allows only five methods – that can easily be transferred to three categories Create (POST), Change (PUT, PATCH, Delete) and Retrieve (GET).

Users can execute REST API Calls in the browser or using command line (I.e. curl). Developers can execute the same commands using a programming language, pretty much any language.

There are opensource and commercial products that can be integrated by vendor and assist with building queries based on task, not on your knowledge of API syntaxis. The company I work for (Avi Networks) uses Swagger – here is the demo that I did for this presentation.

So now think of the biggest problem that datacenter operator has…. Multiple products, multiple consoles, multiple dashboards. If components of every single software are providing API access it means you as an operator select the best dashboard the best configuration tool and make sure that ALL your products receive the properly formatted calls.

Legacy vs Fresh start

Now let’s imagine you’re a software vendor that developed the amazing life changing product about 15 years ago. You want to integrate it with a new automation tools that use API calls – what would you do? Right – you will write an API module and bolt it in to your product. Now new generation API based automation tools, configuration utilities and dashboards can connect and do their job.

What is the problem here – you don’t use API internally between your components, so every time a new request comes – you need to add it to the list of supported API calls, test it and most importantly make sure the internal logic knows how to translate API request to a process inside your application.

Now lets say we’re starting a new product in 2018 – what would be the best approach here, based on what was mentioned above – we can architect using SoA and implement microservices in our product. What does it mean – your UI is an independent module that works with your processing engine the way as third party UI would work. Your dashboard – is no different than third party dashboard – using the same API calls. Your CLI does only one thing translates the commands you issued into API calls. Another

here shows how REST API is used by both UI and CLI to work with the product.

Are All APIs the same?

Certainly not, every developer comes with her/his way to name variables, objects, functions etc. Then the idea of integrating all those product together seems difficult, wrong or maybe impossible. Certainly it’s possible and works very well even in very large enterprise deployments – Grafana and Graphite are examples of very well functioning monitoring tools. We’re still in the beginning. And no surprise Martin Casado father of VMware NSX/Nicira comes again with a product that allows integration of different products, systems, components that support API – API Gateways. I’m sure there are plenty of API Gateway products on the market today

Nirvana

Would it be amazing if your product only uses API between components – universal language that is opened for others and the end-user wins as they get the best products from different vendors working as the same system. And most importantly automation – at the end we can build self-monitoring, self-healing and self-improving systems. It’s another topic of Machine learning, but the language of communicating between ML and the systems that do the job already exist we just need to turn it to our DNA.

It sounds like a dream, and it’s well-known that we have tons of work before we all leave in this imaginary world of nirvana. However, in this post here I’m trying to make my contribution to this movement of happiness.

About Petr McAllister

Petr is 20+ years experienced IT infrastructure professional during his career working internationally in different countries, continents and industry verticals.

For an extended period of his career, Petr focused on Network and Security aspects of infrastructure. During his 5 years at VMware Petr had the opportunity to participate in large projects as a solution architect,
designer, and system engineer. VMware NSX becomes the specialization for Petr in 2016.