DevOps | Cloud | Software

Menu ☰

Prologue

I met a friend at DevOps conference who used to worked for Sabre. He told me about how huge Sabre is and how it drives world’s travel businesses. I went home and researched the company and left it at that. The “Aha” moment happened when I was working on an assignment related to API management. The sheer magnitude of what Sabre was doing and what many others are doing with APIs was huge and I could make sense of it only after looking at it from API perspective. I was fascinated by the economy of APIs and possibilities it offered to businesses. In this post I briefly cover basics of API Economy, show you some cool things organizations are doing and talk a bit about API management platforms.

Real world use cases

Before we understand the drivers of why API economy is significant now and how to go about it, let’s take a look at some cool real world implementations, They will give us insights into what people are doing by opening up APIs!

Sabre a GDS company is used by more than 350K travel agents and companies worldwide. When you book tickets through one of the online portals – they are actually hitting one of GDS provider APIs. GDS companies in turn manage the tickets from multiple airlines. So tomorrow if you want to start a travel company – choose one of API providers from GDS companies and get started. Other companies in this space are Amadeus, Galileo

Netflix handles roughly 2B API requests daily by audience consuming content from various devices. Managing so many different devices, applications and consumers at this scale would not have been possible without sound APIs in place.

Evernote – the editor I used to write this post has it’s API. You must be wondering what business would you generate by exposing API of a editor to others? Co-creation is the name of the game! Developers and companies out there have created apps which have generated additional revenue for Evernote and potential acquisition targets. GTD apps like SmartTM and many others are built on top of Evernote

Ordr.in provides API to view restaurants, their menus and order food based on menu items. They give you access to data & a few bundles to get started, building actual customer facing application is done by others. One can build online food ordering apps by using ordr.in apis without having to worry about the data.

APIs for Business

So in above examples what are the organizations doing? They are using API for their business to achieve one of:

Sell/expose data/service/function to either internal consumers or external parties

Enable and scale the interaction between multiple systems and devices/device types within organization or/and with partners

Co-innovate with partners, developers and world at large.

Use a product/service in ways not imagined by organization- for example GTD app uses Evernote API

Let’s not be mistaken here: API is not or will NOT create a new business on it’s own – it’s in fact an enabler, multiplier and a channel of delivery to existing business value (data/services). The offered business value might be perceived and used in a completely different way never imagined by organization. Once you have a “value” proposition you will need to bake it into your API, but the value proposition has to be present in first place. So what can be APIzed? At a very broad level:

Data: Some form of data which might be valuable is the basic building block of an API business.

Service/Function: Some sort of service or function which is offered to internal consumers or partners can be opened to wider audience and might be utilized in ways never imagined within organization

For any business value an enterprise exposes, the level of access plays a pivotal role. Based on how open you want to be there are a few buckets of communities that you want to open API for. The real world use cases might be a mixed combination of following:

Internal business units/developers: Internal applications, various business units talking to each other, systems integrating with each other.

Partners/Community: Partners/Suppliers/Vendors/ providing you various services or parts of your product, partners co-creating the products with you, communities working with you on your platform

Public/External Developers: The world pretty much where developers can sign up your API and build applications on top of it!

The pricing model, how much access to throw in and what to expose as an API is a much more detailed discussion – and some pointers in references might be helpful there.

API Management : Why & How

Identifying the business value you offer and baking it into an API is only first step. APIs are not destination themselves instead are a journey for an enterprise. Once you have an API in place, you would like to potentially:

Manage access control on APIs, decide what kind of users get what kind of access and how much they can exhaust the system with requests. Have appropriate kill/disabling switches in place for users exhausting APIs.

Monitor the usage in terms of which APIs are used most, which are used least, what is the pattern of API usage?

What kind of devices and application are using API? Which API is working best for end users and who is making unnecessary requests and might be optimized.

If your API is open – then managing the pricing and plans, managing access based on plans, enabling new users to sign up learn and start using your API seamlessly.

Needless to say – not all of these features/functions will be required on your API journey from day one. For example if you are beginning the API journey and the only consumers to begin with are internal developers – you might need access control, well designed API and that’s about it. Once the API usage has started off and you have iterated through couple of improvements may be you will need to add versioning and monitoring. The key is to understand what you need to get started with and what you will need as you scale and build/buy as you go.

How can you build/buy API management solutions? There are quite a few players here like Mashery, Apigee, 3scale. You can find a brief overview of all players in article at ProgrammableWeb. Interestingly as the offerings converge and overlap some of iPaaS players have started offering API management as a logical extension to existing offerings like MuleSoft

API & Platform thinking

Thinking about your business as not only a product or service but as a platform is essential in today’s world. Disruptions are coming from industries least expected. Online travel and retail companies disrupting their traditional counterparts or music and camera industry being disrupted by smartphones - examples are many. Thinking of your business as a platform enables crowd-sourcing, innovation at scale and out of the box ideas to be evaluated and tested. For you to build a platform – API serve as the backbone.

Thoughts & Conclusion

APIs enable you to do business for the new age, they enable you to build platforms of your business. With more and more connected devices (Marketed as IoT – Internet of things), & complexity of applications API make perfect sense for multiple scenarios. As much APIs are important to your business, much more important is managing growth of APIs and in turn your business – that’s why a sound API Management strategy and execution makes the difference. They are doors to new growth engines and for disruption.

References and further reading

A really good article by 3Scale is a good overview of what exactly is “API Management”, what things to look for and how you to implement one (With or without 3Scale – which I really liked for their honesty and openness). The 3rd part covers the API Management while first two parts are summary of John Musser’s presentation in previous link

Customers often ask us, What is DevOps? Or how do I get started and learn more about DevOps? In this series of a few posts we are going to answer some of these questions. While a lot has been discussed on this topic already – a lot of it is fragmented and not connected. We will offer here a cohesive picture of what DevOps is. At the end of series you will have a good understanding of DevOps and some practical insights in how to go about it. This series is broken into few posts to tackle various areas as:

Getting basics right: What is DevOps anyway? (This post)

It’s all about delivering: CI & CD

Configuration Chaos : Managing your infrastructure, as code!

Monitor, measure and move: From logs to performance and beyond

Putting it all together: Collaborate, communicate and continuously improve

Further Reading

So let’s get started with understanding the very basics: What is DevOps anyway?

If you are bombarded by buzzwords and too many tools & technologies being marketed at you with tag “DevOps” – and are tired of it without really knowing what DevOps means? Then go no further than two part series by Magnus Hedemark at Redhat. The first part explains the basic premise behind DevOps, a brief history and the motive behind the DevOps movement. In second part Magnus goes a step further and offers excellent advice on practical implementation and the traps you should avoid. This is a highly recommended read for beginners and professionals alike.

One of the difficulties in getting a DevOps implementation started is who would do it? What kind of skills are involved? Should you hire someone from outside or leverage the team within organization? I am afraid there are no straightforward answers here but this excellent piece from PuppetLabs gives out some ideas which would definitely help.

In coming posts we will get into more details into various aspects of DevOps and what value we get from those. Keep watching this space.

Spring Integration days

In 2010 I worked on a project which involved Spring integration framework, the framework was introduced by a friend and colleague. Messaging as a programming model in Spring integration was a refreshing way to build applications (I highly recommend reading the linked article by Gregor Hohpe). You have to think about everything as integration & message passing between two components . Fast forward to early 2014 and working on iPaaS platform is next level of integration (iPaaS is well Recognized by Gartner and magic quadrant on iPaaS). iPaaS platform basically provides ability to connect various apps on cloud and on premise with ease. In this post we will briefly see Spring integration’s programming model and jump from there to discussing iPaaS platforms in slightly more detail.

Spring integration – message oriented programming model

In message oriented programming you build components separately and then connect them together. You of course have to agree on what the input and output of each component will be – which is the contract or the “message”. Everything you pass from one component to other is wrapped as “message”. The type of components are many (router, splitter, aggregator) and the way they talk to each other is also configurable (pub-sub, polling based). In addition you can assign more threads to a component which is heavy lifter and assign fewer threads to simpler components thus optimizing thread allocation. And of course you get benefit of being able to develop relatively independently modules and connect them. (I said relatively because in practice there are limits to how much you can really do in parallel vs. the need for collaboration). Similar programming model is followed in Apache Camel, Mulesoft etc. as well.

Cloud integration: The iPaaS problem

With adoption of cloud applications like Salesforce, Twillio, NetSuite etc. the integration now has two aspects:

Cloud apps to on premise apps integration

Cloud apps to cloud apps integration

While we have been integrating applications for long, you must be wondering what is so special about iPaaS now? Gartner classifies primary differentiation between a classic integration vs. iPaaS player as the offering over cloud, which enables among other things

On demand and pay as you go offering of iPaaS enables even SMBs to get started with integration without huge cost involved upfront.

Emergence of APIs , IoT (Internet of things) among other things – which calls for more and more integration between systems & devices which can scale.

For platforms such as Salesforce, NetSuite etc. – standard set of connectors are available. This reduces overall development time for any integration and you can build things in drag and drop fashion by configuring.

Relatively simpler approach enables business to build integration on their own.

iPaaS: the players

Gartner has listed many players in the iPaaS space. I will specifically walk you through “Dell Boomi” and “MuleSoft Cloudhub” very briefly. Head to slides for a quick overview:

What next?

The integration space itself is huge with citizen integration (integration meant for end consumers for example Gmail integration with Dropbox etc.) and enterprise integration. iPaaS platforms primarily aimed at enterprise present a powerful and yet simple way to integrate applications. As more devices and more systems start connecting to each other – more we will see the integration between them. Being able to integrate systems easily and scaling the integration meet increased traffic is essential in today’s world. iPaaS players offer this by making integration “drag and drop” simple and providing cloud scalability.

CI and CD: Introduction

I was reading through the conversation at DevOps tool-chain group. The problem posted was difficulty of using Jenkins as a delivery pipeline. Specially when it comes to complex workflows for example diamond dependencies, fan out & fan in etc. My thoughts after reading post were (Which you will see being different later):

Though you can use Jenkins to build continuous deliver pipelines, there are better suited and specialized “deployment automation” tools for the job:

DeployIT from XebiaLabs: Heard recently and started exploring, got a demo at DevOpsDays. Could not yet test drive (Dosen’t offer a trial version on website as of writing this article)

I was intrigued and started exploring the tools mentioned in the post. Should one use a CI tool like Jenkins to build delivery pipelines? Or use specialized tools mentioned earlier for continuous delivery? Or choose one good tool and use for both purposes? Are there tools which can do both well and how to choose one – these are questions we are trying to answer in this post.

Continuous Integration

Continuous integration primarily focuses on building code, running tests and ensuring the code is in “clean” state. Since latest state of software is primary purpose – older builds and changes are lesser important as you move forward.

Continuous delivery/deployment

Continuous delivery focuses on deploying a set of changes to one/more environments. The process is typically a set of steps with decision points, parallel flows and roll forward and roll back steps. Basically continuous delivery is comparatively complex workflow with some specialized actions like rolling deployments, talking to servers etc.

In contrast to a continuous integration process, in continuous delivery -

Not every build will be deployed – more often than not a deployment candidate might be sum of last few builds. (Ex. Build 450-458 forming deploy version 1.7.4 of software).

Changes done for a deployment candidate are sum of all changes done for all builds since previous deployment. Moreover a continuous delivery process has to have knowledge of which build was deployed when and where.

The level & granularity of access control needed for a CI process is vastly different than for a CD process. You might want most people to be able to kick off a build but not deploy the artifact to certain environments.

The gap!

As we can see CI and CD process use vastly varied semantics, they aim to achieve different objectives. They handle different units of work and the complexity of process are different too! So should you use one tool to do two things? Let’s try it out. We will test drive Go (From ThoughtWorks), TeamCity (from JetBrains), Bamboo (From Atlassian), Jenkins (OpenSource) and and see how do they fit in.

Parameters for using a CI tool for CD

1) Support for CI+CD Semantics: Ability to suppport dual semantics for example information from multiple builds into a release candidate

2) Workflow designer: Visual workflow – to build workflow which is easy to see and modify over time

3) Features: Standard actions available for various CD tasks - like deploy to tomcat, connect to server XYZ etc.

5) Miscellaneous factors: Can you build rolling deployments? Is it agentless or needs agents on target machines? Does tool support horizontal scalability for handling huge volumes (Which is tied to agent based model to some extent) etc.

So let’s test drive one by one!

Go- ThoughtWorks

Go almost felt like a tool which is malleable and would fit well in your organizations’s existing knowledge (Scripts, Tools etc.). It’s minimalist interface coupled with a very good visual workflow builder gives you a perfect candidate which you can use to build continuous delivery pipelines.

TeamCity- JetBrains

TeamCity – which is an excellent CI tool had to be fit in to do continuous delivery tasks. It can be a good fit for doing simpler continuous delivery pipelines if you have a wide variety of technologies- handling which is a unique strength of TeamCity.

Bamboo – Atlassian

Bamboo came out as one of best candidates for doing both Continuous Integration and continuous delivery at the same time. Product understands the two paradigms separately and handles them that way.

Jenkins – Open Source

Though it is possible to build continuous delivery pipelines in Jenkins – it felt a little unnatural. We did not explore all possibilities with many plugins available, but at the surface it looked feasible – but not natural.

Conclusion

We used continuous integration tools to also do continuous delivery – and surprisingly some tools like Go and Bamboo did a real good job of it. Does this means using specialized tools for continuous delivery is not needed? Definitely no! We used a rather simplistic scenario and did not deploy at a scale which is common in many companies today. So that conclusion would be half baked.

Nevertheless I believe that companies which already have these tools can start implementing continuous delivery pipelines – they may not be perfect but you will understand what works well. Some of your pipelines might as well be better residents within CI tools. You will also gain valuable insights into your delivery pipelines and will aid in better decision while buying the continuous delivery tool! Secondly if the budgets for CD tool is not very high – it might make sense to get started with these tools and eventually make decisions on buying CD tools.

DevOpsDays India 2013

I attended DevOpsDays India 2013 held in Bangalore. Overall program can be found here and videos should be uploaded in two weeks approximately. This is a personal account of my experience and what I enjoyed the most!

Building Community by Sidu Ponnappa

Traveling all the way from Pune, I made it just in time to Sidu’s keynote on building communities. Sidu has been founder of few startups and as part of C42 engineering they have built RubyMonk which myself and friends have found immensely helpful in getting started with Ruby.

Sidu talked about how to go about building communities (And what follows is my interpretation of his talk- on same flow). One of key points he made was we generally tend to think of community as a bunch of hackers. While this is wishful thinking, eventually hackers have to create value for customers through businesses. Community is the whole ecosystem which facilitate bartering of value. With that point in mind, when you start a community:

Keep at least one stakeholder for whom you are creating value through community. Is it people learning new language? Or a company benefiting from talent pool? Basically have end goal of community in mind.

Planning and meeting systematically is one of most important things to keep community alive and to grow over time. It will take time – but be patient and keep meeting.

Once you have a set of people who are contributing – identify and promote them. It places value on contribution and motivates more people to build value for community.

Once community has taken off, you will need to ensure:

Identify the value you are creating and people/organizations benefiting from value creation and make it a loop. The community has to continually create value for it’s stakeholders.

Marketing the community is equally important – be it face to face interactions for employers or employees, or people looking for advice or social media. Marketing will definitely make or break a community beyond a point in time.

Eventually you will have to find the next level of leadership. May be you will get busy with other projects or will get into something else – make sure you know when you are not adding value and pass on the baton to somebody else.

Communities are never perfect

Interesting Sessions

Ranjib’s talk on Attaining Resiliency was really good talking about design considerations for designing resiliency into applications. It teaches us to accept failures and build applications for them. The key takeaway for me was to build tools such that they can be eventually chained together with other tools to get bigger tasks done.

Automation is an asymptotic phenomenon

Piyush from MakeMyTrip spoke at length about centralized data/log collection. Centralized log management has been a topic of great interest to me, specially based on my experiments with LogStash, Splunk etc. The amount of data and insights you can generate specially for a travel company can be of tremendous value. You can find the presentation here and slide number 5 talks about overall architecture. Learning from case study of MakeMyTrip is great indeed.

Failures are inevitable, accept them

Folks at apigee spoke about an orchestration tool they built to create a environment of their complex stack and fully automated. This tool could be used for developers to setup their new environment, or a sales guy to create a customer specific environment from scratch. While slides are not yet available online – the concept was definitely good.

Another talk from MakeMyTrip by Konark Modi was on advanced workflow management for jobs instead of using Cron- which can get cumbersome for complex jobs. I learnt a lot from session and could see possibilities of doing things differently using workflow management tools for batch jobs. The talk focused on Celerey, but there are other tools available like Luigi and Apache Oozie.

What is DevOps BTW? : Three C’s of DevOps

Second day started with a session by Nathen Harvey from Opscode delivered over video streaming. Titled “Three C’s of DevOps”, the presentation is best explanation of what DevOps stands for that I have came across so far. I could relate to it even more because earlier evening I was part of an open space discussion on “What is DevOps?”. We could see multitude definitions and misunderstandings that people had about DevOps. Although as a group we had eventually concluded that it’s not a role or tools or a new team you invent – we got a definitive answer very next morning! I highly recommend everyone to attend a live talk from Nathen if possible and at the least go through the deck!

Epilogue

I had indeed great time interacting with DevOps community in India and learning a hell lot in short span of time. I met some very interesting people doing different things in DevOps. I met plenty of Thoughtworkers and ex-thoughtworkers who had kicked off startups or joined some interesting DevOps teams! What more, I spent few hours at airport too with attendees and had great chat on various aspects of not so much DevOps I am thankful to DevOpsDays India organizers for incredible job and I look forward to next one!

TopCoder whose slogan reads “Your solutions, born from competition” is perfect place to participate in various marathons. The categories in which you can compete is huge. To name a few areas where you can compete: Algorithms, Component Design, Software Architecture, Software development, Testing and Content creation. Tutorials, articles and demo exercises will get you started and then you can test your mettle in real battles!

CodeChef which is founded by Directi founder also employs a competition model for solving interesting problem statements where programmers all over world compete in a cut throat competition!

TeamTreeHouse focuses on practical courses such as Website development, Android development, iOS development and Ruby etc.

If you want to learn Spring, Hibernate, Maven, Struts etc from scratch with every basic concept explained well, then go no farther than JavaBrains. If you want to further dive deep into into Design Patterns, or want to learn more about Java collections etc. then Derek Banas is here to help.

Share :

Talent demand and availability of cloud platforms have led to a unique way education is offered. As long as you have a descent internet connection, no matter on which side of earth you are hiding, you can educate yourself at almost no cost or a fraction of cost of education in schools. The best part is anyone in this world can attend courses from best universities of world like Stanford, MIT and such! MIT’s opencourseware has been in existence for few years, but never imagined then what could that grow to!

While this is field is still nascent and nobody knows future for sure, but where we stand today the opportunity to learn is immense. I am posting all such sources I know, and will try to update in future too, but if you know a source, please comment and I will add it up!

If anybody has done really groundbreaking work and really pioneered the field, it is without any doubt “Khan Acadamy” founder Salman Khan. History of Khan Acadamy’s founding is inspiring and has courses mostly in science subjects and humanities.

Coursera has courses ranging from education, medicine, food and nutrition to computer science and information technology. The first time I stumbled on Coursera was when Martin Odersky conducted his MOOC on “Introduction to Scla and Functional programming”. I could not finish the course, but was hooked to concept and content. Recently signed up for course on “Intro to Algorithms” which is being conducted by Princeton University professors.

Udacity is a rather new entrant to game, but has some neat courses on mathematics and computer science. More courses should be available, as a lot of universaties in California are collaborating to offer courses online with credit for university degrees.

Udemy is a platform where you can teach and learn. People who have something to teach all over world add content and people who want to learn get to learn for free or for a fees, based on course etc.

Learnable, which is part of SitePoint network also has some great content on web development. (Disclaimer: I am a writer and curator for CloudSpring section at SitePoint.com)

Edx has courses from leading universities such as Berkeley and MIT and Harvard!

Last but not the least, almost all big universities have their Youtube channel now days where they do publish courses, classes and discussions. For example Stanford’s classes 2 go are a great resource.

My list is by no means exhaustive! Did I miss something worth mentioning?

What is exactly virtual product sampling?

Retail organizations sample their products before they launch. The sampling process basically involves having a physical sample manufactured and reviewed by product designers/creators. Product might undergo change based on look, feel, ergonomics, aesthetics and performance of product’s physical sample. The process is iterative at times and leads to lot of waste not only in material, but also in terms of delay and longer product life-cycles.

With emergence of image processing technologies and close to real life graphics, there are ways you can live without physical samples. Virtual samples look as good as real once, and most of work can be done virtually. In short more retailers are turning to virtual product sampling. Let’s understand the process and tools in depth in this area. The process is not limited to only retail products, just imagine the same thing applied to books. Instead of publishing books on paper for reviews, if virtual samples looked as good as real one, then how much paper would you would save? And this is already happening for example here. I just went ahead and created a sample account to see a virtual book looks like and quite neat IMO:

Let’s take another example and understand it better- NikeID is one such example available publicly. The idea of NikeID is that you can choose a style of shoe you want and then you can apply different materials and colors on various parts of shoe as per your choice to make your custom shoe. The shoe is then ordered specially for you and is unique to you. Now this process of applying colors and materials to different parts of a product is quite close to what a designer of a retail organization would do. Just that the his choices transform into a product which you see in store shelve and you choose one (Or multiple if you are rich haha of them. Let’s try our hands step by step:

I chose a blank shoe, for upper part, I get to choose material from three available and color from few colors available:

I get to choose roughly 9-10 parts and various materials and colors. My choices of material and colors are really few but good enough to customize for myself. Now in actual practice when designers design a shoe, they might have plenty colors and materials to choose from (Color palette and material palette). Also the number of parts they can customize might be much more broad, so they can change even tiniest of parts to differentiate the design.

After I choose almost all parts here is how a zoomed in section of my shoe looks like, the colors and materials are close to real. I can very clearly identify that the black part on which Nike logo resides is leather! That’s the beauty of it!

Technology behind the scenes

One of products which facilitates rendering of photograph like images from author files and color palette is Scene7. It can combine the combinations of color & material on a given product on the fly very well. So the possibilities are infinite in terms of what can be done. And of course that is not the only capability Scene7 has, its quite a versatile product!

These are my thoughts, have you worked in similar area and what are your thoughts would be interesting to hear.

Share :

StackOverflow and StackOverflow Careers

I am huge fan of StackOverflow. I am nowhere to a good programmer envisioned by Joel, but from the day I have found StackOverflow I have fallen in love with it. I have used in past other Q&A websites which were similar but paid and charged insane money, but the quality of answers was rather poor! The day I noticed that they have a “Careers 2.0″ or “StackOverflow careers” website, I admired the concept even more. The basic premise is like this: The StackOverflow Careers is an invitation only careers website. If you don’t have invitations you have to work your way through by either getting enough score or having enough repositories created by you on Github etc. The basic idea is that you should be good in some area and there should be a way to prove it using one of these measures! A very genuine and nice idea IMO. You show that you have done some work and you will be granted access to an exclusive club: “StackOverflow Careers” (For each system there are people who abuse it, but frankly, I don’t care about them anyway). I was under impression that I would have to do something astronomical in order to get there: considering the fact that I was new on StackOverflow and didn’t have any repo on Github etc so far! My best bet was StackOverflow score so targeting a number like 5-8K sounded a good enough way to get into it.

Somehow though I got access to StackOverflow Careers much earlier than I had expected. I don’t know exact algorithms, but being in top 20% on one of topics in terms of answers or something like that might have helped. Anyway that’s just half the story

StackOverflow Careers and Indian employers

The first thing I did when I got access to StackOverflow careers account was to check for companies in India using StackOverflow Careers and guess how many companies in India are using ? Come on guess! I found just ONE and JUST ONE: ThoughtWorks!

I am not saying that everyone has to use StackOverflow for their talent hunt, but in today’s world I think it’s a quite reliable way to find good candidates (I don’t personal experience with doing this, but I believe in the paradigm!) When I twitted about this, one of my friends asked me which big companies use it in US? Well not all of them but I found quite a few like Google, Amazon, VMWare etc! So one of possible reasons might be marketing of StackOverflow in India and Indian employers!

Anyway here is my take on this:

Traditional methods of interview are outdated in today’s world IMHO

A old school style “question answer” interview is not good enough or might give wrong judgement.

Instead the new school methods some of which are already employed by some companies are: