Global High-Tech Innovation

April 2014

April 28, 2014

Note: Schedule changes occuring just before EMC World are listed below........

Social media enthusiasts attending EMC World next week will have a choice of interesting speaker sessions to attend at the Social Lounge in EMC World Village. Guest speakers will kick off each session by sharing how they got started in social media, and then dialogue with the audience on a variety of topics ranging from social media to high-tech futures.

EMC Chief Technology Officer John Roese was one of the most widely-read Executive bloggers in the high-tech industry while CTO of Nortel. I will host a 30 minute session with John at the Social Lounge from 2-2:30 PM on Tuesday.

VMware blogger Chuck Hollis got his start in social media in October of 2006. EMC Fellow Steve Todd followed Chuck's lead and started the Information Playground in January of 2008. Steve and Chuck will be interviewed by legendary blogger StorageZilla from 10-11 9-10 AM on Wednesday.

For those interested in the sessions I've put together a table listing the exact dates and times for each session.

April 23, 2014

I recently finished writing a series of articles discussing next-generation applications and the new models of IT infrastructure that are often required to support them. My focus was primarily on the building blocks used in the new infrastructure.

I’d like to turn my attention now to the application side of the discussion. I’ve mentioned several times that one of the key value propositions of Pivotal is the reduction of software development time, enabling companies to deliver better software experiences to their customers, at a pace faster than their competitors.

Pivotal is trying to reduce application development time from months down to days (and even hours). This model is akin to what VMware did with driving down IT provisioning times (it’s no coincidence that Pivotal as a company was created via a significant collection of VMware software assets).

Let’s start with the following use case:

A corporation has an idea for a business app that could drive significant revenue if it crosses over the 300,000 concurrent user limit within 2-3 months.

As the app scales it needs to allow users to use social messaging via an eventually consistent model.

Any downtime and/or quality issues would be the kiss of death in user adoption.

Scaling to 1,000,000 concurrent users is the plan.

In other words, the corporation wants the app to be wildly successful in a short amount of time and therefore it needs a scalable, high quality messaging framework that knows how to start small and go big fast.

If you look around the industry, there’s really only one choice: Pivotal RabbitMQ. RabbitMQ is used by many of the 'consumer grade' companies that operate in the manner to which many enterprises aspire. Consider the list of users below as a proof point.

If you’re Googling, Facebooking, Tivoing, reading the news, booking travel or doing pretty much anything else online, chances are you’re using a system that involves RabbitMQ. In fact, it’s hard not to spend a day online without using a service that depends on it.

RabbitMQ is also part of one of the biggest, active IT projects in the world: UIDAI – the Unique Identification Authority of India (first described by VMware here). This project, which is being implemented by Aadhar, currently has 1.3 Billion end users! So if you want to dress your application for success at scale, RabbitMQ is a proven product.

The goal of the UIDAI application is to enable social services and microfinance for over 1+ billion residents. It’s the biggest online identity project in the world, and RabbitMQ is part of a very significant, country-wide IT infrastructure with a lot of moving parts. Whenever moving parts require coordination, RabbitMQ is the reliable messaging framework to deliver it.

The Indian government has a strategy overview and several other interesting documents describing the initiative here and here.

If a new enterprise application does experience wild success, the choice of RabbitMQ as the messaging framework will free the application to scale dramatically. As I’ve started to study the reasons for RabbitMQ’s success, I’ve learned a number of key insights from Helena Edelson’s blog (with a portion cut-and-pasted below):

It’s wicked fast and reliable

Supports security and transactions, persistence, client acks, and flow control

It is open source with support services and several very active user lists

Hugely interoperable

RabbitMQ brokers run on many platforms and offer several clients (Java, .net and others), all which speak AMQP protocol to each other. Anyone can write a RabbitMQ client in any language: Ruby, Java, Spring, Python, C, C#, Perl, Erlang, Lisp, Haskell, PHP, _your_language_here.

Easily clusterable

Works with several protocols: AMQP, STOMP, XMPP, 0MQ

Highly-flexible topologies

Can work with Pacemaker and associated tools to provide High Availability

RabbitMQ command line tool and plugin for real-time activity and management

Messaging services hosted in the cloud with RabbitMQ as part of overall cloud infrastructure services are highly-supported

Going forward you will witness RabbitMQ messaging and the Spring framework driving the Industrial Internet and Internet of Things (IoT), serving as bridge from sensors to both existing and new applications:

What makes this possible is that RabbitMQ supports a range of messaging protocols, including MQTT, the emerging standard to send messages to and from sensors and mobile devices; JMS, used by many legacy Java apps built in the 90s and 00s; and AMQP, developed several years ago and increasingly used for server-to-server messaging.

I wouldn’t want to leave the impression that RabbitMQ and Spring only run in a “3rd platform” context. The truth is that their ability to run on 1st, 2nd, 3rd, and Industrial Internet platforms is a key reason for choosing them. They are the bridge that facilitates interoperability and/or migration between different infrastructures. It’s imperative to build application interoperability at the topmost layer as well as application data mobility at the bottom-most layers. As the diagram below shows, RabbitMQ is accessible by languages spanning four decades of the IT industry, from some of the oldest (Cobol) to some of the newest (Go). As well some of the most popular, like Java, C#, C, Ruby, and Python.

As an example, you can have a Cobol application streaming messages into an Android or iPhone app using websockets. Very few products allow developers to interoperate between IT platforms from different eras.

In upcoming posts I will take a deeper look at how Pivotal’s RabbitMQ and Spring are driving new applications in for the Industrial Internet. In the meantime, here are some additional examples of scale:

RabbitMQ Scales to 74,000 Continuous Integration Builds a Day with TravisCI.

See how C24 was able to move from 20K to 1 MM complex event processed per second

In order to inspire large (or potentially small) groups of people to work with you on your strategy, there are three key guarantees you must offer them:

Purpose

Autonomy

Mastery

These guarantees were shared by Daniel Pink at a recent EMC leadership meeting, and in previous posts I have already reflected on the first two items. I shared my personal story of trying to build a global innovation measurement framework. It was a big task and there was no way that I could carry it out on my own. I needed at least one person from nearly every country where EMC innovators and researchers were located.

So I started my recruitment pitch by appealing to a higher purpose: we need to measure global innovation across EMC.

I followed up by ceding any authority that I had and giving out a significant amount of autonomy. This autonomy resulted in an initial burst of creativity and results that propelled the strategy forward. I ended up with an aggregation framework for collecting innovation and research activities.

To complete the final phase of execution on the strategy I have consistently been offering my "invisible team" the opportunity to master two new skills that will significantly benefit their careers: innovation and data analytics.

My Own Skill Growth

As I mentioned in the very first post that kicked off this discussion (Stop Playing Chess, Start Playing Risk), I look back about seven years ago and see a set of new career skills (beyond software engineering) that I acquired on a year-by-year basis:

This list is an undeniable message that careers can be reinvented via the acquisition of new skills. The timing of adding these skills is critical. For example, I added my social media skills at a point in corporate history where few people were doing the same. As a result I had a skill that gave me a unique value within my corporation.

It follows that when I'm recruiting a new invisible team for an unapproved strategy, the end game is that they should walk away from my project with new, relevant skills that significantly help their careers.

In 2011, when I was launching this new project, the new buzzword skill was data scientist. I had many, many friends inside EMC that wanted to explore this new skill and see if it was for them.

By signing up for my team, they were immediately thrown into a situation where they could develop their mastery. The results of their efforts were creative to the point where they exceeded expectations, and I credit them with actually launching the new field of Innovation Analytics, something that I have blogged continually about for several years.

As the project proceeded, I saw many of them assume new roles in the corporation that leverage the skills that they were practicing as part of our project. They developed mastery of something new, which was a highly motivating factor for why they joined up in the first place.

So consider purpose, autonomy, and mastery for any project that you are running, whether it is done openly or in stealth.

Often times the manager of one of these employees are quite hesitant to allow their direct reports to perform any task in stealth mode. For these situations I have often relied on commando mentoring, which I will cover in a future post.

April 09, 2014

In a recent post I wrote about mobile connectivity and the need for a robust object storage layer in a data center architecture. This allows thousands of global users to create and access content via technologies such as Syncplicity.

A next logical topic for mobile devices would be the need for fluid user interfaces that reflect two realities:

Users have very little real estate to navigate on their tablets and mobile devices.

The use of social networking paradigms facilitates global collaboration on content.

It’s worthwhile to give a close-up screen shot of some Syncplicityscreens that highlight these two features. The first screen shot shows how Synplicity can use finger swipe on folder navigation to filter through massive amounts of shared content in multiple layers of folders.

The diagram above not only highlights folder navigation but it also shows how easy it is to visualize the people you want to enable content sharing with (e.g. share it with the people from my last meeting).

Syncplicity also has the ability to highlight (on a map) the geographic location of users who either have or have not accessed the content you just shared with them:

The user interface has indeed become so critical that many large data center operators have hired user interface innovation teams to come up with novel ways to provide services to users.

One common question about mobile analyticapplications is their relation to the analysis of content being generated by mobile contentapplications such as Syncplicity. The simple answer is “minimize data movement by keeping it in-place”. In the context of Syncplicity, if the data is ingested using one protocol (object) the infrastructure should also provide a separate mechanism (e.g. HDFS) to analyze it without having to move it.

The diagram below highlights EMC’s approach to solve this problem, which is to use ViPR’s capability to ingest data with one protocol and analyze it with another. The mobile devices on the right leverage the mobile and analytic capabilities within the Spring framework to analyze the data via HDFS.

This concludes a series of articles I’ve written on how to build a third platform infrastructure that is supportive of mobile, social, and analytic applications, from the storage layer all the way up to the pixels on a mobile device.

In upcoming posts I plan a deeper dive into the Spring framework to highlight the usage of Spring to quickly build new applications to run on top of this infrastructure.

April 07, 2014

Over the last few posts I've highlighted some of the building blocks for building a new form of IT infrastructure, often referred to as the 3rd platform. One of the stated reasons for building a new platform was to enable new applications running on mobile devices.

In this post I'll examine mobile applications and their relation to the infrastructure I've introduced in previous posts.

I'd like to discuss two key aspects related to mobile:

The need for newer, fluid user interfaces on mobile devices

The shift away from block/storage APIs towards object APIs.

When reading and writing content from a mobile device, the two items above are equally relevant (and realizable) on both traditional and 3rd platform infrastructures. This point can be highlighted in the context of traditional file management (depicted below).

Traditional content-generating applications would mount on top of a storage device located in a well-known location. This storage device was typically accessed via block and/or file APIs, and applications would use an E: drive or a home directory as the location into which files would be stored and retrieved.

For use cases where there are millions of files being created and accessed from thousands of mobile devices, this infrastructure model doesn't scale.

One of the (many) reasons for this is that often times the mobile content is generated using the internet-based "put" and "get" HTTP method. This approach often maps more naturally to an object-based API approach rather than a file-based approach.

An object-based approach also offers another advantage: there is no longer a need to specifically mount a directory off of a static target storage system. Roaming mobile users are therefore abstracted away from a specific mount point in a specific geographic data center. Object-based APIs are much more friendly to globally-distributed (cloud) storage architectures (e.g. Atmos).

As data center operators begin deploying new infrastructure for new forms of applications, presentation of object services becomes a key piece of the architecture. The provisioning and presentation of object services should be fluid and on-demand (e.g. the storage catalogue approach).

In addition, there is no reason not to implement object services within traditional IT architectures, e.g. re-using the devices that are already in the data center. For example, no matter what type of infrastructure already exists in a data center, EMC's ViPR data services was designed as a translation layer to the underlying storage (ViPR of course can also surface existing object-based systems like Atmos).

Once object services have been architected for mobile content, the next consideration for mobile is the design of new interfaces that can facilitate fluid and social engagement with the consumer (which drives content access to the object service layer).

I will discuss design considerations for these interfaces in an upcoming post.

April 02, 2014

Over the last several weeks I've been establishing a foundational set of building blocks for a new form of IT infrastructure. This infrastructure is well-suited to run new forms of analytic, mobile, and social business applications.

In this post I'd like to describe the final building block: the container within which these new applications are developed and deployed. Before getting there, let me review the layers of building blocks that have been discussed so far:

HDFS is a foundational part of the new infrastructure, but it should be layered underneath an in-memory data grid (IMDG) as an enabler for a data lake architecture. While the architecture seems simple (HDFS + IMDG), the issues associated with creating a true data lake are anything but simple.

In-memory data grids are a part of the industry trend towards server-based storage. Deploying storage at the server tier has ramifications that should be understood before massive scaling of the new infrastructure begins.

Understanding the balance and issues associated with deploying server-based storage alongside an HDD-based tier will be another critical deployment decision.

Given the massive capacity sizes for a data lake architecture, a secure storage tiering strategy to a storage cloud service provider will provide an option to minimize (or eliminate) HDD deployment in a private environment.

The point of this entire exercise is to characterize a next generation infrastructure to run new applications.

What type of infrastructure is required to deploy on top of an SDDC? As I've mentioned previously, the Pivotal framework provides a 3-layer solution to make deployment (and rapid development) happen.

The key benefit that this architecture enables is coding and deployment speed for applications. This speed is realized by combining the benefits of the three layers:

CloudFoundry abstracts the developer away from the underlying cloud infrastructure. Developers write their code to a set of services and avoid any reference to specific infrastructure. This saves time by removing responsibility for creating infrastructure-specific code.

The Data Fabric contains a set of services that put data access at the fingertips of the developers. If the developers favor map/reduce, they can use PivotalHD. For SQL they use HAWQ, and for fast, real-time analytics they use GemFire. The data fabric also contains the Pivotal Analytics toolset for data exploration (via a GUI) as well as data ingest functions (e.g. schema recognition).

The application fabric contains all of the agile tools needed to write software quickly. As I mentioned previously, Spring plays a key role in providing re-usable components to developers, which facilitates coding speed.

The Pivotal approach (and the SDDC constructs underneath it) make a great deal of sense for the creation and deployment of next generation data analytic applications.

What other forms of applications are suitable to run on this platform? The Spring framework has capabilities for social, mobile, etc. In a future post I will dive more deeply into Spring, as well as discuss how to integrate mobile devices into this third platform architecture.

In the meantime, this brings the 3rd platform infrastructure discussion up to the application layer. This unerstanding will facilitate future posts on interoperability between the two infrastructures listed below.

Employer

Volunteer

Disclaimer

The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by DELL Technologies and does not necessarily reflect the views and opinions of DELL Technologies nor does it constitute any official communication of DELL Technologies.